Co-Principal Investigator: Roberts, Greg; Martinez, Leticia; Wanzek, Jeanne

Purpose: The purpose of this study is to test the efficacy of the reading comprehension intervention, PACT—Promoting Adolescents' Comprehension of Text, for middle school learnerswith disabilities. PACT has demonstrated efficacy for improving outcomes for students without disabilities, but has not been rigorously evaluated for learners with disabilities. Data from the prior efficacy study do suggest the intervention is promising for learners with disabilities. Many adolescents with disabilities do not adequately understand complex texts, which can reduce their school success, access to postsecondary learning, and future career opportunities. Since most students with disabilities are educated in the general education classroom, there is a clear need for effective instructional practices for the teachers in these classrooms. The goal of the current study is to test the efficacy of a evidence-based reading comprehension intervention, PACT, for middle school students with disabilities in general education social studies classrooms. In addition, the study will examine the mediating effects of implementation fidelity and teachers' academic interactions with students, the moderating effects of student characteristics, and the costs and cost-effectiveness of the intervention.

Project Activities: A multi-site randomized controlled trial will be conducted to determine the efficacy of PACT for eighth-grade students with disabilities in general education social studies classrooms. Two cohorts of eighth-grade general education social studies teachers and their students (from four classes per teacher) will be recruited to participate in the study for two years. Matched pairs of classes within teacher will be randomized to either the PACT intervention or the business-as-usual condition. Teachers will receive professional development and coaching through workshops, booster sessions, and in-class coaching to implement the intervention. The researchers will examine the impact of PACT on the reading outcomes of students with disabilities and the conditional effects of teacher- and student-level factors using mediation and moderation analyses. They will also examine the costs and cost-effectiveness of PACT.

Products: The products of this project will include evidence of the efficacy of the PACT reading comprehension intervention for students with disabilities. The project will result in a final dataset to be shared, peer-reviewed publications and presentations, and additional dissemination products that reach education stakeholders such as practitioners and policymakers.

Structured Abstract

Setting: The research will take place in urban and suburban middle schools in Texas and Tennessee.

Sample: Approximately 32 schools and 80 eighth-grade social studies teachers will participate in the study. Students in four social studies classes taught by each teacher in each of the two years will also participate for a total of approximately 16,000 eighth-grade students. Each teacher's class is expected to have two or more students with disabilities; thus the sample will include approximately 1,280 students with disabilities.

Intervention: PACT is a fully developed text- and inquiry-based reading comprehension intervention that was developed and tested for efficacy with secondary students in social studies classrooms as part of a previously-funded IES project. The PACT intervention is a set of instructional supports for teachers and materials for students that is made up of the following components: an overarching question that guides students through a unit, essential word instruction, content acquisition through text reading, and team-based learning activities. The intervention is implemented by teachers daily in general education social studies classrooms for approximately 8-10 consecutive weeks during the fall semester. The research team provides a 1-day professional development wokshop to participating teachers on PACT practices which addresses the following topics: (a) an overview of the study, (b) a detailed description of the experimental design with an emphasis on adherence to intervention procedures, (c) implementation of PACT intervention components, and (d) classroom practices to facilitate discussion focused on text evidence. After the first unit, teachers receive a 1-day follow-up booster session where researchers review components of the intervention, address questions, and problem solve any areas of challenge. In-class coaching is provided during implementation of each of the three units.

Research Design and Methods: Researchers will conduct a multi-site randomized controlled trial to determine the efficacy of the PACT intervention for students with disabilities in eighth-grade general education social studies classes. In Year 1, eighth-grade general education social studies teachers will be recruited to participate in Cohort 1 of the study with data collected in Years 2 and 3. Additional eighth-grade general education social studies teachers will be recruited to participate in Cohort 2 with data collected in Years 4 and 5. Each cohort will include 20 teachers per state (Texas and Tennessee), and each teacher will teach approximately 4 social studies classes for a total of 80 classes per state per cohort and approximately 2,000 students (including 160 students with disabilities) per state per cohort. Researchers will randomize classes within teacher to either the PACT intervention or business-as-usual condition. First, classes will be matched based on pre-reading scores of students with disabilities and then the matched pairs will be randomized to either the PACT intervention or business-as-usual condition within teacher. The researchers will examine the impact of PACT on the outcomes of students with disabilities and the mediating and moderating effects of teacher- and student-level factors. Researchers will assess implementation fidelity and account for any crossover between the intervention and business-as-usual classes. The researchers will conduct a cost analysis to determine the cost of the intervention over and above what schools typically spend on instruction and an incremental cost-effectiveness analysis.

Control Condition: Classes assigned to the business-as-usual condition will engage in the school's usual social studies instruction.

Key Measures: Teacher measures include observations of teachers' academic interactions with students with disabilities and fidelity of PACT implementation using researcher-created data collection instruments from prior studies. Measures to assess student learning include a standardized measure of reading comprehension (Gates-MacGinitie Reading Test-4th Edition) and a measure of social studies content knowledge (Assessment of Social Studies Knowledge and Comprehension). Student demographic data will also be collected to assess potential moderators of English language learner status and special education status.The costs of implementing the program will be assessed using the ingredients approach. Data will be collected from the project budget and interviews with project and school personnel to catalogue all resources that would be needed to replicate every aspect of PACT beyond those already required for typical instruction. Costs for all ingredients will be based on national prices provided in the Center for Benefit-Cost Studies in Education's Cost Out Tool Kit.

Data Analytic Strategy: The researchers will use multilevel multiple-group regression and structural equation models (estimating three-level models with classes at Level 1, teachers at Level 2, and students at Level 3) to estimate the impact of PACT. Moderating effects of student variables and mediating effects of fidelity will be evaluated as cross-level interactions within the multi-level model. For the implementation study, researchers will estimate compliance-adjusted effects using an instrumental variable approach, where "compliance" represents levels of fidelity of implementation. For each key student outcome in the efficacy study, the resulting effect sizes and costs associated with the program will be used to compute incremental cost-effectiveness ratios.

Related Projects: Understanding Malleable Cognitive Processes and Integrated Comprehension Interventions for Grades 7–12 (R305F100013)

Project Website: https://www.meadowscenter.org/projects/detail/promoting-adolescents-comprehension-of-text-pact