Evaluation of the Content Literacy Continuum: Report on Program Impacts, Program Fidelity, and Contrast

This report presents the findings of a rigorous experimental impact evaluation and implementation study of one such intervention, the Content Literacy Continuum (CLC), developed by researchers at the University of Kansas Center for Research on Learning. This evaluation of CLC was conducted by three partnering organizations: REL Midwest, MDRC, and Survey Research Management. Thirty-three high schools in nine districts across four Midwestern states agreed to participate in this evaluation, and 28 of those 33 schools continued their participation throughout the entire study period. Full implementation of this intervention began in the 2008/09 school year and continued through the 2009/10 school year. Given that CLC was designed to address the dual needs of high schools to support both the literacy and content learning of students, the evaluation focused on program impacts on reading comprehension test scores and students' accumulation of course credits in core content areas. To assess the impacts of CLC on these outcomes, the study team conducted a cluster randomized trial. That is, participating high schools within each district were randomly assigned either to implement CLC (CLC schools) or to continue with "business as usual" (non-CLC schools). Impacts were estimated by analyzing the outcomes of students at the CLC schools compared with those at the non-CLC schools. The evaluation's primary research questions focused on the impact of CLC on students' reading comprehension and course performance at the end of the second year of implementation. Secondary research questions compared the first-year impacts and second-year impacts and also investigated program impacts on other student outcomes. In addition, the evaluation examined the implementation of the CLC framework within the CLC schools. This report presents findings regarding the degree to which schools assigned to implement CLC set up the necessary structures and organizational processes needed to support implementation of CLC (referred to as structural fidelity in this report) and the degree to which the pedagogical practices emphasized in CLC-related professional development were apparent within the instruction of core content teachers in participating schools (referred to as instructional fidelity). The structures and instruction at CLC schools and non-CLC schools also were compared to provide information about the contrast that CLC implementation provided compared with business as usual. Appended are: (1) Sampling of Classrooms for GRADE Testing and Classroom Observations; (2) Measuring Instructional Practice with the ACE Observation Protocol; (3) Characteristics of the Student Analysis Samples; (4) Characteristics of the Sampled and Observed Classrooms; (5) Technical Notes for the Impact Analysis; (6) Statistical Power and Minimum Detectable Effect Size; (7) Description of Frequently Introduced Content Enhancement Routines and Learning Strategies; (8) Explanation of Shared Pedagogical Practices Obtained through Classroom Observations Using the ACE Protocol; (9) Interrater Reliability for Shared Pedagogical Features; (10) Robustness of Estimated Impacts on Student Outcomes; (11) Model Fit Information; (12) Descriptive Statistics for Student Outcomes and Impact Model Covariates; (13) Impact Findings and Sample Characteristics for All Study Schools in Year 1; (14) Additional Impact Findings: Credits Attempted, Successful Credit Completion, and Attendance; and (15) Impacts by Student Subgroup. (Contains 17 figures, 89 tables, 1 box, and 198 footnotes.)