Purpose: This project provides evidence about the effectiveness of Early College High Schools (ECHS) in North Carolina (NC) which are small schools of choice primarily located on campuses of two- or four-year colleges or universities. Students who attend ECHS have the opportunity to earn, at no financial cost to them, two years of transferable college credit or an associate's degree while simultaneously satisfying high school graduation requirements. This study leveraged existing data to provide information about whether, how, and for whom NC ECHS are effective. Prior IES-funded research (Follow-Up to the Study of the Efficacy of North Carolina's Early College High School Model) examined impacts from 19 of the 70 ECHS sites in NC. These 19 sites used a lottery for admissions to ECHS. Lotteries approximate random assignment designs and allow for rigorous assessment of efficacy. This study built on the prior study by examining longer term outcomes, including civic outcomes such as incarceration and voting. In addition, this study used the data and results of the prior study to build complex statistical models to replicate the RCT and apply, retrospectively, to the other 51 NC ECHS to estimate their effects. A secondary aim of the project was to develop and refine this new statistical technique. A third aim was to assess the generalizability of the ECHS effects to sites that did not hold a lottery.

Project Activities: In order to complete this retrospective efficacy study, the researchers obtained administrative data from all NC middle and high schools, including ECHS, and matched the data with longitudinal data from other local, state and federal databases to examine secondary, postsecondary, and other life outcomes (incarceration and voting) for the students. Leveraging data from a prior RCT study on cohorts of students attending 19 ECHS, the researchers conducted a within-study comparison (WSC) to validate quasi-experimental approaches to estimating the impact of attending an ECHS. The WSC utilized a variety of propensity score matching and weighting techniques for the quasi-experimental arm of the study. Using the quasi-experimental approach, the study examined civic outcomes as well as post-secondary education outcomes. The study included an examination of heterogeneity in the effects across subgroups. The researchers also employed a cross design synthesis (CDS) approach to creating a pooled estimate to generalize the effects of the intervention from the RCT sample to the broader set of schools in North Carolina.

Key Outcomes: The main findings of this study are as follows:

• The study found that many benefits of attending an ECHS on high school and post-secondary outcomes were similar to those reported in prior studies, although there are some differences across subgroups of students and schools due to sample selection and moderation effects (Fuller, Lauen, & Unlu, 2020).
• The benefits documented in prior RCT studies are not limited solely to the types of schools that are overenrolled and willing to participate in lottery studies (Fuller, Lauen, & Unlu, 2020).
• Attending an ECHS has small positive impacts on voter registration and voting and large impacts on criminal convictions. These conclusions are drawn from propensity score models because the RCT study impacts on these outcomes are highly imprecise. (Swiderski, et al., 2021).
• A pretest measure is very important to replicating RCT results with a quasi-experimental design, but contrary to best practices recommended within the literature, local matching estimates do not always perform better when the restriction to a local match leads to worse balance on observables (Unlu, et al., 2021).

Structured Abstract

Setting: Data were obtained from NC high schools (including ECHS), community colleges, universities, and public databases which include incarceration and voter registration information in North Carolina. In addition, data on students' postsecondary completion (within and outside NC) were obtained from the National Student Clearinghouse.

Sample: This study includes data on 479,805 students who attended high school in North Carolina between the 2005–06 and 2011–12 school years. The treatment group (ECHS students) is 55% white, 27% black, 10% Hispanic, 48% economically disadvantaged, and 4% students with disabilities compared to the comparison group (non-ECHS students) which is 58% white, 28% black, 7% Hispanic, 39% economically disadvantaged, and 10% students with disabilities. The treatment samples included 2,174 students who attended ECHS at 19 sites that participated in the lottery study and 10,986 students who attended ECHS at the 51 sites that did not participate in the lottery. These two treatment groups have similar characteristics, though the lottery sites have slightly more black students and slightly more students economically disadvantaged students, while the non-RCT ECHS had slightly more Hispanic students and slightly higher 8th grade algebra passing rate. The traditional high schools in NC have similar student populations to the ECHS, though students in non-ECHS schools have lower baseline achievement, more students with disabilities and fewer gifted students in traditional schools. All differences between treatment and comparison students were considered in estimating impacts.

Intervention: The ECHS model includes a core set of design principles: college readiness, powerful teaching and learning, personalization, redefined professionalism, leadership, and purposeful design. ECHS are limited to 400 students. ECHS are generally co-located on the campus of a two- or four-year college and allow students to obtain two years of college credit or an associate's degree concurrent with completing a high school diploma at no cost to the student. ECHS may be four- or five-year programs.

Research Design and Methods: Using statewide administrative data and student-level information about lotteries conducted by ECHS, the best performing (lowest bias) propensity score models were chosen from a within-study replication analysis that tests models using data from the 19 lottery sites as a benchmark. A variety of covariate sets and propensity score techniques were tested separately for each outcome of interest. The best performing quasi-experimental designs were used to conduct a heterogeneity analysis as well as to extend the analysis to civic outcomes, including voting, voter registration, and crime.

Control Condition: The comparison students were those attending a traditional public high school in the state of North Carolina in the same cohorts as the treatment students.

Key Measures: Intermediate outcome measures include administrative outcomes for high schoolers — NC standardized end of grade test scores, absences, retention, ACT scores, and graduation — as well as a researcher developed on track measure based on course taking. Postsecondary outcomes include measures derived from administrative data including enrollment in and graduation from two-year and four-year colleges. Civic outcome measures drawn from publicly available sources include voter registration, voting and criminal convictions. Pre-treatment covariates include levels and trends of middle school NC end of grade standardized test scores, absences and suspensions, economic disadvantage, school enrollment changes, school-based educational classifications, and socio-demographic characteristics.

Data Analytic Strategy: Many different analysis strategies were used in the study including propensity score methods, experimental impact estimation, within-study comparison design approaches, moderation analysis, sensitivity analysis, and cross-design synthesis.

Related IES Projects: Follow-Up to the Study of the Efficacy of North Carolina's Early College High School Model (R305A110085)

Products and Publications

Products: The products of this project are evidence of the efficacy of ECHS for high school students in NC, a newly developed statistical technique, and peer-reviewed publications.

ERIC Citations: Find available citations in ERIC for this award here.

Selected Publications:

Fuller, S., Lauen, D. L., & Unlu, F. (2020). Leveraging Experimental and Observational Evidence to Assess the Generalizability of the Effects of Early Colleges in North Carolina. Education Finance and Policy, 1–62.

Swiderski, T., Lauen, D. L., Fuller, S. C., & Unlu, F. (2021). A path towards citizenship: The effects of early college high schools on criminal convictions and voting. Social Science Research, 99, 102584.

Unlu, F., Lauen, D. L., Fuller, S. C., Berglund, T., & Estrera, E. (2021). Can Quasi-Experimental Evaluations That Rely On State Longitudinal Data Systems Replicate Experimental Results?. Journal of Policy Analysis and Management, 40(2), 572–613.