Project Activities
In order to complete this retrospective efficacy study, the researchers obtained administrative data from all NC middle and high schools, including ECHS, and matched the data with longitudinal data from other local, state and federal databases to examine secondary, postsecondary, and other life outcomes (incarceration and voting) for the students. Leveraging data from a prior RCT study on cohorts of students attending 19 ECHS, the researchers conducted a within-study comparison (WSC) to validate quasi-experimental approaches to estimating the impact of attending an ECHS. The WSC utilized a variety of propensity score matching and weighting techniques for the quasi-experimental arm of the study. Using the quasi-experimental approach, the study examined civic outcomes as well as post-secondary education outcomes. The study included an examination of heterogeneity in the effects across subgroups. The researchers also employed a cross design synthesis (CDS) approach to creating a pooled estimate to generalize the effects of the intervention from the RCT sample to the broader set of schools in North Carolina.
Structured Abstract
Setting
Data were obtained from NC high schools (including ECHS), community colleges, universities, and public databases which include incarceration and voter registration information in North Carolina. In addition, data on students' postsecondary completion (within and outside NC) were obtained from the National Student Clearinghouse.
Sample
This study includes data on 479,805 students who attended high school in North Carolina between the 2005-06 and 2011-12 school years. The treatment group (ECHS students) is 55% white, 27% black, 10% Hispanic, 48% economically disadvantaged, and 4% students with disabilities compared to the comparison group (non-ECHS students) which is 58% white, 28% black, 7% Hispanic, 39% economically disadvantaged, and 10% students with disabilities. The treatment samples included 2,174 students who attended ECHS at 19 sites that participated in the lottery study and 10,986 students who attended ECHS at the 51 sites that did not participate in the lottery. These two treatment groups have similar characteristics, though the lottery sites have slightly more black students and slightly more students economically disadvantaged students, while the non-RCT ECHS had slightly more Hispanic students and slightly higher 8th grade algebra passing rate. The traditional high schools in NC have similar student populations to the ECHS, though students in non-ECHS schools have lower baseline achievement, more students with disabilities and fewer gifted students in traditional schools. All differences between treatment and comparison students were considered in estimating impacts.
The ECHS model includes a core set of design principles: college readiness, powerful teaching and learning, personalization, redefined professionalism, leadership, and purposeful design. ECHS are limited to 400 students. ECHS are generally co-located on the campus of a two- or four-year college and allow students to obtain two years of college credit or an associate's degree concurrent with completing a high school diploma at no cost to the student. ECHS may be four- or five-year programs.
Research design and methods
Using statewide administrative data and student-level information about lotteries conducted by ECHS, the best performing (lowest bias) propensity score models were chosen from a within-study replication analysis that tests models using data from the 19 lottery sites as a benchmark. A variety of covariate sets and propensity score techniques were tested separately for each outcome of interest. The best performing quasi-experimental designs were used to conduct a heterogeneity analysis as well as to extend the analysis to civic outcomes, including voting, voter registration, and crime.
Control condition
The comparison students were those attending a traditional public high school in the state of North Carolina in the same cohorts as the treatment students.
Key measures
Intermediate outcome measures include administrative outcomes for high schoolers — NC standardized end of grade test scores, absences, retention, ACT scores, and graduation — as well as a researcher developed on track measure based on course taking. Postsecondary outcomes include measures derived from administrative data including enrollment in and graduation from two-year and four-year colleges. Civic outcome measures drawn from publicly available sources include voter registration, voting and criminal convictions. Pre-treatment covariates include levels and trends of middle school NC end of grade standardized test scores, absences and suspensions, economic disadvantage, school enrollment changes, school-based educational classifications, and socio-demographic characteristics.
Data analytic strategy
Many different analysis strategies were used in the study including propensity score methods, experimental impact estimation, within-study comparison design approaches, moderation analysis, sensitivity analysis, and cross-design synthesis.
Key outcomes
The main findings of this study are as follows:
- The study found that many benefits of attending an ECHS on high school and post-secondary outcomes were similar to those reported in prior studies, although there are some differences across subgroups of students and schools due to sample selection and moderation effects (Fuller, Lauen, & Unlu, 2020).
- The benefits documented in prior RCT studies are not limited solely to the types of schools that are overenrolled and willing to participate in lottery studies (Fuller, Lauen, & Unlu, 2020).
- Attending an ECHS has small positive impacts on voter registration and voting and large impacts on criminal convictions. These conclusions are drawn from propensity score models because the RCT study impacts on these outcomes are highly imprecise. (Swiderski, et al., 2021).
- A pretest measure is very important to replicating RCT results with a quasi-experimental design, but contrary to best practices recommended within the literature, local matching estimates do not always perform better when the restriction to a local match leads to worse balance on observables (Unlu, et al., 2021).
People and institutions involved
IES program contact(s)
Products and publications
The products of this project are evidence of the efficacy of ECHS for high school students in NC, a newly developed statistical technique, and peer-reviewed publications.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Selected Publications:
Fuller, S., Lauen, D. L., & Unlu, F. (2020). Leveraging Experimental and Observational Evidence to Assess the Generalizability of the Effects of Early Colleges in North Carolina. Education Finance and Policy, 1-62.
Swiderski, T., Lauen, D. L., Fuller, S. C., & Unlu, F. (2021). A path towards citizenship: The effects of early college high schools on criminal convictions and voting. Social Science Research, 99, 102584.
Unlu, F., Lauen, D. L., Fuller, S. C., Berglund, T., & Estrera, E. (2021). Can Quasi-Experimental Evaluations That Rely On State Longitudinal Data Systems Replicate Experimental Results?. Journal of Policy Analysis and Management, 40(2), 572-613.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
