Co-Principal Investigators: Corrin, William; Kemple, James

Purpose: Amidst modest increases in high school graduation rates for all students, stark differences still exist between the supports and quality of secondary schooling provided to low-income students of color. In response to this equity issue, there has been a growth in third-party "school support" organizations like Urban Assembly (UA): groups created to independently interface with schools to provide additional monetary, organizational, instructional, personnel and professional development resources. This growth has occurred nationally, within school districts as "intermediary organizations" or "partnership support organizations" and within the charter school sector as "charter management organizations." Despite the large monetary investment in these organizations there is little rigorous, peer-reviewed research on their effectiveness. The project will help fill this gap with a robust set of activities that would yield lessons for other school support organizations on best practices, as well as apply the most rigorous causal approach to understanding what works to improve student outcomes.

Project Activities: The research team will evaluate the effects of the UA school support network on the academic and socio-emotional outcomes of students who were assigned to a UA school through a naturally occurring lottery as part of the NYC high school admissions process. A complimentary propensity score matching design will be used to estimate effects for all students assigned to a UA school. An implementation study will complement these findings by exploring the fidelity of implementation and facilitators/barriers to implementation. A landscaping study of school support networks in NYC to understand the NYC school support system and a cost study to understand the cost of implementation and cost effectiveness of the model.

Products: The research team will produce at least three peer-reviewed academic journal papers: one on the average impact of UA, one on variation in UA impacts, and one on the cost effectiveness of the UA intervention. In addition, they will publish at least two policy briefs and practitioner briefs and share them on MDRC's website. The researchers will make presentations to multiple stakeholders, including education researchers, policymakers, practitioners, and education reform funders.

Structured Abstract

Setting: New York City, the country's largest school district, where more than 82,000 8th graders choose from more than 700 high school programs. Students are not automatically assigned to their neighborhood school; rather, students submit an application to the district and rank up to 12 of their high school choices, including UA schools.

Sample: The target population will be all students that are assigned to UA schools through the high school admissions process, roughly 2,500 a year through the 13 years of the study period. 54 percent are Hispanic and 35 percent Black. 91 percent of the UA assignees have at least one indicator of poverty and 27 percent of the students were chronically absent in 8th grade. 47 percent and 31 percent of the UA assignees performed in the lowest level (not meeting learning standards) on their 8th grade state standardized math and ELA exams, respectively.

Intervention: UA is an experienced school support network based in NYC that has been working with schools since 1997. Under contract with the New York City Department of Education (NYC DOE), in the 2019–20 school year UA supported 23 high schools, serving roughly 10,000 students throughout Manhattan, Brooklyn, and the Bronx. UA provides their schools with professional development to school-based staff, using two formats: professional learning communities (PLCs) and coaching. UA offers schools five main program areas of support: Academic, Leadership, Social Emotional Learning (SEL), Postsecondary Readiness (PSR), and Alumni Success. Each program area consists of several established approaches grounded in evidence-based education reform strategies.

Research Design and Methods: The research team will take advantage of naturally occurring lotteries between the 2011–12 and 2024–25 school years, which create the equivalent of a student-level randomized controlled trial, to estimate the causal impact of the UA high school support network on students' academic outcomes. The impact study will be anchored in a high powered, naturally occurring RCT design executed to meet What Works Clearinghouse (WWC) standards; complemented by a uniquely rigorous propensity score matching approach. An implementation study will document the UA intervention, focusing on its fidelity to the theory of action and each UA school's exposure to these components. A landscaping study will explore the context for the intervention by capturing the NYC DOE system of school support organizations (these data will be used to document the comparable practices available to non-UA schools and students).

Control Condition: Control group students are those who randomly lost the opportunity to attend a UA school through the high school admissions lottery. These control students are spread across 186 NYC public schools each year. The team will focus instead on capturing the counterfactual student experience through student-, teacher- and parent-level NYC DOE Learning Environment and Senior Exit surveys to assess the treatment contrast. This analysis will inform the implementation study, the landscaping study, and the mediator analysis.

Key Measures: The research team specify three confirmatory outcomes: (1) a measure of whether or not a student is on-track for high school graduation at the end of ninth grade, (2) an indicator of whether a student graduated from high school after four years, and (3) an indicator of whether a student graduated from postsecondary education after six years. In an exploratory analysis, the researchers will measure UA impacts on measures of students' social and behavioral competencies. Students' socio-emotional outcomes will be explored through their responses on student-level surveys.

Data Analytic Strategy: The basic approach for the present analysis is to estimate, for each lottery, differences in mean outcomes for treatment and control group members and to average the results across lotteries. This averaging will identify the intent to treat (ITT) effect — i.e., the effect of winning a lottery on students' outcomes, regardless of enrollment patterns. We will conduct identical regression analyses with different outcomes. An analysis of impact variation (moderator and mediator) and an analysis of UA's costs—will draw on data collected through both the impact and implementation work.

Cost Analysis: For the cost analysis, researchers will use the ingredients approach. This method first identifies all the "ingredients" necessary to implement the intervention, and then monetizes them using a variety of quantitative and qualitative data sources. The ingredients are summed and averaged across the student sample to obtain the average per pupil costs of the intervention. The research team will use estimates from the impact analysis in the study to conduct cost-effectiveness analysis at its conclusion.