Project Activities
The proposed evaluation will assess the impact of NGAs in BCPS on a range of short-term and longer term student engagement and performance outcomes. It will also assess the fidelity with which the NGA components are being implemented and examine the way ninth-graders are treated in high school that do not use NGAs. The evaluation will also measure the costs associated with implementing NGAs, including new resources that are brought into the high schools and existing resources that are redeployed. A second phase of the evaluation will estimate the impact of NGAs in a set of schools outside BCPS but within the State of Florida. Using early results from the evaluation, the research team will work with BCPS to develop a range of enhancements to the existing NGAs in the district and submit them to a rigorous randomized controlled trial (RCT).
Structured Abstract
Setting
The core study will include 25 schools from BCPS, which serves a unique urban/suburban mix of more than 258,000 students; 25 comparison schools not implementing NGAs for the core study will come from similar districts in Florida. A second component includes a statewide sample of 50 schools from Florida, half implementing NGAs and half not, which will also be a mix of urban and suburban settings. The third component is an evaluation of enhancements to NGAs already operating in BCPS.
Sample
The evaluation focuses on outcomes for cohorts of ninth-grade students from ninth grade through graduation. In BCPS, the student population is quite diverse with 38% black, 30% white, and 26% Hispanic. Over 40% of students are economically disadvantaged. For the core study, the study body in matched comparison schools will be similar to that of BCPS. The statewide sample will focus on schools with significant populations of students from low-income families.
NGAs are self-contained units located in a designated area of the school. The academy is staffed with its own dedicated teaching faculty, guidance staff, and social services to create an intact community for this transition year. These NGAs are organized around interdisciplinary teacher teams that have students and planning times in common. NGAs support personalized relationships among and between students and teachers. The teaming of teachers and students support more consistent classroom composition and student peer groups while decreasing anonymity and increasing students' sense of community. Students have a consistent group of teachers who are accountable for their success, and teachers have a chance to coordinate their coursework to better meet the needs of their students.
Initial research
The impact analyses will employ both comparative interrupted time series (for the core and statewide components) and school-level random assignment designs (for the enhancements) to estimate the impact of NGAs on student outcomes. The accompanying implementation analyses will use a mixed methods approach to produce a comprehensive understanding of the social and academic supports available to ninth-graders across sample schools.
- Core Broward County Study: This component of the evaluation focuses on high schools in BCPS that have implemented Ninth Grade Academies and uses similar high schools in Florida school districts outside of Broward County as the comparison group in a comparative interrupted time series (CITS) design. The study will also examine trends in "untreated academic outcomes" to assess the validity of making causal inferences from the CITS design;
- Statewide Extension: This will include high schools from across the State of Florida that are implementing NGAs, as well as high schools in the state that are not implementing NGAs as the comparison group in a second comparative interrupted time series design. The study will also examine trends in "untreated outcomes."
- Enhancement Study: In partnership, MDRC and BCPS will work together to use early findings from the core study to identify enhancements for NGAs that have the most promise of increasing the impact on student outcomes. The enhancement study will use a school-level random assignment design within BCPS to assess the effectiveness of NGAs that have been strengthened with additional components or supports. Half of the BCPS schools will be randomly assigned to implement enhancements and half will continue "business as usual" in their NGAs.
- Implementation Study: Information from quantitative surveys of teachers and students will be used to gauge NGA implementation in treatment schools and to investigate the practices and strategies that support ninth graders in comparison schools. On-site observations and interviews will be used to verify and augment implementation survey data, identify elements of the intervention not anticipated by the survey, and gain a deeper understanding of historical, contextual and qualitative aspects of implementation that may limit or enhance the impact of the school's NGA on student engagement and achievement.
Control condition
Control schools are to continue with their ongoing practices (a "business-as-usual" control). To the extent possible comparable information is collected in the control and treatment conditions (e.g., staff composition, curriculum/course offerings and enrollments, evidence of teaching teams and common planning time for teachers, support services, sense of personalization).
Key measures
For the impact study, primary measures will be related to student engagement, attitudes and achievement. Key outcomes include grade-level promotion, credits earned, attendance rates, graduation rates, and test scores. For the implementation analyses, measures include structural and procedural elements of NGAs, as well as measures to capture conditions and practices that enhance or limit implementation and effectiveness of NGAs. School-level surveys and site visits will be the primary sources for implementation data.
Data analytic strategy
Data for the impact analyses is nested—individual students are nested within schools—and observations within the same group are not statistically independent of one another. Multilevel models will be used to estimate the effect of the NGAs and examine and influence of implementation and potential mediators of effects. The study will attend to variations in school conditions and the degree to which schools are implementing core NGA elements and/or additional practices.
People and institutions involved
IES program contact(s)
Products and publications
Publications:
Nongovernment report, issue brief, or practice guide
Legters, N., and Parise, L. (2016). A Community of Practice Approach to Support for Ninth-Graders in Urban High Schools. Policy Brief. MDRC.
Legters, N., Parise, L., and Rappaport, S. (2013). Implementing Ninth Grade Academies in Broward County, Florida. MDRC.
Somers, M. A., and Garcia, I. with Quint, J. (2016). Helping Students Make the Transition into High School: The Effect of Ninth Grade Academies on Students' Academic and Behavioral Outcomes. MDRC.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
