Previous Award Number: R305A090301
Previous Awardee: University of Wisconsin

Co-Principal Investigators: Burch, Patricia; Meyer, Robert, Mehrotra, Nidhi

Purpose: The project will evaluate the efficacy of supplemental education services provided to low-income students as required under the No Child Left Behind Act of 2001. As part of this work, the project will: (1) estimate the net impacts of supplemental education services on student achievement; (2) identify the particular design elements and curricular and instructional components of programs that contribute to these impacts; (3) investigate whether those students who are most in need of extra academic assistance are enrolling in and attending these services; (4) investigate which factors influence parent and student choices in selecting and staying with service providers; and (5) identify what policy levers are available to state and local educational agencies to increase service program effectiveness.

Project Activities: The project will examine the provision of supplemental education services and their impact on student achievement in five urban school districts in four states. A longitudinal mixed method study design will include in-depth field research on the implementation of the services and quasi-experimental evaluation methods to estimate the impact on students' mathematics and reading achievement and their performance in regular school courses. The field research will involve repeated observations of service sessions with different providers, interviews with service tutors and program directors, interviews with district and state officials, focus groups with parents, and analysis of relevant instructional and policy documents. Secondary data analysis will be used to estimate the impact of the services on student achievement.

Products: The products of this project will be published reports on the impact of supplemental education services on students and information on how to improve these services and increase student participation in them.

Structured Abstract

Setting: The project will include students and service providers in five urban school districts in four states and include: Milwaukee, Wisconsin; Minneapolis, Minnesota; Chicago, Illinois; and Austin and Dallas, Texas.

Population: The project will examine approximately 275,000 students eligible for supplemental education services in about 350 schools required to offer such services in the five urban districts. The number of eligible students varies per district ranging from over 6,000 students (Austin) to more than 230,000 (Chicago). The sample of service providers will consist of four to six providers from each of the five districts and are to include at least one provider from each of the four primary program models (online, in-school, community-sited, and in-home).

Intervention: Supplemental educational services were introduced by the No Child Left Behind Act of 2001. Public schools not making adequate yearly progress in increasing student academic achievement for three years are required to offer children in low-income families the opportunity to receive extra academic services paid for by Title I funds. Service providers can include nonprofit, for-profit, faith-based, and community organizations, and local educational agencies. The services they provide to eligible students are to focus on reading/language arts and mathematics, and must take place outside the regular school day. States are to withdraw approval from service providers that fail to increase student academic achievement for two years.

Research Designs and Methods: The design includes three phases. Phase 1 will be an in-depth qualitative study to define key elements of service program models and policy and practice variables that mediate implementation of the models. Phase 1 will use interviews, observations, focus groups, and curriculum analysis to describe the service programs and the treatments. The purpose of this phase will be to identify similarities and dissimilarities in how instruction is organized and practiced within the different service program models.

Phase 2 will be a quantitative study investigating student selection into services (i.e., who registers and participates) and net service program impacts, using an interrupted time series design with nonequivalent (internal, no-treatment) comparison groups. Student data from the 2007–2008 through the 2012–2013 school years are to be used in this analysis.

Phase 3 will be a follow-up qualitative study to examine whether program features identified in Phase 1 continue over time and to further inform the interpretation of quantitative findings of program impact from Phase 2.

Control Conditions: The comparison will be made within the population of students in the districts eligible for receiving supplemental education services, specifically those who take them compared to those who do not. To reduce the possibility of selection bias (e.g., more motivated students using the services or student in most need using the services), the comparison group will be selected using propensity score matching in which participants are matched with individuals in a comparison group based on an estimate of the probability that the individual receives treatment. Selection bias is also to be addressed in the methods used to analyze the data.

Key Measures: The key student outcome measures are the standardized tests administered to students in the districts. These include: (1) the Texas Assessment of Knowledge and Skills in Austin and Dallas; (2) the Northwest Achievement Levels Test (NALT) or Computerized Achievement Levels Test (CALT) and the spring Minnesota Comprehensive Assessments–Series II (MCA-II) or the Mathematics Test for English Language Learners in Minneapolis; (3) the Wisconsin Knowledge and Concepts Examination (WKCE) and the Milwaukee Public Schools Benchmark Assessments in Milwaukee; and (4) the Illinois Standards Achievement Test (ISAT) in Chicago. To the extent that the school transcript data allow, course grades in mathematics and grade point average (GPA) will also be used as measures of student achievement.

Data Analytic Strategy: Two methods of analysis will be used with the longitudinal data. First, a difference-in-differences analysis that can eliminate bias resulting from time invariate differences between participants and nonparticipants will be applied. Second, a fixed-effects model specification will be used to account for the possibility that students who participate in services differ from non-participants in ways that are correlated with achievement growth.

Project Website: http://sesiq2.wceruw.org/papers.html

Products and Publications

Journal article, monograph, or newsletter

Good, A., Burch, P., Stewart, M., Acosta, R., and Heinrich, C. (2014). Instruction Matters: Lessons From a Mixed-Method Evaluation of Out-of-School Time Tutoring Under No Child Left Behind. Teachers College Record, 116 (3).

Heinrich, C., and Nisar, H. (2013). The Efficacy of Private Sector Providers in Improving Public Educational Outcomes. American Educational Research Journal, 50 (5): 856–894.

** This project was submitted to and funded under Education Policy, Finance, and Systems in FY 2009.