Precedent from Welfare Policy in 80s and 90s Shows How It Can Work

Problem to be addressed: The dearth of scientifically-valid knowledge about "what works" in K–12 education that schools/districts can use to meet Adequate Yearly Progress requirements.

The No Child Left Behind Act (NCLB) provides strong incentives for schools and districts to improve educational outcomes of their students, in order to meet "Adequate Yearly Progress" requirements. NCLB also requires them to implement educational practices backed by "scientifically-based research." Yet schools and districts are missing a critical piece needed to implement these requirements: scientifically-valid knowledge about what works they can use to improve practice. Specifically, educational practices that have been proven effective in randomized controlled trials—research's gold standard for establishing what works, and the preferred study design set out in NCLB—are rare or, in many areas of education, nonexistent, leaving schools and districts with few research-proven tools they can use to meet NCLB requirements.

Recommendation 1—That the Department use its "waiver" authority to build valid knowledge about which educational interventions (e.g., classroom curricula, teacher professional development programs) are truly effective in enabling schools to make Adequate Yearly Progress.

Specifically, NCLB gives the Education Department the authority to waive certain provisions of federal law and regulation to enable states and districts to carry out innovative demonstration projects of new educational practices. We recommend that the Department, using this authority, put in place an official policy of granting waivers for demonstration projects that include two elements:

1. They implement an educational intervention which, based on existing evidence, shows major promise in enabling schools to make Adequate Yearly Progress (e.g., could potentially have a sizeable impact on reading or math achievement, based on evidence from small-scale randomized controlled trials or matched comparison-group studies).
   
   As illustrative examples, such interventions might include highly-promising teacher recruitment and professional development programs, classroom curricula, supplemental services such as tutoring, or schoolwide reform programs. Demonstration projects would generally be designed to implement such an intervention on a large scale, in typical public school settings.
   
   
2. They provide for an independent, randomized controlled trial to evaluate the effectiveness of the intervention in this typical school setting (i.e., an "effectiveness" trial).

The Department's waiver would allow the schools and districts carrying out these projects to calculate their Adequate Yearly Progress either with or without the students who participate in the randomized controlled trial—whichever calculation yields the higher score.

Such a waiver policy would be attractive to many troubled schools and districts, giving them the flexibility to conduct demonstration projects designed to make Adequate Yearly Progress, in a way that also creates scientifically-valid knowledge that many schools and districts can then use to make Adequate Yearly Progress. Importantly, this policy would require no additional expenditure of federal funds, and could be implemented by the Department within its existing statutory authority.

Recommendation 2—That the Department also use its waiver authority to rigorously evaluate the effect on student achievement of variations it allows in NCLB's accountability rules themselves (e.g., in the sanctions for schools not making Adequate Yearly Progress, or in the formula for calculating Adequate Yearly Progress).

Examples of waivers in NCLB accountability rules that the Department has granted include: (i) allowing several school districts that are not making Adequate Yearly Progress to provide free tutoring to their students before they offer public school choice (rather than offer school choice first, as required by NCLB); and (ii) allowing several states to use "growth models" to determine whether schools make Adequate Yearly Progress (rather than a model based on the absolute level of student achievement, as set out in NCLB).

We recommend that, in the future, whenever the Department grants such a waiver in NCLB's accountability rules to test a new approach that could potentially have important effects on student achievement, it require the entity receiving the waiver to conduct a randomized controlled trial to evaluate the waiver's effect on such achievement. For example, if the Department provides additional waivers to allow troubled school districts to provide free tutoring to their students before offering public school choice, it would require schools in each district to be randomly assigned to a group that provides tutoring first versus a group that offers school choice first, so as to rigorously evaluate what effect this variation in NCLB rules has on student achievement.

Our proposed requirement for a randomized evaluation would not apply to waivers in NCLB accountability rules that serve purposes other than testing new approaches to accountability, are minor rule variations unlikely to have important effects on student achievement, cannot feasibly be evaluated in a randomized controlled trial, and/or are a response to local or regional emergencies. Examples include the waivers that the Department granted to certain schools and districts that enrolled large numbers of students displaced by Hurricane Katrina, allowing them to calculate Adequate Yearly Progress in a way that would not penalize them for accepting these students.

The precedent from welfare policy: Such waiver policies can greatly accelerate the development of valid knowledge about what works, and have a major impact on program effectiveness.

Specifically, from the Reagan through the Clinton Administrations, the U.S. Department of Health and Human Services (HHS) had in place a "demonstration waiver" policy. Under this policy, HHS waived certain provisions of federal law to allow state grantees to test new welfare reform approaches, but only if the grantees agreed to evaluate their reforms in randomized controlled trials. This policy directly resulted in more than 20 large-scale randomized controlled trials of welfare reform programs from the mid-80s through the mid-90s.

These trials—along with those that HHS funded directly—built valuable, scientifically-valid knowledge about what works in moving people from welfare to work. Of particular value, they showed conclusively that welfare reform programs that emphasized short-term job-search assistance and training, and encouraged participants to find work quickly, had larger effects on employment, earnings, and welfare dependence than programs that emphasized basic education. The work-focused programs were also much less costly to operate. This knowledge was a key to the political consensus behind the 1988 welfare reform act and helped shape the major 1996 welfare reform act including its strong work requirements. These legislative changes led to dramatic changes in state and federal programs, resulting in major reductions in welfare rolls and gains in employment among low-income Americans.

Conclusion: A similar waiver-demonstration policy in education could help supply a critical missing piece that states, districts, and schools need to achieve the goals of NCLB—namely, scientifically-valid knowledge about which educational interventions, and accountability rules, are truly effective in improving student educational achievement.