September 2010

From the National Center for Education Evaluation and Regional Assistance (NCEE)

WWC to Increase Scope of Research Reviews

The What Works Clearinghouse (WWC) is expanding its research design standards to cover single case studies and regression discontinuity designs. Both sets of new standards reflect IES's desire to expand coverage in the clearinghouse of credible evidence on the effectiveness of programs, policies, and practices as much as possible.

Single case studies are used heavily in special education research to evaluate the effectiveness of interventions for low-incidence population groups, such as blind students. In this type of study, an intervention participant acts as his/her own control, with the treatment being applied and withdrawn, repeatedly. Throughout the study, researchers make continuous assessments of an individual and use that data to gauge whether a specific intervention was effective.

The addition of the regression discontinuity design standards allows WWC to assess the evidence from studies that capitalize on situations in which decisions about who gets an intervention are based on their score on a continuous criterion variable relative to a threshhold. For example, students scoring below the 30th percentile on the state reading test might be assigned to mandatory summer school. In a regression discontinuity study, impacts are estimated by comparing outcomes for cases just above and just below the threshold.

Early experience of the WWC applying these standards suggests that a majority of existing studies using each type of design may not meet standards. However, as the WWC gains experience applying these new standards and the researchers using the standards become familiar with the standards and their application, we expect that the quality of and reporting on studies using these designs to improve.

Visit the WWC website to learn more about these new study design standards.

States, Schools Report on Usefulness of What Works Clearinghouse

A recent Government Accountability Office report recommending some improvements to the What Works Clearinghouse provided some perspective on state and school district awareness of WWC activities.

The report, Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse, contained findings from a survey of state education agencies and school districts. They were reported in a review GAO recently presented to the Department of Education.

Established by the Institute of Education Sciences in 2002, the What Works Clearinghouse acts as a source for scientific evidence in education. The WWC produces user-friendly practice guides for educators, assesses the rigor of research evidence on the effectiveness of interventions such as curricula and other programs, and develops and implements standards for education research.

The findings around awareness and use include:

• Some 42 percent of school districts and 33 out of 38 state education agencies were familiar with the work of WWC.
• About 72 percent of the districts that have used WWC reports (practice guides, intervention reports and quick reviews) reported that these products informed their decisions a small or moderate amount, while 18 percent of districts reported a large or very large impact on their decision making.
• Intervention reports were the most popular among schools. About 21 percent of districts used these reports to a large or very large extent, compared to 10 percent who used the practice guides in this extensive way.
• Both states and districts reported they would use the Clearinghouse more often if it reviewed more programs used in their local area, produced reports showing positive effects, and provided more timely information.

The key recommendations in the study have already been implemented or are in the process of being implemented. IES hopes to have all aspects of these recommendations fully implemented early in fiscal year 2011.