Program Announcement: Evaluation of State and Local Education Programs and Policies CFDA 84.305E

Program Officer:
Dr. Allen Ruby
Allen.Ruby@ed.gov
(202) 219-1591

Purpose

Through the research program in Evaluation of State and Local Education Programs and Policies (State/Local Evaluation), the Institute will provide support for rigorous evaluations of education programs or policies that are implemented by state or local education agencies.

Background

Educating children and youth to become productive, contributing members of the society is arguably one of the most important responsibilities of any community. Across our nation, school and district leaders and staff, along with state and national decision-makers, are working hard to strengthen the education of our young people. The Institute believes that improving education depends in large part on using evidence generated from rigorous research to make education decisions. However, education practice in our nation has not benefited greatly from research.

One striking fact is that the complex world of education—unlike defense, health care, or industrial production—does not rest on a strong research base. In no other field are personal experience and ideology so frequently relied on to make policy choices, and in no other field is the research base so inadequate and little used. (National Research Council, 1999, p. 1)

The Institute recognizes that evidence-based answers for all of the decisions that education decision-makers and practitioners must make every day do not yet exist. Furthermore, education leaders cannot always wait for scientists to provide answers. One solution for this dilemma is for the education system to integrate rigorous evaluation into the core of its activities. The Institute believes that the education system needs to be at the forefront of a learning society — a society that plans and invests in learning how to improve its education programs by turning to rigorous evidence when it is available, and by insisting that when we cannot wait for evidence of effectiveness that the program or policy we decide to implement be evaluated as part of the implementation.

In evaluations of the effectiveness of education interventions, one group typically receives the target intervention (i.e., treatment condition), and another group serves as the comparison or control group. In education evaluations, individuals in the comparison group almost always receive some kind of treatment; rarely is the comparison group a "no-treatment" control. When a state or district implements a new program for which there is little or no rigorous evidence of the effectiveness of the intervention, the education decision-makers are, in essence, hypothesizing that the new program is better than the existing practice (sometimes referred to as "business-as-usual") for improving student outcomes. Is this a valid hypothesis or assumption? Maybe, but maybe not. The only way to be certain is to embed a rigorous evaluation into the implementation of the new program.

Making rigorous evaluation of programs a standard education practice will enable educators to improve specific programs and ultimately lead to higher quality education programs in general. Through rigorous evaluations of education programs and practices, we can distinguish between those programs that produce the desired outcomes and those that don't; identify the particular groups (e.g., types of students, teachers, or schools) for which a program works; and determine which aspects of programs need to be modified in order to achieve the desired outcomes. For example, rigorous evaluations have shown that Check & Connect, a dropout prevention program, reduces dropout rates (Sinclair, et al., 1998; Sinclair et al., 2005). On the Institute's What Works Clearinghouse website (http://whatworks.ed.gov), readers will find reports on the effects of over 170 education interventions.¹ The intervention reports are based on findings from rigorous evaluations, and many of these reports record positive impacts on student outcomes.

Determining which programs produce positive effects is essential for improving education. However, the Institute also believes that it is important to discover when programs do not produce the desired outcomes. Over the past five years, the Institute has found that when the effectiveness of education programs and policies is compared to business-as-usual or other practices in rigorous evaluations, the difference in student outcomes between participants receiving the intervention and those in the comparison group is sometimes negligible (e.g., Dynarski, et al., 2007; Dynarski, et al., 2004; Ricciuti, et al., 2004; Wolf, et al., 2007).

States and districts can use the results of rigorous evaluations to identify and maintain successful policies and programs while redesigning or terminating ineffective ones, thereby making the best use of their resources. Rigorous evaluations also can identify ways to improve successful interventions. For example, the evaluation of the federal Early Reading First program to improve preschool children's literacy and language skills found positive impacts on students' print and letter knowledge and none of the feared negative impacts on social-emotional skills. In addition, it also identified the need for greater attention on improving children's oral language and phonological awareness.²

If "new" is not necessarily "better," and "good" programs could become even more effective, then it behooves us to evaluate the effects of programs on their intended outcomes (e.g., math achievement, graduation completion rates) when the new programs are implemented. Only appropriate empirical evaluation can sift the wheat from the chaff and identify those programs that do in fact improve student outcomes. The Institute believes that substantial improvements in student outcomes can be achieved if state and local education agencies rigorously evaluate their education programs and policies. To this end, the Institute will provide resources to conduct rigorous evaluations of state and local education programs and policies.

References

Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Means, B., Murphy, R., Penuel, W., Javitz, H., Emery, D., & Sussex, W. (2007). Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office.

Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., & Mansfield, W. (2004). When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program: New Findings. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office.

National Research Council. (1999). Improving Student Learning: A Strategic Plan for Education Research and Its Utilization. Committee on a Feasibility Study for a Strategic Education Research Program, Commission on Behavioral and Social Sciences and Education, National Research Council. Washington, DC: National Academy Press.

Ricciuti, A.E., St.Pierre, R.G., Lee, W., Parsad, A., & Rimdzius, T. (2004). Third National Even Start Evaluation: Follow-Up Findings From the Experimental Design Study. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office.

Sinclair, M. F., Christenson, S. L., Evelo, D. L., & Hurley, C. M. (1998). Dropout prevention for youth with disabilities: Efficacy of a sustained school engagement procedure. Exceptional Children, 65(1), 7–21.

Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional Children, 71(4), 465–482.

Wolf, P., Gutmann, B., Puma, M., Rizzo, L., Eissa, N., & Silverberg, M. (2007). Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year. NCEE-2007-4009. U.S. Department of Education, Institute of Education Sciences. National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office.