2011 National Board for Education Sciences Annual Report
National Center for Education Evaluation and Regional Assistance

The work of NCEE falls into four broad categories:

Conducting evaluations of education programs, with a particular focus on assessing the impacts on student achievement;

Translating and disseminating evidence to policymakers and practitioners regarding the effectiveness of strategies for achieving education goals;

Providing resources and technical guidance to improve the quality, efficiency, and dissemination of education evaluations; and

Providing technical support and assistance to state and local education agencies and evaluators of federally supported education initiatives.

NCEE oversaw 37 major evaluation and technical assistance contracts during the year (Appendix C-1), in addition to overseeing the 10 Regional Educational Laboratory contracts, the What Works Clearinghouse, and the National Library of Education. These contracts include congressionally mandated studies of federally funded programs as well as evaluations of non-federally funded programs. In addition to studies conducted by NCEE's evaluation division, each of the Regional Educational Laboratories (RELs) also conducted one or more evaluations of an education intervention. In keeping with current policy interests, a large share of NCEE's studies addressed issues related to literacy, teacher quality, and students with disabilities.

Evaluations

A total of 15 evaluation reports were released during the year, including three studies that were conducted under the Regional Education Laboratory contracts described below (Appendix C-2). The following are highlights of the findings from three of these evaluations—a study of an early elementary math curricula, a study of middle school mathematics professional development, and a study of mandatory random drug testing—each of which involved a large-scale, randomized controlled trial:

The evaluation of Early Elementary School Math Curricula examined whether some curricula are more effective than others at improving math achievement in schools serving a high percentage of disadvantaged students. Understanding the relative effectiveness of math curricula is important because curricula tend to be aligned with particular strategies for teaching math, yet little rigorous evidence exists to support one approach over another. The study compared the student math achievement effects of four distinct math curricula that represent several of the diverse approaches used to teach elementary school math in the United States—Investigations in Number Data, and Space; Math Expressions; Saxon Math; and Scott Foresman-Addison Wesley Mathematics (SFAW). First- and second-grade math achievement differed among students in volunteer schools that were assigned at random to use one of these four curricula. Among first-graders, the results favored Math Expressions over both Investigations and SFAW, but not over Saxon⁹. Among second-graders, the results favored Math Expressions and Saxon over SFAW, but not over Investigations.

The evaluation of Middle School Mathematics Professional Development examined the impact of intensive mathematics professional development (PD) on teachers' knowledge and teaching skills for seventh grade mathematics in rational number topics such as fractions, decimals, percentages, ratios, and proportions. Experts argue that professional development for teachers, particularly in Science, Technology, Engineering, and Mathematics (STEM) areas, is an important strategy to improve schools through increasing teachers' knowledge and skills. However, there is limited evidence about effective professional development activities. This study randomly assigned schools in 12 districts to either receive intensive PD activities or to receive only the PD activities normally provided by the district. The study then compared teacher knowledge of mathematics and the math achievement of their students in schools that did and did not receive intensive PD. The intensive PD intervention included over 100 hours of support for teachers over a two-year period in the form of summer institutes, seminars, and in-school coaching. In six of the districts, teachers in the intensive PD condition received training from Pearson Achievement Solutions. In the other six districts, teachers in the intensive PD condition received training from America's Choice. The intensive PD intervention was implemented as planned. However, due to high teacher turnover, on average, teachers in the treatment group received much less professional development than was intended. There was no evidence that the intensive PD resulted in improved teacher knowledge or led to improved student achievement on rational numbers topics.

The evaluation of Mandatory Random Drug Testing assessed the effectiveness of this approach for reducing substance use among high school students. In schools that were randomly assigned to implement this policy, students and their parents agreed to students being tested for drug use (and in some cases, tobacco or alcohol use) on a random basis as a condition of participation in athletic or other school-sponsored competitive extracurricular activities. The study found that, over the course of a single year, students involved in those activities and subject to in-school drug testing reported less substance use than comparable students in the high schools that were randomly assigned to not conduct such drug testing. However, there were no statistically significant differences in reported intentions to use drugs in the future and no "spillover effects" on students who were not subject to drug testing (e.g., through peer effects). There were also no impacts on student participation in activities subject to drug testing (which could have confounded the key results), or on students' connection to or attitudes towards school. Sensitivity testing to examine possible reporting bias found no such evidence.

Translational and Dissemination Work

NCEE's translational work is intended to "translate" education research findings for audiences of non-researchers. The bulk of the Center's translational and dissemination work falls under the Regional Educational Laboratories, the National Library of Education (NLE) including the Education Resources Information Center (ERIC), and the What Works Clearinghouse (WWC).

Part of NLE's mission is to respond to public inquiries. In 2010, about 71 percent of these inquiries came directly from the K–12 education community and approximately 12 percent were referrals from other IES programs, such as the Regional Educational Laboratories Virtual Reference Desk and ERIC Help Desk. The NLE also operates the Education Resources Information Center (ERIC) and the Department of Education's research library, providing information services to the general public, including educators and scholars, agency staff and contractors, other government agencies, and other libraries. ERIC is the world's largest education library, featuring an electronic collection of more than 1.3 million bibliographic records from 1966 to the present, along with about 332,000 full-text documents.¹⁰ The ERIC website expands the outreach of the RELs and WWC by providing a customized RSS feed of their latest materials.

While ERIC maintains and disseminates education journal articles, contractor reports, and conference papers, the What Works Clearinghouse focuses on translational and dissemination activities related specifically to the effectiveness of education programs, policies, and practices. During 2010, the WWC released 37 intervention studies. Of these studies, 11 focused on strategies for improving outcomes for students with disabilities, 10 addressed strategies for improving literacy among adolescents, five focused on high school math, four focused on English language learners, and the remainder examined various other topics (Appendix C-3).

The WWC also issued 21 Quick Reviews and two Practice Guides. Quick Reviews provide education practitioners and policymakers with timely and objective assessments of the quality of the research evidence in recently released research papers and reports whose public release is reported in a major national news source. Among the Quick Reviews conducted this past year, several addressed high profile initiatives, including reviews of five studies of school choice and charters; reviews of four studies of teacher incentives and performance initiatives; and reviews of two studies of financial incentives for students and/or their parents (Appendix C-4).

The WWC's Practice Guides build on the available evidence about a particular educational goal and develop concrete guidance for practitioners seeking to address that goal. This past year, the What Works Clearinghouse released Practice Guides on Improving Reading Comprehension in Kindergarten Through 3rd Grade and Developing Effective Fractions Instruction for Kindergarten Through 8th Grade, bringing the total number of Practice Guides available to 14 (http://ies.ed.gov/ncee/wwc/publications/practiceguides/).

Research Support

Research to support improvements in the quality of education evaluation is carried out primarily through NCEE's funding of commissioned technical methods papers, which are made available to the public on the IES website. This year, two methods reports were released: one on error rates in measuring teacher and school performance based on test score gains and another examining the precision gains from using publicly available school-level pretest scores (http://ies.ed.gov/ncee/pubs/20114003/index.asp and http://ies.ed.gov/ncee/pubs/20104004/).

Regional Educational Laboratories

A major initiative within NCEE is the provision of applied research and evaluation support to the regions through the Regional Educational Laboratories (RELs). As established by law, there are 10 RELs, each operating under contract to NCEE and serving a particular region of the country. Beginning in 2006, the REL program invested in more than 20 multi-year evaluations to determine the effectiveness of specific interventions on student outcomes. The RELs have released a total of seven evaluation reports to date, five of them during the past year (see Appendix C-2).

In addition, the RELs have continued to produce empirical analyses on a wide range of topics, including teacher and administrator characteristics and correlates of student achievement. These studies include some with national relevance that are reviewed by panels of outside scholars as well as "quick turnaround" empirical analyses and literature reviews that are produced for just-in-time use by states and districts (Appendix C-2).

The RELs serve as a bridge between the world of researchers and the world of education practitioners. In addition to producing original applied research, the RELs translate research for policymakers and practitioners through research-to-practice forums, or "Bridge Events." These are typically full-day events with formal presentations by researchers and practitioners, as well as group activities designed to promote participants' ability to apply the knowledge in their work settings (Appendix C-6). A vast majority (about 70) of these events have been tied to What Works Clearinghouse Practice Guides.¹¹

Program Highlights

A major activity for NCEE staff over the past year has been planning for 10 new Regional Education Laboratory contracts by the end of 2011. In addition, staff members have embarked on major initiatives to improve the reach and usefulness of the What Works Clearinghouse and to improve access to research reports and other products of the Center, including restricted-use data files.

New Regional Education Laboratory Contracts

The priority of the Regional Educational Laboratories is to help states and districts systematically use data and analysis to address important issues of policy and practice with the goal of improving student outcomes. Each REL is expected to build research capacity and a knowledge base in states and districts through various means, including:

• Assisting states, districts, and schools in using their data systems;
• Conducting and supporting high quality research and evaluation that focuses on a few key topics related to regional and national needs expressed in the region; and
• Helping education policymakers and practitioners incorporate data-based inquiry practices into regular decision making.

The expectation is that RELs will achieve these goals by working with new or existing partnerships of practitioners, policymakers, and others, referred to as "research alliances." These alliances may include regional, cross-state, and cross-district research partnerships. Both the REL program priorities and the mechanism for delivering REL services are intended to yield several important benefits to the regions and to the nation, including:

• Development of a cohesive and potentially deep body of knowledge in core priority topics that address regional and, often, national needs, rather than spreading REL work thinly over many issues;
• Increased use of evaluation, data, and analysis by educators and education policymakers to identify problems, choose programs and strategies, and learn from initiatives;
• Completion of a range of rigorous evaluation and research studies, methodologically appropriate to the questions the studies attempt to answer;
• Expansion of the capacity of states and localities to use their own data, conduct high quality research and evaluation, and appropriately incorporate findings into policy and practice;
• Distribution of REL work across each region through a transparent and equitable process for determining where REL assistance is applied; and
• Establishment of strong partnerships among practitioners, policymakers, and researchers that are not dependent on ongoing REL support.

Top

---

⁹ These first grade results become statistically insignificant if adjusted for multiple comparisons.
¹⁰ In 2010, ERIC was searched more than 13 million times per month through Internet search engines, the public website at http://www.eric.ed.gov, and commercial database vendors. Articles in the ERIC digital library are seen by individuals searching for education-related materials through Google, EBSCO, ProQuest, and various state and local education networks that provide access to ERIC.
¹¹ Since 2007, the I has published 12 Practice Guides. Currently, there are several guides in process.