2011 National Board for Education Sciences Annual Report

The National Board for Education Sciences (NBES, or the Board) oversees the primary functions of the Institute of Education Sciences (IES, or the Institute). The Board is designed to include representatives of both the research community and the general policy community. The Board's design calls for rotating membership, with presidential appointments and Senate confirmation. The 15 members provide impartial advice on the key operations of IES along with fulfilling statutory requirements for review and approval of Institute activities.

The Board's ability to perform its role depends upon a nomination and confirmation process that ensures the regular appointment of highly qualified members. During 2010 and 2011, NBES membership dipped to low levels due to the completion of members' appointed terms. This diminished membership threatened the full functioning of the Board. We urge continual vigilance to ensure that the Board operates at its congressionally authorized level.

Since the Institute of Education Sciences was established by Congress in 2002, it has made remarkable progress toward a goal that many of us on the Board and in the larger policy community had hoped for: building a sizable body of scientifically valid evidence about "what works" in education. For example, IES has sponsored more than 20 large randomized field experiments to assess the effectiveness of roughly 100 different educational programs, practices, and strategies ("interventions"). Because of these and other IES-funded studies, we now have—for the first time—convincing evidence about the impact of these interventions on key educational outcomes.

Of particular importance, we now have several examples of interventions that produce sizable positive effects—such as a college financial aid simplification program for low- and moderate-income families with children approaching college age. A large, multi-site field trial, co-funded by IES and other government and philanthropic funders, found that the program increased college enrollment the following fall by more than 25 percent. Such evidence has been a critical missing piece that policy officials and practitioners need in order to improve educational and life outcomes for U.S. children. Given IES's scientific expertise and independence, we believe it is uniquely positioned to provide this essential evidence, and thereby help bring sustained progress to American education.

—Jon Baron

Background

The Board consists of 15 voting members appointed by the President of the United States, by and with the advice and consent of the U.S. Senate. Serving as nonvoting ex officio members are the Director of the Institute, each of the four Commissioners of the National Education Centers, the Director of the National Institute of Child Health and Human Development, the Director of the Census, the Commissioner of Labor Statistics, and the Director of the National Science Foundation. As shown in Appendix A, the Board is currently operating with 10 appointed members, including three who were nominated by President Barack Obama and confirmed by the Senate in 2011:

Anthony S. Bryk is the president of the Carnegie Foundation for the Advancement of Teaching at Stanford University and the cofounder of the Consortium on Chicago School Research.

Kris D. Gutiérrez is a professor in the School of Education at the University of Colorado at Boulder and the president of the American Educational Research Association.

Robert A. Underwood is the president of the University of Guam and a former delegate from that territory to the U.S. House of Representatives.

The Board met in September 2010 and November 2010 under the leadership of Board chair Eric Hanushek and vice chair Jon Baron. At the November 2010 meeting, the Board elected Jon Baron chair and Bridget Terry Long vice chair. The Board met under their leadership in March 2011. Since the Board's inception in 2004, it has approved 12 resolutions to serve as guidance for Congress, the Secretary of Education, and the Director of IES (see Appendix B).

The overall mission of the Institute is to expand fundamental knowledge and understanding of education and to provide education leaders and practitioners, parents and students, researchers, and the general public with unbiased, reliable, and useful information about the condition and progress of education in the United States; about education policies, programs, and practices that support learning and improve academic achievement and access to educational opportunities for all students; and about the effectiveness of federal and other education programs. The Institute seeks to understand causal linkages to the greatest extent possible by conducting or sponsoring rigorous studies that support such inferences.

The work of the Institute is also grounded in the principle that effective education research must address the interests and needs of education practitioners and policymakers, as well as students, parents, and community members. To this end, the Institute will encourage researchers to develop partnerships with stakeholder groups to advance the relevance of the Institute's work, the accessibility of its reports, and the usability of its findings for the day-to-day work of education practitioners and policymakers. Further, the Institute will seek to increase the capacity of education policymakers and practitioners to use the knowledge generated from high-quality data analysis, research, and evaluation through a wide variety of communication and outreach strategies.

The Institute's priorities address a broad range of education-related outcomes for all students, including those with disabilities. These outcomes include: developmental outcomes for infants and toddlers with disabilities; school readiness; learning, achievement and higher-order thinking in reading and writing, mathematics, and the sciences from preschool through postsecondary schooling; behaviors, skills, and dispositions that support learning in school and later success in the workforce; and educational attainment in postsecondary, vocational, and adult education.

Within these areas, the work of the Institute is to compile statistics, support research, conduct evaluations, and promote and facilitate the use of scientific evidence. The Institute will sponsor work to: examine the state of education in the United States; develop and evaluate innovative approaches to improving education outcomes; understand the characteristics of high-quality teaching and how better to train current and prospective teachers and to recruit, support, and retain effective teachers; understand the processes of schooling through which education policies, programs, and practices affect student outcomes; and understand classroom, school, and other social contextual factors that moderate the effects of education practices and contribute to their successful implementation and sustainability.

In doing so, the Institute will seek to identify education policies, programs, and practices that improve education outcomes, and to determine how, why, for whom, and under what conditions they are effective. The Institute will promote research to improve education outcomes for all students, and particularly for those students whose prospects have historically been hindered because of their socioeconomic status, race/ethnicity, gender, disability, limited English proficiency, and/or residential or school mobility. Such research will generate knowledge to assist educators and policymakers in assessing and improving the equity of the education system. In addition to supporting new research, the Institute will promote the synthesis and dissemination of existing and ongoing research to construct coherent bodies of scientific knowledge about education.

The Institute will maintain rigorous scientific standards for the technical quality of its statistics, research, and evaluation activities, ensuring that the methods applied are appropriate to the questions asked and the results are valid and reliable. The work of the Institute will include a variety of research and statistical methods. The Institute will support the development and use of improved research methods to address a wide range of questions relevant to education; improved measures of a broad range of education processes, systems, and outcomes; and improved analytical approaches for designing and conducting education research particularly regarding classroom processes, and the enactment and implementation of programs. Where needed, the Institute will develop and publish rigorous technical standards for these methods. The Institute will ensure the quality and objectivity of its work by submitting all Institute products to rigorous scientific review. The Institute will build the capacity of the education research community by supporting post-doctoral and interdisciplinary doctoral training in the education sciences, providing training to equip education researchers with the skills to conduct rigorous research and effectively engage stakeholders in that research, and conducting training in research design and methods and in the use of longitudinal data.

Rebecca Maynard became Commissioner for the National Center for Education Evaluation and Regional Assistance on June 14, 2010. (Although this occurred in the last reporting period, the appointment was not announced in the 2010 Annual Report.) Dr. Maynard served as University Trustee Chair Professor of Education and Social Policy at the University of Pennsylvania. For the previous five years, she directed the University's Predoctoral Training Program in Education Sciences, an IES-funded program that supports 26 Ph.D. students annually from Arts and Sciences, Business, and Education. She helped develop the What Works Clearinghouse, and her work was instrumental in the creation of the Campbell Collaboration, an international association of public policy professionals who work to solve societal problems through scientific research and analysis. Dr. Maynard has published numerous academic pieces over the course of her career, including a recent update of her edited volume, Kids Having Kids:The Economic Costs and Social Consequences of Teen Pregnancy. Before joining the faculty of the University of Pennsylvania, she spent 18 years at Mathematica Policy Research Inc., where she served as senior vice president. Dr. Maynard earned her B.A. in economics from the University of Connecticut and her Ph.D. in economics from the University of Wisconsin.

Jack Buckley was confirmed by the U.S. Senate as Commissioner for the National Center for Education Statistics (NCES) on December 22, 2010. He began work on January 3, 2011. Dr. Buckley was an associate professor of applied statistics at New York University. He also served as Deputy Commissioner of NCES from 2006 to 2008 under former commissioner Mark Schneider. Dr. Buckley was an affiliated researcher with the National Center for the Study of the Privatization in Education at Teachers College, Columbia University, and in 2007 he published a book with Schneider entitled Charter Schools: Hope or Hype? He has served as an adjunct assistant professor at Georgetown University, an assistant professor at Boston College, and an instructor at the State University of New York at Stony Brook. Dr. Buckley spent five years in the U.S. Navy as a surface warfare officer and nuclear reactor engineer, and he also worked as an analytic methodologist at the Central Intelligence Agency. Dr. Buckley earned his bachelor's degree in government from Harvard University, and holds a doctorate and master's degree in political science, with a focus on statistical methodology and public policy, from the State University of New York at Stony Brook.

Ruth Curran Neild became Associate Commissioner for Knowledge Utilization at the National Center for Education Evaluation and Regional Assistance on January 31, 2011. Her scholarly interests focus on the transition to ninth grade; high school graduation and dropouts; high school reform; high school choice; and teacher quality. Much of her work has involved analyses of longitudinal administrative data sets from school districts and data merged across agencies. Prior to joining IES, Dr. Neild was a research scientist at the Center for Social Organization of Schools at Johns Hopkins University. There, she was the Co-Principal Investigator of a randomized trial of two curricula for helping high school freshmen who are underprepared in mathematics to succeed in Algebra 1. Her publications have appeared in peer-reviewed journals, in popular journals for practitioners, and as broadly disseminated research reports. Dr. Neild earned her bachelor's degree in history and sociology from Bryn Mawr College and her Ph.D. in sociology from the University of Pennsylvania.

* Estimates as of May 15, 2011. Amounts could change slightly.

Research Grants Between July 2010 and May 2011, the Standards and Review Office (SRO) handled the processing and scientific peer review of applications to the Institute's FY 2011 research competitions. Across the competitions, 1,254 applications were scientifically reviewed by 27 review panels comprising 510 external reviewers. In addition, SRO worked on plans for receiving, processing, and reviewing applications to the Institute's FY 2012 research competitions, with submission deadlines of June 23 and September 22, 2011.

Institute Reports During the period from July 2010 through May 10, 2011, SRO handled the scientific peer review of 95 reports from IES Centers. Of the 95 reports, 48 were from NCES, 35 from NCEE, 11 from NCSER, and 1 from NCER and NCSER, jointly.

Program Highlights

Redesign of Longitudinal Studies

The Center has incorporated redesign work into the development of its longitudinal studies to keep their content current with changes in the educational experiences of our children and youth. For example, in the Early Childhood Longitudinal Study: Kindergarten Class of 2010–11 (ECLS-K:2011), NCES developed a new reading assessment for children who speak Spanish, but not English. It is intended to address a problem identified in the first longitudinal kindergarten study in 1998–99, which could not capture information about these children's reading skills, making the development of growth models in reading a challenge.

Redesign has also been incorporated into NCES's newest secondary school longitudinal study. The design of the High School Longitudinal Study of 2009 (HSLS:09) is similar to past studies but with several innovations: all surveys and assessments were administered via computer; school counselor and parent input into students' decision-making about courses and postsecondary choices was and will continue to be collected; and the mathematics assessment developed specifically for this study focuses on algebra skills critical to success in secondary and postsecondary education.

NCES also has redesigned the timing and frequency of data collection in these longitudinal studies. The first longitudinal kindergarten study from 1998–99 has provided a wealth of useful information. However, because of budget constraints, the study had to skip the second and fourth grade data collections originally scheduled. As research has shown, the learning curves for children are steepest in the earliest years of their lives. The lack of data from the second and fourth grade years has limited analyses useful for understanding when children start to experience problems in school or when learning problems are ameliorated. To address this shortcoming, the new ECLS-K:2011 has collected data in the fall and spring of kindergarten, and the plans are to continue to collect in the spring of every year for cohort members from first through fifth grade. NCES also intends to include at least limited collections in the fall from first and second grades to better understand academic gains and losses associated with summer breaks and gains during a single academic year and how they relate to instruction and other aspects of schooling. NCES has also for the first time taken the assessment of science down to the kindergarten level in order to capture the beginning scientific knowledge and skills of young children.

For HSLS:09, NCES moved the first year of collection to the beginning of ninth grade, when most youth start high school. The most recent previous high school longitudinal study, the Education Longitudinal Study or ELS, began at the end of tenth grade with the goal of studying the transition from high school into college or the workforce. However, by starting at the end of tenth grade as opposed to the beginning of ninth grade, the study missed data for most of the first two years of this cohort's high school experiences, years that research suggests are critical to decisions about dropping out or pursuing further schooling. Moreover, in HSLS:09 students will be followed in eleventh grade and again directly after their intended high school graduation date to determine if and where they applied to postsecondary education, financial aid offers, acceptances, intended institution of matriculation, and immediate post-high school plans. This cohort will continue to be followed through their post secondary education and entry into the workforce.

At the same time, NCES is also working to establish new links between HSLS:09 and longitudinal administrative data from state data systems. The Center has worked with 10 states to include representative samples for these states in HSLS. NCES is currently working with these states to develop Memorandums of Understanding that will provide key information about the sample of students and schools included in HSLS from the states' administrative data systems. The combination of these state data with HSLS data will provide information about student academic achievement prior to ninth grade.

Working in collaboration with education economists, the postsecondary staff at NCES has put the Beginning Postsecondary Students Longitudinal Study (BPS) on a new conceptual footing, grounding its study of student persistence and degree attainment in human capital theory, and developing innovative instrumentation to elicit key study concepts, such as wage expectations and discount rates. Postsecondary, Adult, and Career Education (PACE) is also exploring opportunities to expand their collaboration with Federal Student Aid (FSA). NCES is initiating work to match historical federal financial aid records to existing longitudinal data sets. These matching opportunities will not only expand the usefulness of older data sets for researchers, but meet the need of FSA to understand the relationship between student aid and student outcomes.

NAEP

The National Assessment of Educational Progress (NAEP) is transitioning to eventual computer administration of all assessments for upcoming NAEP cycles. The first subject to go completely online was a national sample of eighth- and twelfth-graders who responded to writing prompts administered electronically in winter 2011. Electronic administration of a writing assessment provides a number of benefits. First, it eliminates the errors inherent in scorers trying to decipher illegible handwriting. Second, it captures information not previously available in group administration, such as the number of keystrokes, time spent on task, and number of revisions. Third, it allows students to use the mode which many now prefer for writing, the use of a keyboard. Fourth, it easily permits revisions without the need for pencil erasures. Fifth, it provides an opportunity for more engaging prompts. Sixth, it allows for easily incorporating universal design principles to accommodate student needs. Finally, it opens up the possibility of computer-aided intelligent scoring, which conceivably could make scoring much less expensive.

NCES piloted an adaptive version of the NAEP mathematics assessment in 2011. As a result of this tailoring of the assessment, more precise ability estimates can be obtained. Also, the computer delivery allows analysts to identify those items where the student was not engaged, based upon time taken to respond. NAEP is currently developing a computer-delivered Technology and Engineering Literacy (TEL) assessment. Computer delivery will be leveraged to offer innovative, interactive assessment tools allowing for meaningful realistic problems to be solved and offering a rich context for demonstration of TEL skills. Realistic problems create better evidence of student ability because TEL requires them to do a meaningful piece of work online. Computer delivery allows for more direct measures of TEL skills, new types of measures, a more engaging assessment environment, and multimodal information presentation.

International Studies

In international studies, NCES is conducting or facilitating linking studies to obtain greater efficiency and enhance the usefulness of its statistical portfolio. The intent is to learn more about how the international assessments relate to assessments used more regularly in U.S. schools and to enable states to benchmark their performance internationally without the cost and burden of fielding international assessments themselves. The largest effort links the National Assessment of Educational Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS) in mathematics and science at eighth grade in 2011. To create the link, during the NAEP administration window, a subsample of students was administered special booklets that contained both NAEP and TIMSS items. Likewise, during the TIMSS administration window, a subsample of students was also given special booklets. By analyzing the relationship between student performance on NAEP and TIMSS items, a linking function will be created that will allow the projection of TIMSS-like scores for all states. In addition to the national data collection for TIMSS, eight states were invited to participate in TIMSS with independent state representative samples to provide the data needed to validate the linking function.

Through NCES, the United States is connecting internationally by participating in the new international assessment of adult literacy, called the Program for the International Assessment of Adult Competencies (PIAAC). PIAAC is being fielded in 2011, and results will be released in 2013. PIAAC builds on previous work but is much larger than prior international assessments of adults in terms of the number and variety of countries. Some 26 countries are participating, and they include nearly all the advanced economies in the Organization for Economic Cooperation and Development, which coordinates the Program for International Student Assessment (PISA). PIAAC will also include adults' reports of the skills they use on the job. This component of the study is based on similar national work done in the United Kingdom and the United States, but is new to international assessments of adults and is being conducted in collaboration with the U.S. Department of Labor, as well as representatives of labor ministries internationally. PIAAC will be administered on computers, except in cases in which respondents are not familiar enough with computers to use them, or have literacy skills so limited that a paper-and-pencil assessment makes more sense for them. In addition, PIAAC will include an assessment of problem-solving in a technology rich environment, which is also new to adult assessments. One of the benefits of administering PIAAC on computers is the opportunity to adapt assessment items to responses in real-time—that is, to make the assessment easier or harder depending on the items each respondent is able to successfully complete.

NCES is also improving the accessibility of international data, including assessment results and additional contextual information about education systems around the world. In 2010, NCES launched the International Data Explorer, an online analysis tool on the NCES website that enables users to create and download their own tables and charts with international assessment results. And for the first time, in an effort to improve school participation rates in its international assessments, NCES provided participating schools with school-level reports on their performance relative to international and U.S. averages. This is another way of connecting with practitioners and improving their access to information they can use.

Higher Education Data and P-20W Data

NCES' Integrated Postsecondary Education Data (IPEDS) program has spent much of the last few years implementing data collection requirements in the Higher Education Opportunity Act of 2008. Several new data items, such as the net price of attendance after grant and scholarship aid, were collected and have been made available in online data tools, including College Navigator, a college search site for prospective students and parents. IPEDS is currently seeking approval from the Office of Management and Budget (OMB) to make additional data changes to future IPEDS collections, including data that distinguishes online enrollments from other enrollments and programs.

IES and the Department have spent approximately $514 million and invested considerable staff resources to help states establish or improve P-20W (early childhood through workforce) longitudinal data systems. In order to improve data quality within these state data systems, NCES has initiated a comprehensive Common Education Data Standards initiative to help define data definitions and standards from early childhood to postsecondary state data systems. NCES also launched two national programs in 2011 to assist states with their data system development: the Privacy Technical Assistance Center (PTAC) and the Educational Technical Assistance Program (EDTAP). These initiatives serve as a "one-stop" resource for states and districts to learn about best practices and receive expert technical assistance on all matters of data system development.

Statewide Longitudinal Data Systems

The development of statewide longitudinal data systems (SLDS), including their extensions into postsecondary education and the workforce, has focused attention on the need for high-quality, consistent guidance on issues of data stewardship, such as privacy, confidentiality, and data security. Requirements to protect personally identifiable information (PII) are delineated in the Family Educational Rights and Privacy Act (FERPA) and related regulations, other legislation, and guidance from OMB and the National Institute of Standards and Technology. Understanding and complying with privacy regulations can be a complex task for the organizations and individuals tasked with assimilating and using student-level data, especially as they balance these regulations with the goal of using the richness of the data to improve education at local and even individual levels.

NCES—in consultation with the Department of Education's Chief Privacy Officer, the Family Policy Compliance Office (which oversees FERPA), the Office of Planning, Evaluation, and Policy Development, and the Office of the General Counsel—is providing technical assistance for states and organizations that are engaged in building and using student-level longitudinal education data systems. NCES began this process with the release of three technical briefs in the last year on such topics as definitions and concepts; data stewardship and managing PII; and statistical methods for protecting PII in aggregate reports. The newly established Privacy Technical Assistance Center (PTAC) will extend efforts in the areas of privacy, confidentiality, and security by disseminating information, answering individual questions, conducting training and, as appropriate, referring questions to experts in the Department.

Program Highlights

New Research and Research Training Grants

The new FY 2011 awards cover a wide range of topics including research on mathematics and science education, behavioral interventions, early learning, English learners, education technology, teacher effectiveness, and school accountability. Below are four examples of new projects.

• Mathematical equivalence is an important concept in children's development of algebraic thinking. Although decades of research have shown that children struggle to understand this fundamental concept, researchers have yet to develop an intervention that produces mastery-level understanding. University of Notre Dame researchers are building on the success of their previous IES award to develop and pilot-test a comprehensive intervention to help elementary school children achieve mastery-level understanding of mathematical equivalence.
• With about 70 percent of fourth and eighth grade English learners scoring below the Basic level on the NAEP reading assessment in 2009, identifying effective approaches for teaching English to students who enter the school system with limited English skills remains a challenge. Researchers at the University of Houston and the University of Texas at Austin will evaluate the effectiveness of a first-grade supplemental reading intervention for English language learners with reading difficulties when it is implemented under routine practice conditions.
• Young adolescents' vulnerability to a range of academic, behavioral, and social problems associated with poor school adjustment can make the transition to middle school challenging. Researchers at Pennsylvania State University and the University of North Carolina at Chapel Hill are evaluating the efficacy of an intervention intended to support adolescents' successful transition to middle school in urban schools. This work is an extension of earlier research supported by IES to develop the intervention and evaluate its efficacy in rural schools.
• Very little is known about the effectiveness of education programs for gifted students. A collaborative effort between the University of California at Berkeley and Broward County Public Schools will evaluate the impact of the district's gifted education program on student achievement. The team will assess the effect of gifted education in early elementary grades on subsequent academic achievement and the impact of universal screening for gifted students (a strategy adopted to correct for the under-representation of Black and Hispanic students) on the size and composition of the gifted population.

In addition, NCER awarded funding for a new research and development center on postsecondary education. The National Center on Postsecondary Education and Employment will analyze state and national longitudinal data to identify the employment and earnings benefits of specific postsecondary educational pathways and awards. It will also examine the institutional programs and public policies that are associated with postsecondary completion and employment and earnings outcomes. The Center will analyze outcomes for a full range of undergraduate degrees and pathways but will focus on occupationally oriented educational pathways leading to credentials, particularly at the sub-baccalaureate level.

Activities of the National Research and Development (R&D) Centers

• At the fourth annual conference of the National Center for the Analysis of Longitudinal Data in Education Research, policy analysts and education researchers discussed the extent to which Race to the Top reforms matter for student achievement, teacher quality, and effective schools.
• At its 2010 conference, the National Center for Research on Early Childhood Education brought together early childhood researchers, policymakers, and practitioners to discuss the implementation and evaluation of teacher professional development interventions for early childhood educators.
• At its annual conference, the National Center for Performance Incentives released findings from its three-year evaluation of the impact on student achievement of teacher pay for performance. For this three-year evaluation, middle school math teachers were randomly assigned to a treatment or control group with treatment teachers eligible for bonuses of up to $15,000 per year based on student test score gains. The primary finding was that rewarding teachers with bonus pay, in the absence of other support programs, did not raise student test scores.

Reading for Understanding

NCER launched the Reading for Understanding Research Initiative last year to develop effective approaches for improving and assessing reading comprehension. All five intervention teams have been developing curriculum materials for their grade spans. Four of the five teams have completed initial versions of their curriculum materials, and two of the teams have completed teaching trials, in which classroom teachers have implemented these initial versions, and been engaged in the ongoing revision of these materials. The fifth team has completed a detailed mapping of learning trajectories in literature, history, and science, which is informing development of their secondary curricula intended to support comprehension. Simultaneously, teams are gathering cross-sectional and longitudinal data which will enable mapping of the development of cognitive, linguistic, and reading skills from pre-kindergarten through twelfth grade. Analysis of these data will be complete by the fall of 2011 and will feed directly into the revision of the intervention materials. Across the teams, cognitive researchers are examining how linguistic features of the learner and of the instructional texts shape observed learning outcomes. Finally, the assessment team has completed a draft assessment framework and is exploring the costs and benefits of different item formats that can be used to assess comprehension of complex texts.

Small Business Innovation Research (SBIR)

Through the SBIR program, NCER provides awards of up to $1.05 million to small businesses and their partners for the full-scale development of products that facilitate student learning and teacher efficiency, or for tools to improve education research. One measure of program success is commercialization of products developed under program auspices. In September 2010, Measured Progress, a New Hampshire-based not-for-profit specializing in the development of state- and district-level assessments, acquired Nimble Assessment Systems, a two-time recipient of SBIR awards. Measured Progress will incorporate Nimble Assessments Systems' computer-based testing tools—NimbleTools and NimblePad systems—into its product line.

Another SBIR awardee, Polyhedron Learning Media, Inc., developed a web-based virtual physics laboratory for use in introductory college physics courses. In a pilot study of the technology, students in introductory-level physics classes were randomly assigned to the virtual lab or to a traditional hands-on lab. Results suggest that students in the virtual labs are able to learn as much as students in the hands-on lab. In 2010, Brooks/Cole Cengage Learning signed a contract with Polyhedron for exclusive rights to distribute Virtual Physics Lab to colleges, universities, and Advanced Placement high school physics programs.

Quantum Simulations, another SBIR recipient, has developed a suite of web-based artificial intelligence (AI) tutors and assessments for chemistry and math education that serve students at the kindergarten through adult levels. The AI tutors support learning through an interactive process that poses questions, analyzes the students' answers, provides feedback, and adjusts to the student's level while answering the student's questions. Quantum has commercialized its tutors through direct sales, as well as distribution partnerships with textbook publishers such as McGraw-Hill and Holt, Rinehart and Winston.

Research Findings

Studies completed by NCER researchers in 2010–11 include:

Evaluation of Tennessee's Voluntary Pre-Kindergarten Program NCER's Evaluation of State and Local Education Programs and Policies program supports the evaluation of programs and policies that states or districts have implemented. Under this program, a team of researchers at Vanderbilt University is evaluating the impact of Tennessee's voluntary pre-kindergarten program on the school readiness of economically disadvantaged children and their subsequent academic performance. The project includes two studies. One is a randomized controlled trial of oversubscribed programs, which assign the limited available places in the program by lottery. The second study is a regression discontinuity study comparing children who are eligible for the program based on their age at the cut-off date to children who have to wait a year because they just missed the cut-off date. The data show substantial improvements on school readiness outcomes for children who have access to the state pre-kindergarten program.

Early Childhood Education Research conducted by the National Center for Research on Early Childhood Education indicates that many young children who are at risk for school failure attend classrooms of mediocre quality that do not maximize children's learning to the extent found in high-quality classrooms. For example, analyses of state-funded pre-kindergarten programs in 11 states with mature programs indicated that just over half the school day was spent on learning activities. Children in early childhood classrooms may participate in very few of the types of interactions that are associated with improved school readiness.

Perceptual Learning A team at the University of California, Los Angeles decided to exploit the potential of perceptual learning by developing computer-delivered modules, in which, for example, students are asked in 30-minute practice sessions to match multiple instances of different representations of the same equation (e.g., matching a number sentence to the correct word problem or graph). Students are asked only to identify the representations that are equivalent but not to solve the equations. This repeated exposure to examples, with feedback to the student regarding whether his or her match is correct, draws upon the human capacity to seek out structure. In an experiment with high school students, the perceptual learning module was found to substantially improve students' performance on mapping the relations between word problems, equations, and graphs.

Program Highlights

New Research and Research Training Grants

In FY 2011 over 350 applications to NCSER competitions were reviewed, resulting in 42 new research and postdoctoral research training awards totaling $89 million. These projects cover a broad range of issues related to improving outcomes for children with or at risk for disabilities, including autism spectrum disorders, developmental speech and/or language impairments, emotional and behavioral disorders, hearing impairments, intellectual disabilities, and learning disabilities. For example:

• Computer-based testing holds promise for increasing accessibility of state assessments for children with disabilities. However, providing appropriate accommodations presents a challenge, particularly for making computer-based test items accessible to students who are Braille readers. Researchers at the Educational Testing Service are developing enhancements to an existing platform that delivers an eighth-grade reading assessment. The enhancements are intended to allow students who are blind or have low vision to have access to the computerized test.
• Few evidence-based vocabulary interventions exist, particularly for students most at risk for language and learning disabilities. University of Connecticut researchers are conducting a randomized, controlled trial of an intensive vocabulary intervention designed to supplement classroom vocabulary instruction for kindergarten students who are at risk for language and learning disabilities.
• Current methods of reading instruction have not been highly effective for children with Down syndrome. A team of researchers at the University of Pittsburgh is developing an intervention for teaching reading to children with Down syndrome that incorporates components of early reading (e.g., decoding and fluency skills) that have been modified to support the challenges with expressive language, memory, and motivation that are often exhibited by children with Down syndrome.
• Longitudinal studies indicate that students who perform poorly at the end of kindergarten are likely to continue to perform poorly in mathematics through the later elementary grades. University of Oregon researchers are developing an iPad-based mathematics intervention for kindergarten students who are at risk for developing learning disabilities in mathematics. The KinderTEK intervention is being designed as an intervention that builds conceptual and procedural fluency with whole number concepts.
• Researchers at the University of Florida are conducting a randomized controlled trial to test the efficacy of BEST in CLASS, an intervention implemented by early childhood teachers to improve social, emotional, and behavioral functioning of preschool children at high risk for developing emotional and behavioral disorders. BEST in CLASS was developed with a previous IES Development and Innovation grant and has pilot data indicating the promise of the intervention for ameliorating problem behaviors in young children.
• There is a lack of empirical evidence on the relationship between physical therapy provided in schools and student outcomes. A research team at the University of Kentucky is conducting a multi-site observational study of 120 physical therapists and their students to examine the relationship between characteristics of school-based physical therapy and changes in student outcomes, including participation in school activities, self-care, posture and mobility, and recreation and fitness.

In addition, NCSER funded a new special education research and development center—the National Research and Development Center on Assessment and Accountability for Special Education. The increased demand for accountability in education and improved student academic performance has led to many questions about the most accurate method for capturing individual student progress, particularly for students with disabilities. Although a substantial amount of research exists about the characteristics of students with disabilities and about assessment of their abilities and skills for purposes of classification and intervention, far less is known about the natural developmental progression of achievement for students with disabilities. Under the auspices of this Center, researchers at the University of Oregon, the University of Arizona, and North Carolina State University have joined together to develop and test various approaches for measuring the reading and mathematics achievement growth of students with and without disabilities. They will examine alternative accountability models using student academic growth to evaluate schools' effectiveness in serving students with disabilities.

Reports

Over the past year, NCSER released four reports from its longitudinal studies and evaluations.

• Comparisons Across Time of the Outcomes of Youth With Disabilities up to 4 Years After High School (September 2010). This report uses data from the two National Longitudinal Transition Study datasets to provide comparison data across time (1990 to 2005) on a wide range of post-high school outcomes for youth with disabilities who had been out of high school up to four years. The outcomes cover several key areas, including: postsecondary education enrollment and educational experiences; employment status and characteristics of the youth's current or most recent job; productive engagement in school, work, or preparation for work; household circumstances, including residential independence, parenting and financial independence; and social and community involvement.
• Facts from NLTS2: The Secondary School Experiences and Academic Performance of Students With Hearing Impairments (February 2011). This report uses data from the National Longitudinal Transition Study-2 to provide a national picture of the secondary school experiences and academic achievement of students with hearing impairments who received special education services. The outcomes cover several key areas, including: students' experiences in general education academic courses and non-vocational special education courses, accommodations, supports, services provided to students, and academic achievement.
• Access to Educational and Community Activities for Young Children with Disabilities (October 2010). This report, based on the Pre-Elementary Education Longitudinal Study (PEELS), describes the kindergarten classroom and community experiences of young children with disabilities and examines how access and participation in these activities may vary by child, family, and school district characteristics.
• A Study of States' Monitoring and Improvement Practices under the Individuals with Disabilities Education Act (IDEA) (October 2010). This report completes NCSER's study of states' monitoring practices. It describes the nature and scope of states' Part B and Part C monitoring systems and finds substantial variability in states' Part B and Part C monitoring and improvement practices under IDEA. The report describes contextual factors that may affect states' monitoring systems, states' approaches to monitoring, and how states' monitoring systems and processes map onto a framework developed for the study.

Special Education Research and Development (R&D) Centers

The Center on Response to Intervention in Early Childhood hosted its Second Annual RTI in Early Childhood Summit in Kansas City in October 2010. Practitioners, researchers, policymakers, and higher education faculty gathered to discuss instructional strategies to promote successful readers and the importance of screening and progress monitoring for improving early language and literacy outcomes of children at risk for disabilities.

Researchers from the National R&D Center on Serious Behavior Disorders at the Secondary Level finalized development of an intervention package that includes multiple components designed to improve student behavior and academic performance. They are preparing for an efficacy trial in 54 schools beginning in the fall of 2011 and running through the spring of 2013. The research team has also actively presented its work at local and national conferences, discussing topics such as the Center's intervention development process, academic and behavioral needs of high school students with emotional and behavioral disorders, and results of supplemental studies (e.g., understanding external placement decision-making by school staff for students with emotional and behavioral disorders).

The National Mathematics Advisory Panel in its 2008 report asserted that high priority should be given to improving curricula and instruction for teaching fractions (including decimals and percents). This year a new center—the National Center on Mathematics Instruction for Students with Mathematics Difficulties—was launched. Its purpose is to increase knowledge of how children acquire or fail to acquire an understanding of rational numbers (i.e., fractions) and how children with math difficulties can be taught to understand and operate fluently with rational numbers.

Research Findings

Over the past five years, NCSER's researchers have begun expanding their knowledge and understanding of infants, toddlers, and children with disabilities. Three studies are highlighted below.

The success of intervention models hinges on an accurate determination of which children are at risk for future difficulty. Quick assessments are needed that accurately identify 1) which children are at risk and therefore could benefit from intervention and 2) which children are not at risk and do not need intervention. Historically, education researchers have not been successful at developing screening measures that are accurate predictors of risk for reading difficulties. IES has invested significant resources in the exploration of basic measurement issues surrounding universal screening procedures. In a series of studies, Vanderbilt University researchers developed a two-step screening procedure in which all first-graders are administered a single, brief measure of phonemic decoding, and only children who score within the risk range on this measure complete the longer screening battery. This procedure results in the identification of which children are at risk or not at risk for reading difficulties with 90 percent accuracy.

The prevalence of Autism Spectrum Disorders (ASD) has increased over the past decade, with the Centers for Disease Control reporting a current rate of 1 in 110 children. Despite increased research on ASD, few comprehensive interventions for children with ASD have been developed and tested. One exception is Learning Experiences—An Alternative Program for Preschoolers and Parents (LEAP), a comprehensive intervention for preschool children with ASD. A prior evaluation indicated that LEAP improves child outcomes relative to typical practice. With NCSER funding, LEAP developers recently completed a new experimental evaluation comparing full LEAP implementation to a reduced (and less expensive) model that provided the LEAP materials without training and mentoring by LEAP staff. They found that providing preschool teachers with training by LEAP staff resulted in greater fidelity of program implementation. Moreover, children in the full model condition demonstrated greater developmental gains in cognition and language, reduction in autism symptom severity, growth in social skills, and reduction in problem behavior compared to children in the manual-only group. Overall, the experiment demonstrates the efficacy of the LEAP model utilizing the full-scale training and mentoring for preschool staff.

Although a number of general-purpose measures of student social behavior have been used for behavior progress-monitoring, there is a need for widely-accepted, reliable, and validated tools that allow for frequent measurement because they are relatively easy and quick to administer. In 2006, University of Connecticut researchers received IES funding to develop and test the Direct Behavior Rating (DBR) scales as an assessment method that combined the strengths of systematic direct observation with behavior ratings scales. This team received another IES grant to evaluate DBR scales for use in supporting problem-solving models of service delivery for behavior screening as well as for progress monitoring. Researchers at Louisiana State University are also working on this problem and are developing a series of change-sensitive progress-monitoring tools called Brief Behavior Rating Scales (BBRS) that are intended to be efficient, practical, reliable, and valid. BBRS will be appropriate for classroom educators who need efficient and effective behavior progress-monitoring tools to monitor their students on a consistent basis.

NCEE oversaw 37 major evaluation and technical assistance contracts during the year (Appendix C-1), in addition to overseeing the 10 Regional Educational Laboratory contracts, the What Works Clearinghouse, and the National Library of Education. These contracts include congressionally mandated studies of federally funded programs as well as evaluations of non-federally funded programs. In addition to studies conducted by NCEE's evaluation division, each of the Regional Educational Laboratories (RELs) also conducted one or more evaluations of an education intervention. In keeping with current policy interests, a large share of NCEE's studies addressed issues related to literacy, teacher quality, and students with disabilities.

Evaluations

A total of 15 evaluation reports were released during the year, including three studies that were conducted under the Regional Education Laboratory contracts described below (Appendix C-2). The following are highlights of the findings from three of these evaluations—a study of an early elementary math curricula, a study of middle school mathematics professional development, and a study of mandatory random drug testing—each of which involved a large-scale, randomized controlled trial:

The evaluation of Early Elementary School Math Curricula examined whether some curricula are more effective than others at improving math achievement in schools serving a high percentage of disadvantaged students. Understanding the relative effectiveness of math curricula is important because curricula tend to be aligned with particular strategies for teaching math, yet little rigorous evidence exists to support one approach over another. The study compared the student math achievement effects of four distinct math curricula that represent several of the diverse approaches used to teach elementary school math in the United States—Investigations in Number Data, and Space; Math Expressions; Saxon Math; and Scott Foresman-Addison Wesley Mathematics (SFAW). First- and second-grade math achievement differed among students in volunteer schools that were assigned at random to use one of these four curricula. Among first-graders, the results favored Math Expressions over both Investigations and SFAW, but not over Saxon. Among second-graders, the results favored Math Expressions and Saxon over SFAW, but not over Investigations.

The evaluation of Middle School Mathematics Professional Development examined the impact of intensive mathematics professional development (PD) on teachers' knowledge and teaching skills for seventh grade mathematics in rational number topics such as fractions, decimals, percentages, ratios, and proportions. Experts argue that professional development for teachers, particularly in Science, Technology, Engineering, and Mathematics (STEM) areas, is an important strategy to improve schools through increasing teachers' knowledge and skills. However, there is limited evidence about effective professional development activities. This study randomly assigned schools in 12 districts to either receive intensive PD activities or to receive only the PD activities normally provided by the district. The study then compared teacher knowledge of mathematics and the math achievement of their students in schools that did and did not receive intensive PD. The intensive PD intervention included over 100 hours of support for teachers over a two-year period in the form of summer institutes, seminars, and in-school coaching. In six of the districts, teachers in the intensive PD condition received training from Pearson Achievement Solutions. In the other six districts, teachers in the intensive PD condition received training from America's Choice. The intensive PD intervention was implemented as planned. However, due to high teacher turnover, on average, teachers in the treatment group received much less professional development than was intended. There was no evidence that the intensive PD resulted in improved teacher knowledge or led to improved student achievement on rational numbers topics.

The evaluation of Mandatory Random Drug Testing assessed the effectiveness of this approach for reducing substance use among high school students. In schools that were randomly assigned to implement this policy, students and their parents agreed to students being tested for drug use (and in some cases, tobacco or alcohol use) on a random basis as a condition of participation in athletic or other school-sponsored competitive extracurricular activities. The study found that, over the course of a single year, students involved in those activities and subject to in-school drug testing reported less substance use than comparable students in the high schools that were randomly assigned to not conduct such drug testing. However, there were no statistically significant differences in reported intentions to use drugs in the future and no "spillover effects" on students who were not subject to drug testing (e.g., through peer effects). There were also no impacts on student participation in activities subject to drug testing (which could have confounded the key results), or on students' connection to or attitudes towards school. Sensitivity testing to examine possible reporting bias found no such evidence.

Translational and Dissemination Work

NCEE's translational work is intended to "translate" education research findings for audiences of non-researchers. The bulk of the Center's translational and dissemination work falls under the Regional Educational Laboratories, the National Library of Education (NLE) including the Education Resources Information Center (ERIC), and the What Works Clearinghouse (WWC).

Part of NLE's mission is to respond to public inquiries. In 2010, about 71 percent of these inquiries came directly from the K–12 education community and approximately 12 percent were referrals from other IES programs, such as the Regional Educational Laboratories Virtual Reference Desk and ERIC Help Desk. The NLE also operates the Education Resources Information Center (ERIC) and the Department of Education's research library, providing information services to the general public, including educators and scholars, agency staff and contractors, other government agencies, and other libraries. ERIC is the world's largest education library, featuring an electronic collection of more than 1.3 million bibliographic records from 1966 to the present, along with about 332,000 full-text documents. The ERIC website expands the outreach of the RELs and WWC by providing a customized RSS feed of their latest materials.

While ERIC maintains and disseminates education journal articles, contractor reports, and conference papers, the What Works Clearinghouse focuses on translational and dissemination activities related specifically to the effectiveness of education programs, policies, and practices. During 2010, the WWC released 37 intervention studies. Of these studies, 11 focused on strategies for improving outcomes for students with disabilities, 10 addressed strategies for improving literacy among adolescents, five focused on high school math, four focused on English language learners, and the remainder examined various other topics (Appendix C-3).

The WWC also issued 21 Quick Reviews and two Practice Guides. Quick Reviews provide education practitioners and policymakers with timely and objective assessments of the quality of the research evidence in recently released research papers and reports whose public release is reported in a major national news source. Among the Quick Reviews conducted this past year, several addressed high profile initiatives, including reviews of five studies of school choice and charters; reviews of four studies of teacher incentives and performance initiatives; and reviews of two studies of financial incentives for students and/or their parents (Appendix C-4).

The WWC's Practice Guides build on the available evidence about a particular educational goal and develop concrete guidance for practitioners seeking to address that goal. This past year, the What Works Clearinghouse released Practice Guides on Improving Reading Comprehension in Kindergarten Through 3rd Grade and Developing Effective Fractions Instruction for Kindergarten Through 8th Grade, bringing the total number of Practice Guides available to 14 (//https://ies.ed.gov/ncee/wwc/practiceguides).

Research Support

Research to support improvements in the quality of education evaluation is carried out primarily through NCEE's funding of commissioned technical methods papers, which are made available to the public on the IES website. This year, two methods reports were released: one on error rates in measuring teacher and school performance based on test score gains and another examining the precision gains from using publicly available school-level pretest scores 

Regional Educational Laboratories

A major initiative within NCEE is the provision of applied research and evaluation support to the regions through the Regional Educational Laboratories (RELs). As established by law, there are 10 RELs, each operating under contract to NCEE and serving a particular region of the country. Beginning in 2006, the REL program invested in more than 20 multi-year evaluations to determine the effectiveness of specific interventions on student outcomes. The RELs have released a total of seven evaluation reports to date, five of them during the past year (see Appendix C-2).

In addition, the RELs have continued to produce empirical analyses on a wide range of topics, including teacher and administrator characteristics and correlates of student achievement. These studies include some with national relevance that are reviewed by panels of outside scholars as well as "quick turnaround" empirical analyses and literature reviews that are produced for just-in-time use by states and districts (Appendix C-2).

The RELs serve as a bridge between the world of researchers and the world of education practitioners. In addition to producing original applied research, the RELs translate research for policymakers and practitioners through research-to-practice forums, or "Bridge Events." These are typically full-day events with formal presentations by researchers and practitioners, as well as group activities designed to promote participants' ability to apply the knowledge in their work settings (Appendix C-6). A vast majority (about 70) of these events have been tied to What Works Clearinghouse Practice Guides.

Program Highlights

A major activity for NCEE staff over the past year has been planning for 10 new Regional Education Laboratory contracts by the end of 2011. In addition, staff members have embarked on major initiatives to improve the reach and usefulness of the What Works Clearinghouse and to improve access to research reports and other products of the Center, including restricted-use data files.

New Regional Education Laboratory Contracts

The priority of the Regional Educational Laboratories is to help states and districts systematically use data and analysis to address important issues of policy and practice with the goal of improving student outcomes. Each REL is expected to build research capacity and a knowledge base in states and districts through various means, including:

• Assisting states, districts, and schools in using their data systems;
• Conducting and supporting high quality research and evaluation that focuses on a few key topics related to regional and national needs expressed in the region; and
• Helping education policymakers and practitioners incorporate data-based inquiry practices into regular decision making.

The expectation is that RELs will achieve these goals by working with new or existing partnerships of practitioners, policymakers, and others, referred to as "research alliances." These alliances may include regional, cross-state, and cross-district research partnerships. Both the REL program priorities and the mechanism for delivering REL services are intended to yield several important benefits to the regions and to the nation, including:

• Development of a cohesive and potentially deep body of knowledge in core priority topics that address regional and, often, national needs, rather than spreading REL work thinly over many issues;
• Increased use of evaluation, data, and analysis by educators and education policymakers to identify problems, choose programs and strategies, and learn from initiatives;
• Completion of a range of rigorous evaluation and research studies, methodologically appropriate to the questions the studies attempt to answer;
• Expansion of the capacity of states and localities to use their own data, conduct high quality research and evaluation, and appropriately incorporate findings into policy and practice;
• Distribution of REL work across each region through a transparent and equitable process for determining where REL assistance is applied; and
• Establishment of strong partnerships among practitioners, policymakers, and researchers that are not dependent on ongoing REL support.

Meetings

The Board met on September 29, 2010; November 1, 2010; and March 23, 2011. It is scheduled to meet again on June 29, 2010.

At its November 1, 2010 meeting, the Board approved the Director's proposed priorities for IES.

At its March 23 meeting, the Board unanimously approved the following two Board resolutions:

Resolutions

1. That Congress continue funding for the Regional Educational Laboratories at current levels as part of any Congressional spending agreement for FY 2011, and authorize the Institute of Education Sciences to extend the existing Laboratory contracts for one additional year beyond their scheduled completion date.
2. That Congress include the following reforms in the authorizing language of Education Department grant programs, wherever feasible and cost-effective, to advance the use of evidence of effectiveness in decision-making:
   • Funding incentives for grant applicants to use program models or strategies ("interventions") supported by evidence of effectiveness, as judged by IES standards such as those used in the Department's Investing in Innovation program;
   • Funding to evaluate previously untested but highly-promising interventions, through studies overseen by IES that allow for strong causal conclusions, including randomized controlled trials where appropriate; and
   • Funding incentives for state and local educational agencies to engage in systematic evaluation and improvement of local initiatives, consistent with evidence standards established by IES.

1. Congress, in authorizing and funding evaluations of federal education programs, should require [program] grantees, as a condition of grant award, to participate in the evaluation if asked, including the random assignment to intervention and control groups as appropriate. (April 2005)
2. Congress and the U.S. Department of Education should ensure that individual student data can be used by researchers (with appropriate safeguards for confidentiality) in order to provide evaluations and analyses to improve our schools. (September 2006)
3. Congress should designate the Institute of Education Sciences, in statute, as the lead agency for all congressionally authorized evaluations of U.S. Department of Education programs, responsible for all operations, contracts, and reports associated with such evaluations. (September 2006)
4. Congress should allow the U.S. Department of Education to pool funds generated by the 0.5 percent evaluation set-aside from smaller programs. (September 2006)
5. The U.S. Department of Education should use its "waiver" authority to build scientifically valid knowledge about what works in K–12 education. (September 2006)
6. Congress should create, in statute, effective incentives for federal education program grantees to adopt practices or strategies meeting the highest standard of evidence of sizeable, sustained effects on important educational outcomes. (May 2007)
7. Congress should revise the statutory definition of "scientifically based research" so that it includes studies likely to produce valid conclusions about a program's effectiveness, and excludes studies that often produce erroneous conclusions. (October 2007)
8. The Board will review and advise the IES Director on grant awards where the proposed grantee is selected out of rank order of applicant scores that result from peer review for scientific merit. (January 2008)
9. The Board commends the Secretary and the U.S. Department of Education for moving forward in developing new regulations and guidance about how to maintain confidentiality of educational data under the Family Educational Rights and Privacy Act (FERPA) while also providing for research uses of student and school data. The Department should finalize these regulations quickly, incorporating the major clarifications that have been submitted in comments. (May 2008)
10. Congress should expand on the program of supporting statewide longitudinal data systems by requiring that states accepting funding under this program agree to make data in these systems available to qualified researchers (subject to FERPA) for the purpose of research that is intended to help improve student achievement. (May 2008)
11. The Board recommends that Congress continue funding for the Regional Educational Laboratories at current levels as part of any Congressional spending agreement for FY 2011, and authorize the Institute of Education Sciences to extend the existing Laboratory contracts for one additional year beyond their scheduled completion date. (March 2011)
12. The Board recommends that Congress include the following reforms in the authorizing language of Education Department grant programs, wherever feasible and cost-effective, to advance the use of evidence of effectiveness in decision-making:
    • Funding incentives for grant applicants to use program models or strategies ("interventions") supported by evidence of effectiveness, as judged by IES standards such as those used in the Department's Investing in Innovation program;
    • Funding to evaluate previously untested but highly promising interventions, through studies overseen by IES that allow for strong causal conclusions, including randomized controlled trials where appropriate; and
    • Funding incentives for state and local educational agencies to engage in systematic evaluation and improvement of local initiatives, consistent with evidence standards established by IES. (March 2011)

 This study was completed under one of the Regional Education Laboratory contracts.
Designates that evaluation has been completed.

Completed by the Regional Educational Laboratories.
*Projected to be released

* Projected to be released.