This project is jointly funded by the National Center for Special Education Research and the National Center for Education Research.

Co-Principal Investigator: Elizabeth Tipton

Purpose: The Institute of Education Sciences (IES) was established by the Education Sciences Reform Act in 2002 to "provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study." As IES approaches its twentieth anniversary in 2022, it is appropriate to ask how well it has met this mandate. One important component of the work that IES has undertaken to achieve its mission has been the funding of randomized trials of the efficacy and effectiveness of education interventions products, services, or policies. Throughout IES's history, some have noted that a substantial proportion of those studies have failed to find statistically significant effects of the interventions they investigated. While this reference may be factually correct in a narrow sense (with a broad definition of "failed trial"), it is simplistic, obscuring the complexities of scientific process and the ways in which science actually makes progress. This study will investigate the corpus of large-scale randomized trials that have been funded by IES during its lifetime to determine what has been learned and in what if any ways they have contributed to advances in the education sciences.

Project Activities: This project will examine a sample of the efficacy and effectiveness trials funded by IES and completed by 2020. It will seek to understand and document the direct and indirect impact of each project on practice, fundamental understanding of education, human capital for education research, and research methods. Data sources will include project abstracts from the IES website, study proposals, and reports of those studies, as well as interviews with Principal Investigators. Researchers will use internet searches to identify products (such as papers) and citations to the studies. They will use web scraping to discover mentions of the studies in websites of state education agencies, school districts, regional education labs, and professional organizations.  Data extracted from these sources will be systematically coded and reliability of the coding will be assessed.

The research team will document the products of each study (such as scientific papers and reports) as well as indications that they influenced practice either directly or indirectly. Scientific knowledge is not just created from single studies in isolation, but by studies building upon one another. They will build a database showing relations between each study and both prior and subsequent related studies. They will document the ways in which earlier studies influenced later studies through what they learned, though the personnel that they trained, and through the fundamental knowledge that they created. They will also document situations in which a study either created a methodological innovation that was later used in other studies or motivated other methodological research. They will also include a set of case studies designed to illustrate the ways in which particular studies have made contributions to knowledge about education.

Data Analysis: The data analysis will largely be descriptive. Researchers will identify each study's direct and indirect contributions to practice, to fundamental knowledge and understanding of education, contribution to human capital for education research, and impact on methodology (or lack thereof). They will summarize the data identify the network structure of relations among the studies and the network structure of personnel who worked on the studies.

Key Outcomes: Researchers will describe:

• the number of efficacy and effectiveness trials ended with statistically significant outcomes, were not completed, or had non-significant results, along with the basic demographics of these trials
• the reasons the trials did not result in statistically significant outcomes, including those that experienced recruitment, methodological, implementation issues, or problems with the theory of change
• results that were used directly to improve practice, including inclusion in the WWC and IES practice guides, direct commercialization, or having influence on other products
• indirect effects of each study, such as contributing to a larger research agenda that expanded fundamental knowledge and understanding of education (e.g., design of future interventions, relationships with stakeholders)
• how each study affected the creation of human capital for education science, and improvements in scientific infrastructure
• how the results of trials led to the improvement of methods for designing interventions or the design, analysis, or interpretation of education research.