IES Blog

Education research is often a slow and costly process. An even more difficult challenge is replicating research findings in a timely and cost-effective way to ensure that they are meaningful for the wide range of contexts and populations that make up our nation’s school system.

In a recent op-ed, IES Director Mark Schneider and Schmidt Futures Senior Director for Technology and Society Kumar Garg pitched the idea that digital learning platforms may be a way to accelerate the research enterprise. These platforms will enable researchers to try new ideas and replicate interventions quickly across many sites and with a wide range of student populations. They could also open the door for educators to get more involved in the research process. For example, Learn Platform supports districts as they make decisions about purchasing and implementing products in their schools, and ASSISTments provides infrastructure for researchers to conduct cheaper and faster studies than they would be able to do on their own.

IES Director Mark Schneider and NCER Commissioner Liz Albro recently attended a meeting sponsored by Schmidt Futures focused on these issues. Two major takeaways from the meeting: first, there is already progress on building and using platforms for testing interventions, and, second, platform developers are enthusiastic about integrating research capabilities into their work.

As we consider how we can support platform developers, researchers, and education personnel to co-design tools to enable more efficient, large scale research on digital learning platforms, several questions have arisen:  

1. What digital learning platforms already have a large enough user base to support large scale research studies?
2. Are there content areas or grade levels that are not well supported through digital learning platforms?
3. What are the key features that a platform needs to have to support rigorous tests and rapid replication of research findings? 
4. What are the barriers and challenges for companies interested in participating in this effort?
5. What kinds of research questions can best be answered in this research environment?
6. What kind of infrastructure needs to be developed around the platform to enable seamless collaborations between education stakeholders, researchers, and product developers?

We know there are some of you have already given these questions some thought. In addition, there are other questions and issues that we haven’t considered. We welcome your thoughts. Feel free to email us at Erin.Higgins@ed.gov and Elizabeth.Albro@ed.gov. And join NCER’s Virtual Learning Lab in their virtual workshop “Designing Online Learning Platforms to Enable Research” on April 17th, 3:00pm-5:00pm Eastern Time. Learn more about the workshop here.

Data sharing agreements are critical to conducting research in education. They allow researchers to access data collected by state or local education agencies to examine trends, determine the effectiveness of interventions, and support agencies in their efforts to use research-based evidence in decision-making.

Yet the process for obtaining data sharing agreements with state or local agencies can be challenging and often depends on the type of data involved, state and federal laws and regulations regarding data privacy, and specific agency policies. Some agencies have a research application process and review timeline available on their websites. Others may have a more informal process for establishing such agreements. In all instances, these agreements determine how a researcher can access, use, and analyze education agency data.

What are some guiding principles for successfully obtaining data sharing agreements? 

Over several years of managing projects that require data sharing agreements, I have learned a few key principles for success. While they may seem obvious, I have witnessed data sharing agreements fall apart because one or more of these principles were not met:

Replication research may not grab the headlines, but reproducing findings from previous studies is critical for advancing scientific knowledge. Some have raised concerns about whether we conduct a sufficient number of replication studies. This concern has drawn increased attention from scholars in a variety of fields, including special education.

Several special education researchers explored this issue in a recent Special Series on Replication Research in Special Education in the journal, Remedial and Special Education. The articles describe replication concepts and issues, systematically review the state of replication research in special education, and provide recommendations for the field. One finding is that there may be more replication studies than it seems—but authors don’t call them replications.

Contributors to the special issue include Bryan Cook from the University of Hawaii, Michael Coyne from the University of Connecticut, and Bill Therrien from the University of Virginia, who served as guest editors, and Chris Lemons, from Peabody College of Vanderbilt University. They shared more about the special issue and their collective insights into replications in special education research.

(In photo array, top left going clockwise: Therrien, Lemons, Coyne, and Cook)

How did you become interested in replication work?

Replication is a core component of the scientific method. Despite this basic fact that we all learned in Research 101, it is pretty apparent that in practice, replication is often ignored. We noticed how much attention the lack of replication was starting to get in other fields and in the press and were particularly alarmed by recent work showing that replications often fail to reproduce original findings. This made us curious about the state and nature of replication in the field of special education.

What is the state of replication research in special education?

It depends on how you define replication and how you search for replication articles. When a narrow definition is used and you require the term “replication” to be in the article, the rate of replication doesn’t look too good. Using this method, Lemons et al. (2016) and Makel et al. (2016) reported that the rate of replication in special education is between 0.4 to 0.5%, meaning that out of all the articles published in our field, less than 1% are replications. We suspected that—for a number of reasons (e.g., perceptions that replications are difficult to publish, are less prestigious than novel studies, and are hostile attempts to disprove a colleague’s work)—researchers might be conducting replication studies but not referring to them as such. And, indeed it’s a different story when you use a broad definition and you do not require the term replication to be in the article. Cook et al. (2016) found that out of 83 intervention studies published in six non-categorical special education journals from 2013-2014, there were 26 (31%) that could be considered replications, though few authors described their studies that way. Therrien et al. (2016) selected eight intervention studies from 1999-2001 and determined whether subsequently published studies that cited the original investigations had replicated them. They found that six of the eight original studies had been replicated by a total of 39 different studies (though few of the replications identified themselves as such).

What were some other key findings across the review articles?

Additional findings indicated that: (a) most replications conducted in special education are conceptual (i.e., some aspects are the same as the original study, but some are different) as opposed to direct (i.e., as similar to the original study as possible), (b) the findings of the majority of replications in special education agreed with the findings of the original studies, and (c) most replications in the field are conducted by one or more authors involved in the original studies. In three of the four reviews, we found it was more likely for a replication to produce the same outcome if there was author overlap between the original and replication studies. This may be due to the challenges of replicating a study with the somewhat limited information provided in a manuscript. It also emphasizes the importance of having more than one research team independently replicate study findings.  

What are your recommendations for the field around replicating special education interventions?

The article by Coyne et al. (2016) describes initial recommendations for how to conceptualize and carry out replication research in a way that contributes to the evidence about effective practices for students with disabilities and the conditions under which they are more or less effective:

• Many studies evaluate an approach that has previously been studied under different conditions. In this case, researchers should specify which aspects replicate previous research;
• Conceptualize and report intervention research within a framework of systematic replications, or a continuum of conceptual replications ranging from those that are more closely aligned to the original study to those that are less aligned;
• Design and conduct closely aligned replications that duplicate, as faithfully as possible, the features of previous studies.
• Design and conduct less closely aligned replications that intentionally vary essential components of earlier studies (e.g., participants, setting, intervention features, outcome measures, and analyses); and
• Interpret findings using a variety of methods, including statistical significance, directions of effects, and effect sizes. We also encourage the use of meta-analytic aggregation of effects across studies.

One example of a high-quality replication study is by Doabler et al. The authors conducted a closely aligned replication study of a Tier 2 kindergarten math intervention. In the design of their IES-funded project, the authors planned a priori to conduct a replication study that would vary on several dimensions, including geographical location, participant characteristics, and instructional context. We believe this is a nice model of designing, conducting, and reporting a replication study.

Ultimately, we need to conduct more replication studies, we need to call them replications, we need to better describe how they are alike and different from the original study, and we need to strive for replication by researchers not involved in the original study. It is this type of work that may increase the impact research has on practice, because it strengthens our understanding of whether, when, and where an intervention works.

By Katie Taylor, Program Officer, National Center for Special Education Research

By Ruth Curran Neild, Delegated Director, IES

The mission of the Institute of Education Sciences (IES), at its core, is to create a culture in which independent, rigorous research and statistics are used to improve education. But sometimes research is seen by practitioners and policymakers as something that is done for them or to them, but not by them. And that’s something we’re hoping to change.

The American Educational Research Association (AERA) will hold its annual meeting April 8 through April 12 in Washington, D.C.—the largest educational research gathering in the nation. This will be a special meeting for AERA, as it is celebrating 100 years of advocating for the development and use of research in education. The program includes hundreds of sessions, including opportunities to learn about cutting edge education research and opportunities to broaden and deepen the field. 

About 30 sessions will feature staff from the Institute of Education Sciences (IES) discussing IES-funded research, evaluation, and statistics, as well as training and funding opportunities.

On Saturday, April 9, at 10:35 a.m., attendees will have a chance to meet the Institute’s leadership and hear about the areas of work that IES will be focusing on in the coming year. Speakers include Ruth Curran Neild, IES’ delegated director, and the leaders of the four centers in IES: Thomas Brock, commissioner of the National Center for Education Research (NCER); Peggy Carr, acting commissioner of the National Center for Educational Statistics (NCES); Joy Lesnick, acting commissioner of the National Center for Education Evaluation and Regional Assistance (NCEE), and Joan McLaughlin, commissioner of the National Center for Special Education Research (NCSER).

On Monday, April 11, at 9:45 a.m., attendees can speak to one of several IES staffers who will be available at the Research Funding Opportunities—Meet Your Program Officers session. Program officers from NCER, NCSER, and NCEE will be on hand to answer questions about programs and grant funding opportunities. Several IES representatives will also be on hand Monday afternoon, at 4:15 p.m. for the Federally Funded Data Resources: Opportunities for Research session to discuss the myriad datasets and resources that are available to researchers.

NCES staff will lead sessions and present on a variety of topics, from The Role of School Finance in the Pursuit of Equity (Saturday, 12:25 p.m.) to Understanding Federal Education Policies and Data about English Learners (Sunday, April 10, 8:15 a.m.) and what we can learn from the results of PIAAC, a survey of adult skills (also Sunday, 8:15 a.m.). Dr. Carr will be a part of several sessions, including one on Sunday morning (10:35 a.m.) about future directions for NCES longitudinal studies and another on Monday morning (10 a.m.) entitled Issues and Challenges in the Fair and Valid Assessment of Diverse Populations in the 21^st Century. 

On Monday, at 11:45 a.m., you can also learn about an IES-supported tool, called RCT-YES, that is designed to reduce barriers to rigorous impact studies by simplifying estimation and reporting of study results (Dr. Lesnick will be among those presenting). And a team from the IES research centers (NCER/NCSER) will present Sunday morning (10:35 a.m.) on communication strategies for disseminating education research (which includes this blog!).

IES staff will also participate in a number of other roundtables and poster sessions. For instance, on Tuesday, April 12, at 8:15 a.m., grab a cup of coffee and attend the structured poster session with the Institute’s 10 Regional Educational Laboratories (RELs). This session will focus on building partnerships to improve data use in education.  REL work will also be featured at several other AERA sessions.  

Did you know that the National Library of Education (NLE) is a component of IES? On Friday and Monday afternoon, attendees will have a unique opportunity to go on a site visit to the library. You’ll learn about the library’s current and historical resources – including its collection of more than 20,000 textbooks dating from the mid-19^th century. The Library offers information, statistical, and referral services to the Department of Education and other government agencies and institutions, and to the public.

If you are going to AERA, follow us on Twitter to learn more about our sessions and our work.  And if you are tweeting during one of our sessions, please include @IESResearch in your tweet. 

By Dana Tofig, Communications Director, IES