Replication research may not grab the headlines, but reproducing findings from previous studies is critical for advancing scientific knowledge. Some have raised concerns about whether we conduct a sufficient number of replication studies. This concern has drawn increased attention from scholars in a variety of fields, including special education.
Several special education researchers explored this issue in a recent Special Series on Replication Research in Special Education in the journal, Remedial and Special Education. The articles describe replication concepts and issues, systematically review the state of replication research in special education, and provide recommendations for the field. One finding is that there may be more replication studies than it seems—but authors don’t call them replications.
Contributors to the special issue include Bryan Cook from the University of Hawaii, Michael Coyne from the University of Connecticut, and Bill Therrien from the University of Virginia, who served as guest editors, and Chris Lemons, from Peabody College of Vanderbilt University. They shared more about the special issue and their collective insights into replications in special education research.
(In photo array, top left going clockwise: Therrien, Lemons, Coyne, and Cook)
How did you become interested in replication work?
Replication is a core component of the scientific method. Despite this basic fact that we all learned in Research 101, it is pretty apparent that in practice, replication is often ignored. We noticed how much attention the lack of replication was starting to get in other fields and in the press and were particularly alarmed by recent work showing that replications often fail to reproduce original findings. This made us curious about the state and nature of replication in the field of special education.
What is the state of replication research in special education?
It depends on how you define replication and how you search for replication articles. When a narrow definition is used and you require the term “replication” to be in the article, the rate of replication doesn’t look too good. Using this method, Lemons et al. (2016) and Makel et al. (2016) reported that the rate of replication in special education is between 0.4 to 0.5%, meaning that out of all the articles published in our field, less than 1% are replications. We suspected that—for a number of reasons (e.g., perceptions that replications are difficult to publish, are less prestigious than novel studies, and are hostile attempts to disprove a colleague’s work)—researchers might be conducting replication studies but not referring to them as such. And, indeed it’s a different story when you use a broad definition and you do not require the term replication to be in the article. Cook et al. (2016) found that out of 83 intervention studies published in six non-categorical special education journals from 2013-2014, there were 26 (31%) that could be considered replications, though few authors described their studies that way. Therrien et al. (2016) selected eight intervention studies from 1999-2001 and determined whether subsequently published studies that cited the original investigations had replicated them. They found that six of the eight original studies had been replicated by a total of 39 different studies (though few of the replications identified themselves as such).
What were some other key findings across the review articles?
Additional findings indicated that: (a) most replications conducted in special education are conceptual (i.e., some aspects are the same as the original study, but some are different) as opposed to direct (i.e., as similar to the original study as possible), (b) the findings of the majority of replications in special education agreed with the findings of the original studies, and (c) most replications in the field are conducted by one or more authors involved in the original studies. In three of the four reviews, we found it was more likely for a replication to produce the same outcome if there was author overlap between the original and replication studies. This may be due to the challenges of replicating a study with the somewhat limited information provided in a manuscript. It also emphasizes the importance of having more than one research team independently replicate study findings.
What are your recommendations for the field around replicating special education interventions?
The article by Coyne et al. (2016) describes initial recommendations for how to conceptualize and carry out replication research in a way that contributes to the evidence about effective practices for students with disabilities and the conditions under which they are more or less effective:
- Many studies evaluate an approach that has previously been studied under different conditions. In this case, researchers should specify which aspects replicate previous research;
- Conceptualize and report intervention research within a framework of systematic replications, or a continuum of conceptual replications ranging from those that are more closely aligned to the original study to those that are less aligned;
- Design and conduct closely aligned replications that duplicate, as faithfully as possible, the features of previous studies.
- Design and conduct less closely aligned replications that intentionally vary essential components of earlier studies (e.g., participants, setting, intervention features, outcome measures, and analyses); and
- Interpret findings using a variety of methods, including statistical significance, directions of effects, and effect sizes. We also encourage the use of meta-analytic aggregation of effects across studies.
One example of a high-quality replication study is by Doabler et al. The authors conducted a closely aligned replication study of a Tier 2 kindergarten math intervention. In the design of their IES-funded project, the authors planned a priori to conduct a replication study that would vary on several dimensions, including geographical location, participant characteristics, and instructional context. We believe this is a nice model of designing, conducting, and reporting a replication study.
Ultimately, we need to conduct more replication studies, we need to call them replications, we need to better describe how they are alike and different from the original study, and we need to strive for replication by researchers not involved in the original study. It is this type of work that may increase the impact research has on practice, because it strengthens our understanding of whether, when, and where an intervention works.
By Katie Taylor, Program Officer, National Center for Special Education Research