Inside IES Research

Replication research may not grab the headlines, but reproducing findings from previous studies is critical for advancing scientific knowledge. Some have raised concerns about whether we conduct a sufficient number of replication studies. This concern has drawn increased attention from scholars in a variety of fields, including special education.

Several special education researchers explored this issue in a recent Special Series on Replication Research in Special Education in the journal, Remedial and Special Education. The articles describe replication concepts and issues, systematically review the state of replication research in special education, and provide recommendations for the field. One finding is that there may be more replication studies than it seems—but authors don’t call them replications.

Contributors to the special issue include Bryan Cook from the University of Hawaii, Michael Coyne from the University of Connecticut, and Bill Therrien from the University of Virginia, who served as guest editors, and Chris Lemons, from Peabody College of Vanderbilt University. They shared more about the special issue and their collective insights into replications in special education research.

(In photo array, top left going clockwise: Therrien, Lemons, Coyne, and Cook)

How did you become interested in replication work?

Replication is a core component of the scientific method. Despite this basic fact that we all learned in Research 101, it is pretty apparent that in practice, replication is often ignored. We noticed how much attention the lack of replication was starting to get in other fields and in the press and were particularly alarmed by recent work showing that replications often fail to reproduce original findings. This made us curious about the state and nature of replication in the field of special education.

What is the state of replication research in special education?

It depends on how you define replication and how you search for replication articles. When a narrow definition is used and you require the term “replication” to be in the article, the rate of replication doesn’t look too good. Using this method, Lemons et al. (2016) and Makel et al. (2016) reported that the rate of replication in special education is between 0.4 to 0.5%, meaning that out of all the articles published in our field, less than 1% are replications. We suspected that—for a number of reasons (e.g., perceptions that replications are difficult to publish, are less prestigious than novel studies, and are hostile attempts to disprove a colleague’s work)—researchers might be conducting replication studies but not referring to them as such. And, indeed it’s a different story when you use a broad definition and you do not require the term replication to be in the article. Cook et al. (2016) found that out of 83 intervention studies published in six non-categorical special education journals from 2013-2014, there were 26 (31%) that could be considered replications, though few authors described their studies that way. Therrien et al. (2016) selected eight intervention studies from 1999-2001 and determined whether subsequently published studies that cited the original investigations had replicated them. They found that six of the eight original studies had been replicated by a total of 39 different studies (though few of the replications identified themselves as such).

What were some other key findings across the review articles?

Additional findings indicated that: (a) most replications conducted in special education are conceptual (i.e., some aspects are the same as the original study, but some are different) as opposed to direct (i.e., as similar to the original study as possible), (b) the findings of the majority of replications in special education agreed with the findings of the original studies, and (c) most replications in the field are conducted by one or more authors involved in the original studies. In three of the four reviews, we found it was more likely for a replication to produce the same outcome if there was author overlap between the original and replication studies. This may be due to the challenges of replicating a study with the somewhat limited information provided in a manuscript. It also emphasizes the importance of having more than one research team independently replicate study findings.  

What are your recommendations for the field around replicating special education interventions?

The article by Coyne et al. (2016) describes initial recommendations for how to conceptualize and carry out replication research in a way that contributes to the evidence about effective practices for students with disabilities and the conditions under which they are more or less effective:

• Many studies evaluate an approach that has previously been studied under different conditions. In this case, researchers should specify which aspects replicate previous research;
• Conceptualize and report intervention research within a framework of systematic replications, or a continuum of conceptual replications ranging from those that are more closely aligned to the original study to those that are less aligned;
• Design and conduct closely aligned replications that duplicate, as faithfully as possible, the features of previous studies.
• Design and conduct less closely aligned replications that intentionally vary essential components of earlier studies (e.g., participants, setting, intervention features, outcome measures, and analyses); and
• Interpret findings using a variety of methods, including statistical significance, directions of effects, and effect sizes. We also encourage the use of meta-analytic aggregation of effects across studies.

One example of a high-quality replication study is by Doabler et al. The authors conducted a closely aligned replication study of a Tier 2 kindergarten math intervention. In the design of their IES-funded project, the authors planned a priori to conduct a replication study that would vary on several dimensions, including geographical location, participant characteristics, and instructional context. We believe this is a nice model of designing, conducting, and reporting a replication study.

Ultimately, we need to conduct more replication studies, we need to call them replications, we need to better describe how they are alike and different from the original study, and we need to strive for replication by researchers not involved in the original study. It is this type of work that may increase the impact research has on practice, because it strengthens our understanding of whether, when, and where an intervention works.

By Katie Taylor, Program Officer, National Center for Special Education Research

Although traditional pencil and paper tests provide good information for many purposes, technology presents the opportunity to assess students on tasks that better elicit the real world skills called for by college and career standards. IES supports a number of researchers and developers who are using technology to develop better assessments through grants as well as the Small Business Innovation Research program. 

One example of the power of technology to support innovative assessment  is the Global, Integrated, Scenario-based Assessment (known as ‘GISA’), developed by John Sabatini and Tenaha O’Reilly at the Educational Testing Service as part of a grant supported by the Reading for Understanding Research Initiative.

Each GISA scenario is structured to resemble a timely, real-world situation. For example, one scenario begins by explaining that the class has been asked to create a website on green schools. The student is assigned the task of working with several students (represented by computer avatars) to create the website. In working through the scenario, the student engages in activities that are scaffolded to support students in summarizing information, completing a graphic organizer, and collaborating to evaluate whether statements are facts or opinions. The scenario provides a measure of each student’s ability to learn from text through assessing his or her knowledge of green schools before and after completing the scenario. This scenario is available on the ETS website along with more information about the principles on which GISA was built.

Karen Douglas, of the National Center for Education Research, recently spoke to Dr. Sabatini and Dr. O’Reilly on the role of technology in creating GISA, what users think of it, and their plans for continuing to develop technology-based assessments.

How did the use of technology contribute to the design of GISA?

Technological delivery creates many opportunities over more traditional paper and pencil test designs. On the efficiency side of the argument, items and tasks can be delivered over the internet in a standardized way and there are obvious advantages for automated scoring. However, the real advantage has to do with both the control over test environment and what can be assessed. We can more effectively simulate the digital environments that students use in school, leisure and, later, in the workforce. GISA uses scenario-based assessment to deliver items and tasks. During a scenario-based assessment students are given a plausible reason for reading a collection of thematically related materials. The purpose defines what is important to focus on as students work towards a larger goal. The materials are diverse and may reflect different perspectives and quality of information. 

In 2013, the National Center for Special Education Research (NCSER) launched the Accelerating the Academic Achievement of Students with Learning Disabilities Research Initiative (A3). The goal was to develop and evaluate intensive interventions—such as curricula, instructional approaches and technology—that could improve the academic achievement of students with or at risk of a disability.

A five-year grant in this initiative went to Dr. Douglas Fuchs and Dr. Lynn Fuchs (pictured), of Vanderbilt University’s Peabody College, who for the past three years have been developing and piloting intensive interventions focused on improving students’ reading comprehension of informational texts and fraction and pre-algebra performance.

Earlier this month, the Fuchs joined Dr. Lou Danielson and Dr. Rebecca Zumeta Edmonds from the National Center on Intensive Interventions (NCII) for a webinar: “Intensive Intervention: What is it, Who it’s For, and Why it’s Important?” (NCII is a research initiative funded by the U.S. Department of Education’s Office of Special Education Programs.)

The NCII/A3 webinar was purposely held in October—which is Learning Disabilities Awareness Month—to raise awareness of research and resources to support students with learning disabilities. The session was recorded and is available through the NCII website or you can watch it below.

The panelists discussed the intensive intervention process, methods of identifying students not making adequate academic progress, and recent related research. Specifically, the Fuchs’ shared their research designing and piloting two innovative components that seek to expand responsiveness to intervention:

• Capitalizing on the power of prior knowledge to build informational text comprehension; and
• Capitalizing on the power of executive function to build fractions knowledge

As part of this NCSER A3 Initiative, these and other intervention components are being developed, integrated into comprehensive intervention programs, and rigorously tested. Please visit the project website to learn more and keep up to date with the latest findings from this research. Viewers of the recorded webinar can also learn more about implementation support resources available through NCII.

In the final years of their five-year NCSER grant, Doug and Lynn Fuchs will work to understand the efficacy of intensive interventions for improving outcomes for students with learning disabilities.  

Written by Sarah Brasiel, Program Officer, NCSER

Photo by Wolf Hoffmann,courtesy of Vanderbilt University

A paper based on an IES-funded grant has been recognized as a “must read” by the Council for Learning Disabilities.

IES-funded researcher, Stephen Hooper, and his colleagues were recently recognized by the Council for their paper: Writing disabilities and reading disabilities in elementary school students: Rates of co-occurrence and cognitive burden (PDF). The paper was written by Lara-Jeane Costa, Crystal Edwards, and Dr. Hooper and published in Learning Disability Quarterly. Every year, the Council for Learning Disabilities acknowledges outstanding work published in its journals and selected this paper as one of two Must Read pieces for 2016. The authors will present on the paper at the Council's annual conference in San Antonio this week (October 13-14, 2016).

This paper was funded through a grant from the National Center for Education Research (NCER) to examine written language development and writing problems, and the efficacy of an intervention aimed at improving early writing skills. The results of the paper found that the rate of students with both writing and reading disabilities increased from first to fourth grade and these students showed lower ability in language, fine motor skills and memory compared with students with neither disability or only a writing disability.  

The team continues their IES-funded work by looking at the efficacy of the Self-Regulated Strategy Development intervention on struggling middle school writers’ academic outcomes.

Written by Becky McGill-Wilkinson, Education Research Analyst, NCER

In February, President Obama named two Institute of Education Sciences (IES) grantees as recipients of the prestigious Presidential Early Career Award for Scientists and Engineers (PECASE). Christopher Lemons (Peabody College of Vanderbilt University) and Cynthia Puranik (Georgia State University) were honored in Washington, DC at a White House ceremony this past May, along with 103 other recipients of the award.

Dr. Lemons and Dr. Puranik have recently completed an IES-funded project, Enhancing Reading Instruction for Children with Down Syndrome: A Behavioral Phenotypic Approach. They served as the Co-Principal Investigators of this study to develop an intervention for improving reading for children with Down syndrome (DS). The intervention is a supplemental, 16-week reading curriculum that incorporates critical components of early reading (e.g., vocabulary, decoding skills, fluency) and has been adapted and modified to align with the Down syndrome behavioral phenotype—a set of characteristics commonly shared by children with the syndrome. The characteristics of focus for this intervention include the ability to process visual information and challenges with auditory and expressive language, using working and short-term memory, and motivation. Kim Sprague, program officer in the National Center for Special Education Research, spoke with Dr. Lemons (pictured right) about the project and next steps.   

Why do we need this reading intervention?

Over the last decade, research has demonstrated that students with DS can do better in school than we ever imagined in the past. The purpose of our intervention is to increase reading outcomes by adapting instruction that targets phonological awareness and phonics so that it is more closely aligned with the Down syndrome behavioral phenotype in an attempt to make the instruction more effective for children with Down syndrome.

What is the purpose of studying phenotype?

This is something I am often asked. We were looking for ideas to improve reading interventions for students with DS and there is evidence in the literature base that students with DS have a heightened probability of sharing certain characteristics—a behavioral phenotype. So we sought to develop an intervention based on this specific phenotype and our goal was to determine whether adapting instruction for a group who share common features may hold promise.

What was the intervention that was implemented for children?

Through the iterative process for this development grant, we experimented with applying several adaptations to evidence-based phonological awareness and phonics instruction. Our primary adaptation was to teach students a key word for each letter sound. We selected words that students were likely to know—like “dog.” In each lesson, we focused on three letter sounds paired with a picture and taught the student to match the picture to the printed word. We then used this word-picture pair to support the other components of the lesson that were focused on phonological awareness, alphabetic principle, decoding and scaffolding working memory. We worked with the paraprofessionals and teachers to train and support their implementation of the developed intervention.

What were the study results? 

We have learned that our intervention is effective at increasing reading outcomes for many students with DS who enter the program at the lowest reading levels. These children appear to have benefited from our adaptations. This said, there were some children with DS who needed additional individualization, so special education teachers need to understand how to provide this type of ongoing adaptation. Also, we demonstrated that many students’ who had already “broken the code”—students who already understand and can apply the alphabetic principle—didn’t need adapted instruction. They benefited from the standard version of the phonics intervention.

What are the next steps based on this work?

We still have a lot of work to do. I think we have had success in helping students improve phonological awareness and decoding skills. We need to more strongly focus on comprehension. So, how do we provide assistance to help them answer who, what, where, and when questions? How can we support students’ abilities in interpreting the meaning of texts independently? And, more specifically for students with DS, we need a better understanding of the variability that exists related to characteristics associated with the phenotype. We need stronger empirical work to explore potential aptitude-by-treatment effects that may result from this line of work and continue to explore interventions that are feasible and effective.

For reading instruction provided to students with DS, ensuring that paraprofessionals are appropriately trained and supported to deliver the reading intervention as well as to adapt to the individual needs of the student is critical. Doing so may offer the greatest hope to improve outcomes because paraprofessionals are often able to provide one-on-one instruction more frequently than the special education teacher. During the study, special education teachers and paraprofessionals were willing to provide one-on-one instructional time to participating students for four or five days per week. However, most of the teachers indicated that they would not be able to maintain this level of intensity after the study was over.

We are beginning work to explore how to better support teachers of students with intellectual disabilities as they implement data-based individualization, or DBI, as a method to enhance reading outcomes. We think this approach holds promise. Readers of your blog can learn more about this approach at the National Center on Intensive Intervention at www.intensiveintervention.org.

Wendy Wei, a program assistant in the National Center for Education Research, and Diane Mechner, who worked as an IES intern, contributed to this blog post.