IES Blog

Institute of Education Sciences

Building Evidence: What Comes After an Efficacy Study?

Over the years, the Institute of Education Sciences (IES) has funded over 300 studies across its research programs that evaluate the efficacy of specific programs, policies, or practices. This work has contributed significantly to our understanding of the interventions that improve outcomes for students under tightly controlled or ideal conditions. But is this information enough to inform policymakers’ and practitioners’ decisions about whether to adopt an intervention? If not, what should come after an efficacy study?

In October 2016, IES convened a group of experts for a Technical Working Group (TWG) meeting to discuss next steps in building the evidence base after an initial efficacy study, and the specific challenges that are associated with this work. TWGs are meant to encourage stakeholders to discuss the state of research on a topic and/or to identify gaps in research.  

Part of this discussion focused on replication studies and the critical role they play in the evidence-building process. Replication studies are essential for verifying the results of a previous efficacy study and for determining whether interventions are effective when certain aspects of the original study design are altered (for example, testing an intervention with a different population of students). IES has supported replication research since its inception, but there was general consensus that more replications are needed.

TWG participants discussed some of the barriers that may be discouraging researchers from doing this work. One major obstacle is the idea that replication research is somehow less valuable than novel research—a bias that could be limiting the number of replication studies that are funded and published. A related concern is that the field of education lacks a clear framework for conceptualizing and conducting replication studies in ways that advance evidence about beneficial programs, policies and practices (see another recent IES blog post on the topic).

IES provides support for studies to examine the effectiveness of interventions that have prior evidence of efficacy and that are implemented as part of the routine and everyday practice occurring in schools without special support from researchers. However, IES has funded a relatively small number of these studies (14 across both Research Centers). TWG participants discussed possible reasons for this and pointed out several challenges related to replicating interventions under routine conditions in authentic education settings. For instance, certain school-level decisions can pose challenges for conducting high-quality effectiveness studies, such as restricting the length that interventions or professional development can be provided and choosing to offer the intervention to students in the comparison condition. These challenges can result in findings that are influenced more by contextual factors rather than the intervention itself. TWG participants also noted that there is not much demand for this level of evidence, as the distinction between evidence of effectiveness and evidence of efficacy may not be recognized as important by decision-makers in schools and districts.

In light of these challenges, TWG participants offered suggestions for what IES could do to further support the advancement of evidence beyond an efficacy study. Some of these recommendations were more technical and focused on changes or clarifications to IES requirements and guidance for specific types of research grants. Other suggestions included:

  • Prioritizing and increasing funding for replication research;
  • Making it clear which IES-funded evaluations are replication studies on the IES website;
  • Encouraging communication and partnerships between researchers and education leaders to increase the appreciation and demand for evidence of effectiveness for important programs, practices, and policies; and
  • Supporting researchers in conducting effectiveness studies to better understand what works for whom and under what conditions, by offering incentives to conduct this work and encouraging continuous improvement.

TWG participants also recommended ways IES could leverage its training programs to promote the knowledge, skills, and habits that researchers need to build an evidence base. For example, IES could emphasize the importance of training in designing and implementing studies to develop and test interventions; create opportunities for postdoctoral fellows and early career researchers to conduct replications; and develop consortiums of institutions to train doctoral students to conduct efficacy, replication, and effectiveness research in ways that will build the evidence base on education interventions that improve student outcomes.

To read a full summary of this TWG discussion, visit the Technical Working Group website or click here to go directly to the report (PDF).

Written by Katie Taylor, National Center for Special Education Research, and Emily Doolittle, National Center for Education Research

Improving Research on the Forgotten ‘R’

Writing is often labeled as the “forgotten ‘R,’” because the other R’s—reading and ‘rithmetic—seem to garner so much attention from educators, policymakers, and researchers. Yet, we know writing is a critical skill for communication and for success in school and in career. Writing in middle and high school can be especially important, because secondary grades are where students are expected to have mastered foundational skills like handwriting and move on to the application of these skills to more complex compositions.

IES has been funding research on writing since its inception in 2002, but compared to research on reading, not much work has been done in this critical area, especially writing in middle and high schools. In an effort to learn more about the state of the field of writing in secondary schools and the areas of needed research, IES brought together 13 experts on secondary writing for a Technical Working Group (TWG) meeting in September. During the full-day meeting, TWG participants shared their thoughts and expertise on a variety of topics including: argumentative writing, methods of engaging adolescents in writing, how best to help struggling writers including English learners and students with or at risk for disabilities, and assessment and feedback on writing.

Argumentative writing requires students to explore a topic, collect and evaluate evidence, establish a position on a topic, and consider alternative positions. In middle and high schools, argumentative writing often occurs in content area classrooms like science and history. TWG participants discussed the importance of research to understand how argumentative writing develops over time and how teachers contribute to this development.

Teaching writing to students with or at risk for disabilities and English learners can be challenging when the focus of secondary schools is often on content acquisition and not on improving writing skills. English learners are typically grouped together and receive the same instruction, but little is known about how writing instruction may need to be differentiated for students from different language backgrounds. Additionally, the TWG participants discussed the need to investigate the potential for technology to help with instruction of students who struggle with writing, and the importance to addressing the negative experiences these students have with writing that may discourage them from writing in the future.

It is also important to make sure all students are engaged and motivated to write. Some middle and high school students  may not want to participate in writing or may have internalized beliefs that they are not good at it. TWG participants discussed the need to consider teaching students that writing abilities can be changed, and that introducing new audiences or purposes for writing may motivate students to write. Finally, the group talked about the importance of allowing middle and high school students to write about topics of their own choosing.

Assessing the writing quality of middle and high school students is difficult, because what counts as good writing is often subjective. Technology may offer some solutions, but TWG participants emphasized that it is unlikely that computers will be able to do this task well entirely on their own. Regardless, the TWG participants were in agreement that there is a need for the development of quality writing measures for use both by teachers and by researchers.  Teachers may feel pressure to provide detailed feedback on students’ writing, which can be time-consuming. TWG participants argued that self-assessment and peer feedback could relieve some of the pressure on teachers, but research is needed to understand what kind of feedback is best for improving writing and how to teach students to provide useful feedback.

A full summary of the TWG can be found on the IES website. It’s our hope this conversation provides a strong framework for more research on ‘the forgotten R.’

POSTSCRIPT: Our colleagues at the What Works Clearinghouse recently published an Educator’s Practice Guide, “Teaching Secondary Students to Write Effectively.” It includes three research-based recommendations for improving writing for middle and high school students.

Written by Becky McGill-Wilkinson, National Center for Education Research, and Sarah Brasiel, National Center for Special Education Research

The 2016 PI Meeting: Making it Matter

Hundreds of researchers, practitioners, and education scientists gathered in Washington D.C. for the 2016 IES Principal Investigators (PI) Meeting on December 15 & 16. 

The annual meeting provided an opportunity for attendees to share the latest findings from their IES-funded work, learn from one another, and discuss IES and U.S. Department of Education priorities and programs.

The theme of this year’s annual meeting was Making it Matter: Rigorous Research from Design to Dissemination and the agenda included scores of session that highlighted findings, products, methodological approaches, new projects, and dissemination and communication strategies. The meeting was organized by the two IES research centers—the National Center for Education Research and the National Center for Special Education Research—in collaboration with the three meeting co-chairs: Roberta Michnick Golinkoff, of the University of Delaware; Kathleen Lynne Lane, of the University of Kansas; and Grace Wardhana, CEO of Kiko Labs.

Attendees were active on Twitter, using the hashtag #IESPImtg. Several attendees took the opportunity to highlight why their research matters using a sign and a selfie stick. Below are some Twitter highlights of the 2016 PI meeting.  

 

Research Update: Effective Post-school Transition Practices for Students with Disabilities

The special education research community has increasingly focused on how best to support students with disabilities in the transition from high school to postsecondary education or adult life. 

Transition supports provided in schools for these youth can differ and, as a result, the outcomes for these students during and after high school can vary a great deal.[1] The National Center for Special Education Research (NCSER) has funded several grants on the topic.

Some of these NCSER-funded studies have incorporated and evaluated new approaches to provide educators with concrete information about effective practices to promote positive transition outcomes during and after high school.  Here is a brief update on a few promising programs and practices.

Picture (clockwise from top left) - David Test, Mary Wagner, Erik Carter, Sarah Gennen. Photos from university websites.

David Test, Tiana Povenmire-Kirk, Claudia Flowers, and their colleagues at the University of North Carolina at Charlotte recently completed a four-year study of the effects of a transition-planning service delivery system model on transition outcomes for students with disabilities. Communicating Interagency Relationships and Collaborative Linkages for Exceptional Students (CIRCLES) is a three-tier model of interagency collaboration among community, school, and IEP teams (see graphic).[2] The effect of CIRCLES was studied using a group-randomized controlled trial, the first rigorous evaluation of such an intervention. Results indicated many positive impacts of the program including increased collaboration among teams, and increased rates of self-determination, IEP participation, and academic performance for CIRCLE students as compared to students in the control group receiving business-as-usual supports for transition.[3] One year after exiting high school, No differences in post-school outcomes were observed for those in CIRCLES as compared to those in the control group. However, these data were obtained for fewer than half of the original sample. More research is to be done to determine the true impact of CIRCLES on post-school outcomes.   

Mary Wagner and her team addressed questions about the impact of interventions for high school students with autism spectrum disorders using a quasi-experimental design and longitudinal data from several national datasets. Her team found that 2- or 4-year college enrollment rates were significantly higher among youth with autism who participated in transition planning and those who had a primary transition goal of college enrollment.[4] In addition, the results indicated that these enrollment rates were significantly higher among students with autism who were included in secondary school general education English, math, science, or social studies classes than their peers with ASDs who were not included in these classes.[5]

Erik Carter at Vanderbilt University and his research team undertook a four-year study to examine the effect of peer support and peer network strategies as alternatives to traditional paraprofessional-delivered support to assist adolescents with severe disabilities in the classroom. The research team examined the impact of these interventions and found significant increases in participating students’ progress on individual goals, peer interactions and social relationships, social and academic engagement, and community participation compared to those receiving traditional paraprofessional support.[6] Previous research on transition interventions of this kind helped to identify evidence-based practices but this study was the first to rigorously evaluate them. Peers in the classroom can play a unique and valuable role in the welfare of adolescents with severe disabilities, and paraprofessionals and special educators can serve in a different role as facilitators of the peer support provided in these interventions.

Sarah Geenen, Laurie Powers, and their team at Portland State University conducted a longitudinal, experimental study to assess the efficacy of a supplemental transition program designed for youth in high school who are in both special education and foster care. Foster care students, they note, are disproportionately more likely to receive special education services than non-foster care students. The results were compelling, with meaningful and positive effects on youth participants, with lower rates of involvement in the juvenile justice system and increased independent living preparation and skills as compared to non-participants.[7]

Learn more about NCSER-funded work in the Transition Outcomes for Secondary Students with Disabilities topics on the Institute of Education Sciences website or contact Kim Sprague at Kimberley.Sprague@ed.gov.

Written by Diane Mechner, University of Virginia, and Kim Sprague, Program Officer for Transition. Ms. Mechner, a student, was a 2016 IES summer intern.


[2] Povenmire-Kirk, T., Diegelmann, K., Crump, K., Schnorr, C., Test, D.W., Flowers, C., & Aspel, N. (2015). Implementing CIRCLES: A new model for interagency collaboration in transition planning. Journal of Vocational Rehabilitation, 42, 51-65.

[3] Flowers, C., Test, D. W., Povenmire-Kirk, T., Kemp-Inman, A., Diegelmann, K. M., & Bunch-Crump, K. (in press). A cluster randomized controlled trial of a multi-level model of interagency collaboration. Exceptional Children.

[4] Wei, X., Wagner, M., Yu, J. W., Hudson, L., & Javitz, H. (2016). The effect of transition planning and goal-setting on college enrollment among youth with an Autism Spectrum Disorder. Remedial and Special Education, 37(1), 3-14, doi:10.1177/0741932515581495.

[5] Wei, X., Wagner, M., Yu, J. W., & Javitz, H. (in press). The effect of general education inclusion on college enrollment rates among youth with Autism Spectrum Disorders. Autism.

[6] Carter, E. W., Asmus, J., Moss, C. K., Biggs, E. E., Bolt, D. M., Born, T., Brock, M. E., Cattey, G. N., Chen, R,, Cooney, M., Fesperman, E., Hochman, J. M., Huber, H. B., Lequia, J. L., Lyons, G., Moyseenko, K. A., Riesch, L. M., Shalev, R. A., Vincent, L. B., & Weir, K. (2016). Randomized evaluation of peer support arrangements to support the inclusion of high school students with severe disabilities. Exceptional Children, 82(2), 209-233, doi:0014402915598780.

[7] Powers, L. E., Geenen, S., Powers, J., Pommier-Satya, S., Turner, A., Dalton, L. D., Drummond, D., & Swank, P. (2012). My life: Effects of a longitudinal, randomized study of self-determination enhancement on the transition outcomes of youth in foster care and special education. Children and Youth Services Review, 34, 2179–2187, doi:10.1016/j.childyouth.2012.07.018.

 

Gathering Input on Language and Communication Research and Development

Human interaction in society depends upon language and communication and the Institute of Education Sciences is one of several federal agencies that supports research and development (R&D) activities to further our knowledge in this area. 

High school students sitting in a circle talking.

However, so far, there has been no systematic accounting or description of the range of language and communication R & D that the Federal Government supports.To address this gap, the White House Office of Science and Technology Policy’s National Science and Technology Council (NSTC) convened the Federal Government’s Interagency Working Group on Language and Communication. Led by co-chairs from the Department of Education and the Department of Defense, representatives from 13 different federal agencies developed a report of current and recent federal investments in language and communication R & D activities.

This investment is discussed across four broad areas:

  • Knowledge and Processes Underlying Language and Communication;
  • Language and Communication Abilities and Skills;
  • Using Language and Communication; and
  • Language and Communication Technologies.

In addition, the report describes the types of current R & D activities in these areas, and provides programmatic recommendations for key areas of investment and collaboration in language and communication research going forward. 

On behalf of the working group, IES is gathering information from a wide community interested in language and communication R & D through a recently released request for information (RFI).  The purpose of this RFI is to assist the working group in its efforts to further improve coordination and collaboration of R & D agendas related to language and communication across the Federal Government. If you are interested in submitting a response to the RFI, please do so by the deadline of December 30, 2016.

Written by Elizabeth Albro, Associate Commissioner of Teaching and Learning, National Center for Education Research