IES Blog

Institute of Education Sciences

Building Evidence: What Comes After an Efficacy Study?

Over the years, the Institute of Education Sciences (IES) has funded over 300 studies across its research programs that evaluate the efficacy of specific programs, policies, or practices. This work has contributed significantly to our understanding of the interventions that improve outcomes for students under tightly controlled or ideal conditions. But is this information enough to inform policymakers’ and practitioners’ decisions about whether to adopt an intervention? If not, what should come after an efficacy study?

In October 2016, IES convened a group of experts for a Technical Working Group (TWG) meeting to discuss next steps in building the evidence base after an initial efficacy study, and the specific challenges that are associated with this work. TWGs are meant to encourage stakeholders to discuss the state of research on a topic and/or to identify gaps in research.  

Part of this discussion focused on replication studies and the critical role they play in the evidence-building process. Replication studies are essential for verifying the results of a previous efficacy study and for determining whether interventions are effective when certain aspects of the original study design are altered (for example, testing an intervention with a different population of students). IES has supported replication research since its inception, but there was general consensus that more replications are needed.

TWG participants discussed some of the barriers that may be discouraging researchers from doing this work. One major obstacle is the idea that replication research is somehow less valuable than novel research—a bias that could be limiting the number of replication studies that are funded and published. A related concern is that the field of education lacks a clear framework for conceptualizing and conducting replication studies in ways that advance evidence about beneficial programs, policies and practices (see another recent IES blog post on the topic).

IES provides support for studies to examine the effectiveness of interventions that have prior evidence of efficacy and that are implemented as part of the routine and everyday practice occurring in schools without special support from researchers. However, IES has funded a relatively small number of these studies (14 across both Research Centers). TWG participants discussed possible reasons for this and pointed out several challenges related to replicating interventions under routine conditions in authentic education settings. For instance, certain school-level decisions can pose challenges for conducting high-quality effectiveness studies, such as restricting the length that interventions or professional development can be provided and choosing to offer the intervention to students in the comparison condition. These challenges can result in findings that are influenced more by contextual factors rather than the intervention itself. TWG participants also noted that there is not much demand for this level of evidence, as the distinction between evidence of effectiveness and evidence of efficacy may not be recognized as important by decision-makers in schools and districts.

In light of these challenges, TWG participants offered suggestions for what IES could do to further support the advancement of evidence beyond an efficacy study. Some of these recommendations were more technical and focused on changes or clarifications to IES requirements and guidance for specific types of research grants. Other suggestions included:

  • Prioritizing and increasing funding for replication research;
  • Making it clear which IES-funded evaluations are replication studies on the IES website;
  • Encouraging communication and partnerships between researchers and education leaders to increase the appreciation and demand for evidence of effectiveness for important programs, practices, and policies; and
  • Supporting researchers in conducting effectiveness studies to better understand what works for whom and under what conditions, by offering incentives to conduct this work and encouraging continuous improvement.

TWG participants also recommended ways IES could leverage its training programs to promote the knowledge, skills, and habits that researchers need to build an evidence base. For example, IES could emphasize the importance of training in designing and implementing studies to develop and test interventions; create opportunities for postdoctoral fellows and early career researchers to conduct replications; and develop consortiums of institutions to train doctoral students to conduct efficacy, replication, and effectiveness research in ways that will build the evidence base on education interventions that improve student outcomes.

To read a full summary of this TWG discussion, visit the Technical Working Group website or click here to go directly to the report (PDF).

Written by Katie Taylor, National Center for Special Education Research, and Emily Doolittle, National Center for Education Research

What Are the Payoffs to College Degrees, Credentials, and Credits?

The Center for Analysis of Postsecondary Education and Employment (CAPSEE) is an IES-funded Research and Development Center that seeks to advance knowledge regarding the link between postsecondary education and the labor market. CAPSEE was funded through a 2011 grant from the National Center for Education Research (NCER) and is in the process of completing its work. CAPSEE will hold a final conference to discuss its findings on April 6 & 7 in Washington, DC.

Recently, Tom Bailey (pictured), Director of the Community College Research Center, Columbia University, Teachers College, and the Principal Investigator for CAPSEE, answered questions from James Benson, the NCER Program Officer for the R & D center.

Can you describe some of the original goals of CAPSEE?

We were especially interested in the economic benefits of a college education for community college students, including those who complete awards (Associate’s degrees or certificates) and those who do not, as well as those who transfer to four-year colleges. We were also interested in differences in earnings by field of study. When we started CAPSEE in 2012 there were a lot of studies that used survey datasets to look, in general, at the returns to completing a Bachelor’s degree. The CAPSEE approach was to use large-scale statewide databases and follow college students over time, to look in detail at their earnings before, during, and after college.

In addition, CAPSEE researchers sought to examine two key policy issues. One was how financial aid and working while enrolled affect students’ performance in college and their labor market outcomes. The other was whether for-profit colleges help students get better jobs.

You have synthesized findings from analyses in six states. What are your main findings?

We found that, in general, Associate’s degrees have good returns in the labor market; they’re a good investment for the individual and for society. However, there is quite a bit of variation in returns by program. For students in Associate degree programs primarily designed to prepare them for transfer to a four-year college, if they don’t transfer, their degrees will not be worth very much. But when they complete vocational degrees, especially in health-related fields, the earnings gains are usually strong and persistent (and robust to how we estimated them). Also, we did a lot of research on certificates, credentials that many see as the best fit for students on the margin of going to college. We found benefits to students who completed certificates, again especially in fields that directly relate to an occupation or industry. And finally, we examined outcomes for students who enrolled and took courses without attaining a degree or certificate. We found that their after-college earnings increased in proportion to the number of credits they earned.

"The fundamental policy implication is that college is a good investment."

What do you see as the key policy implications of these findings?

The fundamental policy implication is that college is a good investment. This merits emphasis because there are repeated critiques of college in terms of how much it costs and how much debt students accumulate. That said policymakers do need to think about the value of each postsecondary program. Even within the same institution, programs have very different outcomes. Yet on average, attending college for longer and attaining more credits has beneficial effects. Policymakers should see this evidence as supporting public and private investments in college.

What did you discover about the relationships between financial aid, college outcomes, and labor market outcomes?

In an era of tight public resources, the effectiveness of financial aid policy is a crucial issue. Financial aid does help students persist in college, but one way to promote greater effectiveness is through academic performance standards for students receiving federal financial aid. These have existed in the federal need-based aid programs for nearly 40 years, in the form of Satisfactory Academic Progress (SAP) requirements. These have not received much attention. Our research on SAP suggests such policies have heterogeneous effects on students in the short term: they increase the likelihood that some students will drop out, but appear to motivate higher grades for students who remain enrolled. After three years, however, the negative effects dominate. Though it has little benefit for students in the long term, SAP policy appears to increase the efficiency of aid expenditures because it discourages students who have lower-than-average course completion rates from persisting. But the policy also appears to exacerbate inequality in higher education by pushing out low-performing, low-income students faster than their equally low-performing, higher-income peers.

Many students work while in college.  Does this seem to help or hurt students in the long run?

Our research found that for Federal Work-Study (FWS) participants who would have worked even in the absence of the program, FWS reduces hours worked and improves academic outcomes but has little effect on post-college employment outcomes. For students who would not have worked, the effects are reversed: the program has little effect on graduation, but a positive effect on post-college employment.  Results are more positive for participants at public institutions, who tend to be lower income than participants at private institutions. Our findings suggest that better targeting to low-income and lower-scoring students could improve FWS outcomes. This is consistent with much of the CAPSEE research—you need more detail and specificity to really understand the relationship between education and employment and earnings.

What did you learn about credentials from for-profit institutions?

Our findings on students at for-profit colleges were quite pessimistic. Although enrollment in for-profit colleges grew significantly after 2000, the sector has been declining during the last two years, as evidence on inferior outcomes – particularly with regard to student debt – emerged. In general, our researchers found that for-profit students have worse labor market outcomes than comparable community college students although in some cases the difference is not statistically significant. Our evidence suggests that these colleges need to be monitored to ensure they are delivering a high-quality, efficient education.

You are holding the final CAPSEE conference in April. What do you hope people will get out of it?

At the conference, we will focus on several important and controversial policy questions related to higher education: 

  • Have changes in tuition and the labor market created conditions in which college is not worth it for some students, contributing to an unsupportable increase in student debt? 
  • Has higher education contributed to inequality rather than promoting economic mobility? 
  • Is continued public funding of college a worthwhile investment? 
  • Should public funding be used only for some programs of study?  
  • What are the arguments for and against making community college free?
  • Can changes in the operations and functioning of colleges change the return on investment from a college education for both the individual and society?
  • How important should information on earnings outcomes be for accreditation decisions and/or for eligibility of students to receive financial aid? 

At the conference participants will have the opportunity to discuss and learn about these issues drawing on five years of CAPSEE research as well as input from other experts.

A Growing Body of Research on Growth Mindset

Growth mindset is the belief that we can grow our intelligence by working hard at it and the idea has attracted a lot of interest in the education world in recent years. There have been best-selling books and many magazine and newspaper articles written about the power of a growth mindset.

In a recent national survey*, nearly all (98 percent) of 600 K-12 teachers said they think that a growth mindset improves their own teaching and helps their students learn. However, only 20% reported confidence in actually being able to help their students develop a growth mindset. This disparity highlights a need for additional research and development of growth-mindset based interventions, as well as research to understand how to best optimize implementation and outcomes.

For more than a decade, the Institute of Education Sciences (IES) has supported research on growth mindset. This includes a set of basic research studies to test a theory about growth mindset, an R&D project to build a technology-based growth mindset intervention, and an efficacy study to evaluate the impact of that intervention.

With a 2002 award from the IES Cognition and Student Learning Program (as well as grants from private foundations), researchers at Columbia and Stanford Universities conducted basic research to test and grow the growth mindset theory. This research provided a foundation for future R&D to develop school-based interventions focusing on applying growth mindset to student learning.

With a 2010 award from the ED/IES SBIR program, small business firm Mindset Works developed a web-based intervention to support teachers and grade 5 to 9 students in applying a growth mindset to teaching and learning. The Brainology intervention (pictured below) includes 20 animated interactive lessons and classroom activities for students on how the brain works and how it can become smarter and stronger through practice and learning. The intervention also teaches students specific neuroscience-based strategies to enhance attention, engagement, learning, and memory, and to manage negative emotions.

Brainology includes support materials for teachers to help them integrate the program and growth mindset concepts more generally into their daily activities at school. It is currently being used in hundreds of schools around the country.

And through a 2015 award from the Social and Behavioral Context for Academic Learning Program, researchers are now studying the efficacy of Brainology to improve students’ growth mindset and academic learning. In this four-year study, sixth- and seventh-grade science teachers are randomly assigned to either implement the program along with their school’s regular science curriculum or  continue with the regular science curriculum alone. Impacts of the growth mindset program on student mindsets and achievement (grades and test scores) are being measured in the early spring of the implementation year and in the fall of the following school year.
 
Follow us on Twitter and Facebook, or stay tuned to this blog, for more information about these and other research projects related to growth mindset.
 

Written by Emily Doolittle, NCER’s team lead for Social and Behavioral Research, and Ed Metz, ED/IES SBIR Program Manager

* - The survey indicates that growth mindset is of high interest to the general public and the education community. However, the Institute of Education Sciences was not involved in this survey and has not reviewed the methodology or results. 

IES Grantees Receive Prestigious Presidential Award

President Obama has named two Institute of Education Sciences (IES) grantees as recipients of the prestigious Presidential Early Career Awards for Scientists and Engineers (PECASE). The PECASE awards are the highest honor given by the U.S. Government to science and engineering professionals in the early stages of their independent research careers.

Daphna Bassok and Shayne Piasta, IES grantees who received presidential award

Daphna Bassok (left in picture), of the Curry School of Education at the University of Virginia, and Shayne Piasta (right in picture), of the College of Education and Human Ecology at The Ohio State University, were nominated for the award by the leadership of IES. They are two of 102 award recipients announced by the White House on January 9, 2017.

The awards, established by President Clinton in 1996, are coordinated by the Office of Science and Technology Policy within the Executive Office of the President. The White House said: “the awardees are selected for their pursuit of innovative research at the frontiers of science and technology and their commitment to community service as demonstrated through scientific leadership, public education, or community outreach.”

Bassok and Piasta have both served as principal investigators, and as co-principal investigators on projects funded through IES’ National Center for Education Research (NCER). Both are committed to understanding and improving early childhood education, and work closely with state systems to answer critical questions as the states roll out systemic changes to the provision of early childhood education.

Dr. Bassok is an associate professor of education and public policy at the University of Virginia and is also the associate director of EdPolicyWorks a joint collaboration between the Curry School of Education and the Frank Batten School of Leadership and Public Policy. Her research addresses early childhood education policy, with a particular focus on the impacts of policy interventions on the well-being of low-income children. Dr. Bassok is currently the principal investigator of Building State-wide Quality Rating Strategies for Early Childhood System Reform: Lessons From the Development of Louisiana's Kindergarten Readiness System, and was co-principal investigator for a project which examined links between policy and availability of early childhood care and education in the United States.

Dr. Piasta is an associate professor of reading and literature in early and middle childhood in the Department of Teaching and Learning at Ohio State. She also is a faculty associate for the Crane Center for Early Childhood Research and Policy. Her research focuses on early literacy development and how it is best supported during preschool and elementary years. Dr. Piasta is the principal investigator of the Evaluation of the Effectiveness of the Ohio Department of Education's Literacy Core Curriculum for Early Childhood Educators. She has also been co-principal investigator on several other IES-funded projects, including The Language Bases of Reading Comprehension, one of the six Reading for Understanding Research Initiative projects. Dr. Piasta received her doctoral training as an IES Predoctoral Fellow at Florida State University.

Written by Elizabeth Albro, Associate Commissioner for Teaching and Learning, NCER

Pictures courtesy of grantees

Improving Research on the Forgotten ‘R’

Writing is often labeled as the “forgotten ‘R,’” because the other R’s—reading and ‘rithmetic—seem to garner so much attention from educators, policymakers, and researchers. Yet, we know writing is a critical skill for communication and for success in school and in career. Writing in middle and high school can be especially important, because secondary grades are where students are expected to have mastered foundational skills like handwriting and move on to the application of these skills to more complex compositions.

IES has been funding research on writing since its inception in 2002, but compared to research on reading, not much work has been done in this critical area, especially writing in middle and high schools. In an effort to learn more about the state of the field of writing in secondary schools and the areas of needed research, IES brought together 13 experts on secondary writing for a Technical Working Group (TWG) meeting in September. During the full-day meeting, TWG participants shared their thoughts and expertise on a variety of topics including: argumentative writing, methods of engaging adolescents in writing, how best to help struggling writers including English learners and students with or at risk for disabilities, and assessment and feedback on writing.

Argumentative writing requires students to explore a topic, collect and evaluate evidence, establish a position on a topic, and consider alternative positions. In middle and high schools, argumentative writing often occurs in content area classrooms like science and history. TWG participants discussed the importance of research to understand how argumentative writing develops over time and how teachers contribute to this development.

Teaching writing to students with or at risk for disabilities and English learners can be challenging when the focus of secondary schools is often on content acquisition and not on improving writing skills. English learners are typically grouped together and receive the same instruction, but little is known about how writing instruction may need to be differentiated for students from different language backgrounds. Additionally, the TWG participants discussed the need to investigate the potential for technology to help with instruction of students who struggle with writing, and the importance to addressing the negative experiences these students have with writing that may discourage them from writing in the future.

It is also important to make sure all students are engaged and motivated to write. Some middle and high school students  may not want to participate in writing or may have internalized beliefs that they are not good at it. TWG participants discussed the need to consider teaching students that writing abilities can be changed, and that introducing new audiences or purposes for writing may motivate students to write. Finally, the group talked about the importance of allowing middle and high school students to write about topics of their own choosing.

Assessing the writing quality of middle and high school students is difficult, because what counts as good writing is often subjective. Technology may offer some solutions, but TWG participants emphasized that it is unlikely that computers will be able to do this task well entirely on their own. Regardless, the TWG participants were in agreement that there is a need for the development of quality writing measures for use both by teachers and by researchers.  Teachers may feel pressure to provide detailed feedback on students’ writing, which can be time-consuming. TWG participants argued that self-assessment and peer feedback could relieve some of the pressure on teachers, but research is needed to understand what kind of feedback is best for improving writing and how to teach students to provide useful feedback.

A full summary of the TWG can be found on the IES website. It’s our hope this conversation provides a strong framework for more research on ‘the forgotten R.’

POSTSCRIPT: Our colleagues at the What Works Clearinghouse recently published an Educator’s Practice Guide, “Teaching Secondary Students to Write Effectively.” It includes three research-based recommendations for improving writing for middle and high school students.

Written by Becky McGill-Wilkinson, National Center for Education Research, and Sarah Brasiel, National Center for Special Education Research