IES Blog

Institute of Education Sciences

Should ESSA Evidence Definitions and What Works Study Ratings be the Same? No, and Here's Why!

By Joy Lesnick, Acting Commissioner, NCEE

The Every Student Succeeds Act (ESSA), the new federal education law, requires education leaders to take research evidence into account when choosing interventions or approaches. ESSA  defines three “tiers” of evidence—strong, moderate, and promising—based on the type and quality of study that was done and its findings.  

Are the ESSA definitions the same as those of Institute of Education Sciences’ What Works Clearinghouse (WWC)?  Not exactly.  ESSA definitions and WWC standards are more like cousins than twins.

Like ESSA, the WWC has three ratings for individual studies – meets standards without reservations, meets standards with reservations, and does not meet standards. The WWC uses a second set of terms to summarize the results of all studies conducted on a particular intervention. The distinction between one study and many studies is important, as I will explain below.

You may be wondering: now that ESSA is the law of the land, should the WWC revise its standards and ratings to reflect the tiers and terminology described in ESSA?  Wouldn’t the benefit of making things nice and tidy between the two sets of definitions outweigh any drawbacks?

The short answer is no.

The most basic reason is that the WWC’s standards come from a decision-making process that is based in science and vetted through scholarly peer review, all protected by the independent, non-partisan status of the Institute of Education Sciences (IES). This fact is central to the credibility of the WWC’s work.  We like to think of the WWC standards as an anchor representing the best knowledge in the field for determining whether a study has been designed and executed well, and how much confidence we should have in its findings.

WWC Standards Reflect the Most Current Scientific Knowledge – and are Always Evolving

WWC standards were developed by a national panel of research experts. After nearly two years of meetings, these experts came to a consensus about what a research study must demonstrate to give us confidence that an intervention caused the observed changes in student outcomes.

Since the first WWC standards were developed over a decade ago, there have been many methodological and conceptual advances in education research. The good news is that the WWC is designed to keep up with these changes in science. As science has evolved, the WWC standards have evolved, too.

One example is the WWC’s standards for reviewing regression discontinuity (RD) design studies.  The first version of RD standards was developed by a panel of experts in 2012.  Since then, the science about RD studies has made so much progress that the WWC recently convened another panel of experts to update the RD standards. The new RD standards are now on the WWC website to solicit scholarly comment.  

When it Comes to Evidence, More is Better

The evidence tiers in ESSA set a minimum bar, based on one study, to encourage states, districts, and schools to incorporate evidence in their decision making. This is a very important step in the right direction.  But a one-study minimum bar is not as comprehensive as the WWC’s approach.

In science, the collective body of knowledge on a topic is always better than the result of a single study or observation. This is why the primary function of the WWC is to conduct systematic reviews of all of the studies on a program, policy, practice, or approach (the results of which are published in Intervention Reports like the one pictured here).

The results of individual studies are important clues toward learning what works. But multiple studies, in different contexts, with different groups of teachers and students, in different states, and with different real-world implementation challenges tell us much more about how well a program, policy, practice or approach works. And that, really, is what we’re trying to find out.

An Improved WWC Search Tool and Ongoing Support for States and Districts

One area where WWC will make changes is in how users find studies that have certain characteristics described in ESSA’s evidence tiers.  For the past 16 months, the WWC team has been hard at work behind the scenes to develop, code, and user-test a dramatically improved Find What Works tool.  We expect to release this tool, along with other changes to the WWC website, in fall 2016. (More on that in another post, but the picture below offers a sneak preview!)

These changes should further increase the utility of the WWC website, which already gets more than 300,000 hits each month and offers products that are downloaded hundreds of thousands of times each year.

We know that providing information on a website about evidence from rigorous research is just a first step.  States and districts may need additional, customized support to incorporate evidence into their decision-making processes in ways that are much deeper than a cursory check-box approach.

To meet that need, other IES programs are ready to help. For example, IES supports 10 Regional Educational Laboratories (RELs) that provide states and districts with technical support for using, interpreting, and applying research. At least two researchers at every REL are certified as WWC reviewers (meaning they have in-depth knowledge of the WWC standards and how the standards are applied), and every REL has existing relationships with states and districts across the nation and outlying regions. Because the RELs are charged with meeting the needs of their regions, every chief state school officer (or designee) sits on a REL Governing Board, which determines the annual priorities of the REL in that area.

As states prioritize their needs and identify ways to incorporate evidence in their decisions according to the new law, the WWC database of reviewed studies will provide the information they need, and the RELs will be ready to help them use that information in meaningful ways.

 

 

 

Building a Better RFA

The Institute of Education Sciences (IES) is committed to continuous improvement and that includes the process by which people apply for and access grants.

Since its authorization in 2002, IES’ research centers—the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER)—have  been making efforts to improve the Requests for Applications (RFAs) we put out each year. In this spirit, we have conducted surveys of applicants the past few years and used that feedback to improve the current RFAs.

In Fiscal Years (FY) 2014 and 2015, all Principal Investigators (PIs) who submitted an application to the Education Research Grants Program RFA (CFDA # 84.305A) or the Partnerships and Collaborations Focused on Problems of Practice or Policy Program RFA (CFDA # 84.305H) were contacted via e-mail and asked to participate in the web-based survey. In FY 2015, applicants to the Special Education Research Grants Program (CFDA #324A) were included in the survey request.  The response rates were good for all surveys:

Grant Program FY 2014 FY 2015
Education Research Grants Program 62% 66%
Partnerships and Collaborations Program 59% 73%
Special Education Research Grants n/a 55%

 

Survey respondents generally provided positive feedback in both years. Most respondents indicated they felt the RFAs were clear and helpful, though there were some areas that generated some confusion and criticism.  For example, in FY 2014:

  • Applicants to the Education Research Grants program thought it was inconvenient to have to refer to two separate documents, the RFA and the Application Submission Guide, in order to complete their application.
  • Applicants to the Partnerships and Collaborations program reported some confusion about the distinction between partnership activities and research activities.

In response to the FY 2014 RFA survey results, the Institute made a number of changes. For the FY 2015 Education Research Grants and Special Education Research Grants, changes included combining the RFA and the Application Submission Guide into one document to provide all the necessary information in one place. According to responses from the FY 2015 RFA survey, this change was positively received. The majority of the respondents to the Education Research Grants and Special Education Research Grants surveys (n=398; 83%) reported that combining the RFA and Application Submission Guide was much better or somewhat better than having two separate documents. Overall, a majority of respondents (n = 161, 56%) felt the FY 2015 RFA was much better or somewhat better than in previous years, while another 43 percent felt that it was not better or worse.

For the Partnerships and Collaborations RFA, a number of changes were made to the FY 2015 RFA in response to the surveys. For example, the requirements for the research activities were disentangled from the requirements for the partnership in order to reduce redundancy within the application. Most respondents to the FY 2015 RFA survey (n = 53; 73%) felt this change made the RFA much better or somewhat better.

Respondents to the FY 2015 RFA survey also had some criticisms, and the Institute addressed those concerns in the FY 2016 RFAs. Specifically, in the Education Research Grants and Special Education Research Grants RFAs, more detail was added to the requirements for the dissemination plan and for the cost analysis plan.  For the Education Research Grants RFA, the language around research gaps was expanded to clarify that these are not priorities. Changes made in the Special Education Research Grant RFA in response to the feedback from the survey included streamlining application requirements related to student disability, age range or grade level, outcomes, and settings across its 11 research topics.  More details were added about the partnership tracking strategy (an area of confusion for many applicants) in the Partnerships and Collaborations FY 2016 RFA.

IES continues to strive toward improving RFAs and welcomes comments and suggestions for improvement. More information on the RFA results is available here: https://ies.ed.gov/ncer/projects/.

New FY 2017 RFAs are being posted on the IES Funding Opportunities page. If you have comments, please write to us at IESresearch@ed.gov.

By Christina Chhin (NCER), Rebecca McGill-Wilkinson (NCER), Phill Gagné (NCER) and Kristen Rhoads (NCSER)*

* Since this blog post was written, Dr. Rhoads has taken a position with the U.S. Department of Education's Office of Special Education and Rehabilitative Services. 

Improving Transitions: How NCSER-supported Work is Helping Prepare Students for Success

Talk of “transition” on Capitol Hill frequently focuses on political issues, such as the transition from one administration to the next. But on March 4, the conversation was about a very different type of transition—promoting positive outcomes for students with disabilities after high school.

For students with disabilities, post-high school goals are often similar to their non-disabled peers, but preparing them for success requires planning, support, and targeted interventions.

Over the past several years, the National Center for Special Education Research (NCSER) in the Institute of Education Sciences (IES) has funded research to innovate and develop as well as rigorously assess interventions that help students make successful transitions after high school.

A briefing on Capitol Hill was held this month to share recent research on transition for these students conducted by experts in the field. These experts have all received funding support from NCSER to help us better understand the transition challenges facing students with disabilities and to develop research-based programs and supports to increase the chances of success for students with disabilities.

"Young people with disabilities want the very same things as anyone else. A satisfying job, close relationships, a comfortable and safe place to live, a college degree, involvement in their community, friends they can count on, a chance to give something back, and an opportunity to be part of caring communities."

– Dr. Erik Carter, Vanderbilt University

Mary Wagner, of SRI International, began the briefing by talking about the National Longitudinal Transition Study-2 (known as the NLTS2), the more recent longitudinal study of the experiences of youth with disabilities as they transitioned from secondary school into postsecondary life over a 10-year period. Dr. Wagner presented findings that show that there has been progress in preparing students for, and engaging them in, postsecondary education. Additional academic courses, a paid job, and participation in transition planning and goal setting in high school were associated with increases in postsecondary education enrollment for these students after high school. However, the improvements have been uneven for some groups of students with disabilities and many challenges remain. For example, the rates of employment over time have not increased. 

David W. Test, of University of North Carolina-Charlotte, presented information about the innovative program “Communicating Interagency Relationships and Collaborative Linkages for Exceptional Students” or CIRCLES. This program involves three levels of interagency collaboration to promote positive outcomes for students with disabilities in secondary schools. The program connects students to more information and resources as well as provides mentoring support and partnerships. Ongoing research indicates the CIRCLES program is having a positive impact on student outcomes as compared to students receiving school services typically provided to support transition. In addition, participating students overwhelmingly agreed with the statement that they were “prepared for life after school” and their parents strongly agreed that they had “a better understanding of their child’s needs” and reported playing an active role in transition preparation.

The final two speakers discussed programs aimed at helping with transitions for students who face some of the greatest challenges.

Laurie E. Powers, of Portland State University, presented a research-based intervention program, “My Life,” for youth in foster care who also have disabilities. This program combines youth-directed coaching, workshops, and partnerships and mentoring to assist students in identifying goals and provide information and guidance they need to help them to experience success and to understand that they can achieve their goals. Many youth in foster care face extreme challenges in general: higher levels of unemployment, poverty, homelessness, abuse, and other mental health issues, and face incarceration rates of 10 times more than the general population. In addition, about 6 in 10 receive special education services and many also have developmental disabilities.

Research results have been positive. Students in the My Life program were found to be better prepared for postsecondary education and careers, and more were graduating from high school and fewer were homeless. After one year, postsecondary employment rates were up and rates of incarceration were down compared to the students who received services as usual.

Lastly, Erik Carter, of Vanderbilt University, presented research on improving workplace transitions for youth with intellectual disabilities (ID) in high school through a summer job support program. Although a disability does not predict aspirations, it does often predict post-high school experiences. Based on an analysis of data from the NLTS2, most youth with ID have a goal of employment, but only about 15 percent of all adults with ID are employed. A factor positively predicting outcomes for these students were the high expectations of those teaching them.

Project Summer embodies high expectations for these students and involves individual summer-focused transition planning, identification of community resources, and opportunities for youth to connect to community support and employment opportunities. Research indicates that the youth involved in Project Summer were much more likely to obtain employment or volunteer experiences in their community (66%) than their peers (19%) and all were paid above the minimum wage. This research also demonstrated that schools and communities have the capacity to support and promote the employment of youth with severe disabilities.

The briefing was sponsored by Senator Lamar Alexander, of Tennessee, Representative Suzanna Bonamici, of Oregon, and Representative Michael Honda, of California and was arranged by the Friends of IES, a group that advocates for education research. Certainly, there is much more work to be done to help students with disabilities successfully transition from high school and help them achieve their goals. But this month’s briefing demonstrated that progress is being made.

By Kimberley Sprague, Senior Research Scientist/Education Analyst, NCSER, and Dana Tofig, Communications Director, IES

 

Coming Soon to a Research Center Near You!

The summer movie season is still a couple of months away, but the trailers are already popping up on social media and in movie theaters, building anticipation and excitement. The Institute of Education Sciences (IES) has released its own preview, of sorts, this week.

One of the most important roles IES serves is as the engine for rigorous education research across a wide array of important areas. In the spring, IES will put out its Requests for Applications (RFA) for research grants and will begin accepting proposals from researchers and research organizations.

But an entry on the Federal Register, which went live this week, is sort of a preview for the competition. For instance, it includes the competition areas for each IES research center:

  • National Center for Education Research (NCER): Education research, education research training, statistical and research methodology in education, partnerships and collaboration focused on problems of practice or policy, low-cost, short-duration evaluations, and research networks.
  • National Center for Special Education Research (NCSER): Special education research, special education research training, and low-cost, short-duration evaluations.

The Federal Register entry also includes specific topic areas where NCER and NCSER will be accepting applications in each competition area. It’s a long list and some of it will look familiar to the research world. But there are some new areas of focus. For instance, NCER has introduced three special topics in the education research competition – arts in education, career and technical education, and systemic approaches to educating highly mobile students. And NCER will be accepting applications for a new network to that will study science teaching in elementary school classrooms in an effort to improve practices and student outcomes. This is similar in structure to the Early Learning Network that was announced earlier this year.  

NCSER applicants should note that in FY 2017, only applications that focus on teachers and other instructional personnel will be considered in the special education research competition.

Other information available on Federal Register includes the approximate range of funding in each competition area, information about the process, and the expected timeline for submitting applications. In the coming weeks, IES will be updating its funding opportunities website with the full RFA information and the research centers will be hosting webinars on a variety of topics related to RFAs. You can view previous webinars on the IES website.

To be notified of important dates in the RFA process and new webinars, sign up to receive IES newsflashes or follow IES on Twitter.

This grant competition may not be as funny as the Ghostbusters reboot, as exciting as the next Captain America movie, or as thrilling as latest chapter in the Jason Bourne series. But in the research world, it’s going to be a blockbuster! 

By Dana Tofig, Communications Director, IES

SREEing is Believing: Conference Shows Range of IES-Supported Work

By Ruth Curran Neild, Delegated Director, IES

Spring brings cherry blossoms to Washington D.C., turning the District into a city in bloom. The spring will also bring two major education research conferences to the city and, while these events may not offer breathtaking views like the cherry blossoms, the potential impact of the research being discussed is powerful.

The Society for Research on Educational Effectiveness (SREE) kicks things off this week (March 2 – 5), with the theme “Lost in Translation: Building Pathways from Knowledge to Action.” Of the 60-plus presentations, courses, and forums at the SREE conference, more than 40 involve research funded by the Institute of Education Sciences (IES), feature IES staff, or demonstrate products that have been developed with IES support. These sessions show the many ways that work supported by IES is being used to improve education across the country.

(The second conference, the American Education Research Association (AERA) Annual Meeting, will also include IES-supported work, but more on that later).

Research Studies, Large and Small

The core business of IES includes funding research studies on important topics of policy and practice. Funded work ranges from small pilot studies that test innovative approaches, to studies that are appropriately large – for example, evaluations that examine the impact of major Federal initiatives.  At one Thursday afternoon session, “Improving Mathematics Instructional Practice,” two of the studies being presented were funded through IES research centers—the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER). Later on Thursday, you can attend the “Reading for Understanding: New Findings from the Catalyzing Comprehension for Discussion and Debate Project,” in which all three studies being discussed were IES-funded.

And on Saturday, the “Evaluating the Scaling of Curriculum and Policy” session will feature three IES-funded studies, including a national, large-scale evaluation of the Teacher Incentive Fund pay-for-performance program, which is being conducted by our National Center for Education Evaluation and Regional Assistance (NCEE).

Partnership and Collaboration

Many of the sessions at SREE feature IES-supported work done through alliances and partnerships that bring researchers and practitioners together. Much of this work is being done through our Regional Educational Laboratories (RELs), which are a key way that we help connect research to policy and practice every day.

For example, on Thursday morning, a session on “Pathways to Algebra Success” will feature studies that grew out of a REL research alliance. For all three studies, the questions that were asked came from practitioners and policymakers in the field.

Friday morning, the “Making Meaning of Research Study Findings” session will focus on how you “translate research into action.” The session will cover four studies from REL research alliances across the country, from New England to Indiana to the state of Washington. The studies cover a broad array of important topics in education—the academic performance and reclassification of English learner students; the effectiveness of teacher evaluation systems; and college enrollment patterns of high school graduates.

(If you’re not familiar with the RELs, read this blog post by Joy Lesnick, acting commissioner of NCEE, which oversees the REL program).

Making Science Better, Making Results More Accessible

Another big bucket of IES work that will be featured at SREE is resources and tools that are improving the research field and making it easier for people to access and use research.

For instance, a workshop on Wednesday featured the IES-funded CostOut tool, which can be used to determine return on investment. Another workshop featured the “Generalizer,” a privately funded, web-based tool that improves generalizations from experiments and uses data from IES’s National Center for Education Statistics.

On Thursday afternoon, attendees can preview software intended to make it easier for states and districts to conduct randomized controlled trials and quasi-experiments.  At another session, there will be a demonstration of the new “Find What Works” tool to help practitioners and policymakers find effective programs in the What Works Clearinghouse.

On Friday morning, NCER Associate Commissioner Allen Ruby will be a part of a panel discussing plans for a new “Registry of Effectiveness Studies in Education,” which will be developed with IES grant support. Study registries can contribute to increased transparency in studies of what works.

There is terrific work going on to connect research to practice.  Don’t miss out -- follow us on Twitter at @IESResearch to learn more about IES-supported work at the SREE conference.