IES Blog

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

UPDATED SEPTEMBER 5, 2017 

The Institute of Education Sciences is committed to continuous improvement and this includes gathering public input on our work and our resources. Right now, we are seeking feedback on two important aspects of our work:

• Two of our research goals, Efficacy and Replication and Effectiveness; and
• Revisions to the What Works Clearinghouse Standards and Procedures handbooks.

Brief overviews of these opportunities are below, with links to where you can get more information and how to submit input. And, as always, if you have thoughts or ideas on how IES can better serve the field, please email us at Contact.IES@ed.gov.

IES Research Goals

IES is seeking input on how we can improve our education and special education research programs, specifically around two of our five research goals—Efficacy and Replication (Goal 3), and Effectiveness (Goal 4). We want to know if these goals, as currently configured, are meeting the needs of the field and whether we should consider changes that would support more replication and effectiveness studies.

The request for feedback comes after IES convened a group of experts to discuss what should come after an efficacy study. This Technical Working Group met last fall and looked at the replication and effectiveness studies that IES has funded over the years and made suggestions on actions IES could take to increase the visibility and support of replication studies, encourage more effectiveness research, and further our understanding of causal mechanisms, variability in impacts, and implementation factors. We shared some of the findings and suggestions in a blog post earlier this year and posted a summary of the working group’s discussion on the IES website (PDF).

Please take a few moments to read the Invitation for Public Comment letter to see the specific questions we are seeking to answer, and send your input and ideas to Comments.Research@ed.gov. We ask that you respond by Monday, October 2, 2017.

What Works Clearinghouse Handbooks

IES is also seeking feedback on revisions to the What Works Clearinghouse (WWC) Procedures and Standards Handbooks. The handbooks describe how the WWC reviews effectiveness research to determine what works in public education.

The proposed Handbooks (WWC Standards and WWC Procedures will now have separate handbooks) have been developed by the WWC in consultation with experts and are made available to users in draft form as part of the process for updating WWC standards. The handbooks can be reviewed on the WWC website and any comments can be sent to contact.wwc@ed.gov. Feedback is requested by August 30. (SEPT. 5 UPDATE: The deadline for submitting feedback has passed although questions and ideas are welcome at the same address.) 

The revisions to WWC handbooks are part of the WWC's ongoing work to increase transparency, refine its processes, develop new standards, and create new products. In fact, the WWC recently launched a new product that was developed based on public input.

For the latest IES, follow us on Twitter and Facebook, and receive email updates through the IES News Flash.

Compiled by Dana Tofig, Communications Director, IES

By Ruth Curran Neild, Delegated Director, IES

Stick with it, teachers tell their students.  Don’t give up when the going gets tough. Important work is often difficult. Keep at it.

More than a decade ago, it took this kind of dogged determination to launch an entity that would review education research against rigorous standards—the What Works Clearinghouse (WWC).   There were some skeptics, but also many supporters. And year after year, the WWC persisted, building out new topic areas, adding systematic reviews and practice guides, and identifying more interventions with positive effects on student outcomes.

Over time, WWC standards began to influence the field, with researchers designing studies that would meet its rigorous expectations. At the same time, researchers provided input that helped us improve WWC and its standards. Now, there are about 1,000 effectiveness studies available that meet our high bar for research design.  

By 2014, there was so much content in the WWC and so many people using the WWC website that it was time to retool our online presence and make our resources easier to find.

So IES staff and WWC contractors undertook an extensive project to create a more flexible database and intuitive search engine that meets a variety of user needs. A new relational database was built from the ground up, using study review data previously kept in thousands of different spreadsheets. Focus groups and user testing allowed us to identify the most important functionalities and best design features.

After more than two years of work, we launched the new WWC website today (Sept. 13), with a much-improved Find What Works feature that makes it easier to identify interventions, programs, and policies have improved student outcomes. (The video below can help you learn how to navigate the new site). The new WWC site has something for all types of users:

Do you want just top-level information about a program’s effects?  We can give you that.

Do you want to dig into the details of a particular study, including outcome domains, population, geographic context, and implementation?  We can give you that, too.

Do you want a quick assessment of whether a study was conducted with students like yours?  Find What Works can do that.

Do you want an easy way to compare the research on interventions?  Yes, absolutely – that’s a new feature.  

Do you want to see a list of randomized controlled trials (RCTs) that meet WWC standards and found at least one statistically significant positive effect?  No problem.

This is a proud day indeed for the What Works Clearinghouse, but there’s more to come.  We are always working on new resources for our users. In the next few months, look for a new practice guide on teaching writing in grades 6-12, as well as online WWC reviewer training, which we hope will help to meet the high demand for this credential. And we are continuing to expand our topic areas, including reviews of studies of postsecondary education programs to identify those that are showing promise for improving student outcomes.  

A lot of hard work and persistence went into this project from many people, including IES staff and the teams from Mathematica Policy Research and Sanametrix that worked on the site. I want to say “thank you” to them all. I also want to thank the many people who have been a part of the WWC over the years; including IES team members, study reviewers, and contractors. They have helped get us to this point today.  

Most of all, I want to thank the hundreds of thousands of educators, researchers, and decision makers who visit and use the WWC each year. We are glad that the What Works Clearinghouse has been a part of your work and hope it will continue to be a go-to resource for many years to come. 

By Vanessa Anderson, Research Scientist, NCEE

The What Works Clearinghouse (WWC) was founded in 2002 and, in its first decade, focused mainly on reviewing studies of programs, policies, products and practices—or interventions—for improving student outcomes in pre-K, elementary and secondary schools. But in 2012, the WWC broadened its focus and has been using rigorous standards to review studies of interventions designed to increase the success of students in postsecondary education.

This week, the WWC launches a new topic—Supporting Postsecondary Success—and it is a good time to look at the work we’re doing, and will do, in the postsecondary area. 

The WWC postsecondary topic area includes reviews of studies on a wide range of interventions, including learning communities, summer bridge programs, multi-faceted support programs, academic mentoring, and interventions that aim to reduce performance anxiety. As of today, 294 postsecondary studies have been reviewed by the WWC. Those reviews are summarized in six Intervention Reports, 25 Single Study Reviews, and four Quick Reviews. And there’s much more in the works!  For instance, a WWC Educator’s Practice Guide that includes strategies for supporting students in developmental education is planned for publication later this year. (Learn more about Practice Guides)

Identifying Studies for Review

In the postsecondary topic area, there are currently three main ways that studies are identified by the WWC for review.

The first is studies that are reviewed for WWC Intervention Reports. All WWC Intervention Reports use a systematic review process to summarize evidence from all available studies on a given intervention. The WWC conducts a broad search for all publicly available studies of interventions that are related to the topic. This process often identifies hundreds of studies for review. The effectiveness studies are then reviewed against WWC standards. Only the highest quality studies are summarized in an Intervention Report.

We released two new intervention reports this week as part of our new Supporting Postsecondary Success topic. You can view the new Intervention Reports on Summer Bridge programs and first-year experience courses on the WWC website.

The second way that studies are reviewed by the WWC is through Quick Reviews, which are performed on studies that have received a great deal of media attention. In these reports, the WWC provides a brief description of the study, the author-reported results, and a study rating. We like to think of Quick Reviews as a way to help people decide whether to fully believe the results of a study, based on the research design and how the study was conducted. For example, we released a quick review earlier this month that focused on a study of computer usage and student outcomes for a class at the U.S. Military Academy at West Point.

Finally, the WWC reviews postsecondary studies submitted as supporting evidence for discretionary grant competitions funded by the U.S. Department of Education, such as the Strengthening Institutions Program, First in the World and TRIO Student Support Services. These grant competitions require applicants to submit studies as evidence of the effectiveness of the interventions they propose to implement. The WWC reviews these studies and includes the results of those reviews in our database.

If you want to see all the studies on postsecondary interventions that have been reviewed by WWC you can check out—and download—the Reviewed Studies Database. In the “Topic Areas” dropdown menu, just select “Postsecondary,” and then easily customize the search by rating, publication type, and/or reasons for the review (such as a grant competition).  

For more information, visit the WWC postsecondary topic area on the website. To stay up-to-date on WWC news, information, and products, follow us on Facebook, Twitter and sign up for the WWC newsflash!

By Diana McCallum, Education Research Analyst, What Works Clearinghouse

It’s been more than a decade since the first What Works Clearinghouse reports were released and we have a wealth of information and resources that can help educators and leaders make evidence-based decisions about teaching and learning. Since 2005, the WWC has assessed more than 11,500 education studies using rigorous standards and has published hundreds of resources and guides across many content areas. (View the full version of the graphic to the right.) 

The WWC website has already received more than 1.7 million page views this year, but if you haven’t visited whatworks.ed.gov lately, here are five reasons you might want to click over:

1) We are always adding new and updated reviews. Multiple claims about programs that work can be overwhelming and people often lack time to sift through piles of research. That’s where the WWC comes in. We provide an independent, objective assessment of education research. For example, we have intervention reports that provide summaries of all of the existing research on a given program or practice that educators can use to help inform their choices.  In addition, when a new education study grabs headlines, the WWC develops a quick review that provides our take on the evidence presented to let you know whether the study is credible. In 2015, we added 43 publications to WWC and we’re adding more every month this year.

2) We’ve expanded our reach into the Postsecondary area. In late 2012, the WWC expanded its focus to include reviews of studies within the Postsecondary area to capture the emerging research on studies on a range of topics, from the transition to college to those that focus on postsecondary success.  To date, the WWC has reviewed over 200 studies on postsecondary programs and interventions, and this area continues grow rapidly. In fact, several Office of Postsecondary Education grant competitions add competitive priority preference points for applicants that submit studies that meet WWC standards. (Keep an eye out for a blog post on the postsecondary topic coming soon!)

3) You can find what works using our online tool. Wondering how to get started with so many resources at your fingertips? Find What Works lets you do a quick comparison of interventions for different subjects, grades, and student populations. Want to know more about a specific intervention? We’ve produced more than 400 intervention reports to provide you the evidence about a curriculum, program, software product, or other intervention for your classroom before you choose it.  Recently, we’ve added a feature that allows a user to search for interventions that have worked for different populations of students and in different geographic locations. As we mentioned in a recent blog post, the Find What Works tool is undergoing an even bigger transformation this September, so keep visiting!

4) We identify evidence-based practices to use in the classroom. The WWC has produced 19 practice guides that feature practical recommendations and instructional tips to help educators address common challenges. Practice guides (now available for download as ebooks) provide quick, actionable guidance for educators that are supported by evidence and expert knowledge within key areas.  Some of our guides now feature accompanying videos and brief summaries that demonstrate recommended practices and highlight the meaning behind the levels of evidence. The work of practice guides are also actively disseminated during Regional Educational Laboratory (REL) Bridge events. For instance, REL Southwest held a webinar on Teaching Math to Young Children, which was based on a WWC practice guide. For more information, read a previously published blog post on practice guides.

5) We compile information by topic. Our “Special Features” pages focus on common themes in education, such as tips for college readiness, information for heading back to school, and guidance for what works in early childhood education. These Special Features provide a starting point to access a variety of WWC resources related to a topic.

In the coming months, we’ll post other blogs that will explore different parts of the WWC and tell you about ongoing improvements. So keep visiting the What Works website or signup to receive emails when we release new reports or resources. You can also follow us on Facebook and Twitter.

The What Works Clearinghouse is a part of the National Center for Education Evaluation and Regional Assistance in the Institute of Education Sciences (IES), the independent research, evaluation, and statistics arm of the U.S. Department of Education. You can learn more about IES’ other work on its website or follow IES on Twitter and Facebook.