IES Blog

Institute of Education Sciences

Making the WWC Open to Everyone by Moving WWC Certification Online

In December 2016, the What Works Clearinghouse made a version of its online training publicly available through the WWC Website. This enabled everyone to be able to access the Version 3.0 Group Design Standards reviewer training to learn about the standards and methods that the WWC uses. While this was a great step to increase access to WWC resources, users still had to go through the 1 ½ day, in-person training to become a WWC certified reviewer.

To continue our efforts to promote access and transparency and make our resources available to everyone, the WWC has now moved all of its group design training to be online. Now everyone will have access to the same training and certification tests. This certification is available free of charge and is open to all users. It is our hope that this effort will increase the number of certified reviewers and help increase general awareness about the WWC.

Why did the WWC make these resources publicly available? As part of IES’s effort to increase access to high quality education research, we wanted to make it easier for researchers to use our standards. This meant opening up training opportunities and offering training online was a way to achieve this goal while using limited taxpayer resources most efficiently.

The online training consists of 9 modules. These videos feature an experienced WWC instructor and use the same materials that we used in our in-person courses, but adapted to Version 4.0 of the Group Design Standards. After completing the modules, users will have the opportunity to download a certificate of completion, take the online certification test, or go through the full certification exam.

Becoming a fully certified reviewer will require users to take a multiple choice online certification test and then use the new Online SRG application to conduct a full review using the same tools that the WWC team uses. The WWC team will then grade your exam to make sure you fully understand how to apply the Standards before certifying you to review for the Clearinghouse.

Not interested in becoming a certified reviewer? Online training still has several benefits. Educators can embed our videos in their course websites and use our training materials in their curricula. Researchers can use our Online SRG tool with their publications to determine a preliminary rating and understand what factors could cause their study to get the highest rating. They could also use the tool to use when conducting a systematic evidence review.

Have ideas for new resources we could make available? Email your ideas and suggestions to Contact.WWC@ed.gov!

by Erin Pollard, WWC Project Officer

 

Improving the WWC Standards and Procedures

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

Gathering Public Input to Help IES Improve

UPDATED SEPTEMBER 5, 2017 

The Institute of Education Sciences is committed to continuous improvement and this includes gathering public input on our work and our resources. Right now, we are seeking feedback on two important aspects of our work:

  • Two of our research goals, Efficacy and Replication and Effectiveness; and
  • Revisions to the What Works Clearinghouse Standards and Procedures handbooks.

Brief overviews of these opportunities are below, with links to where you can get more information and how to submit input. And, as always, if you have thoughts or ideas on how IES can better serve the field, please email us at Contact.IES@ed.gov.

IES Research Goals

IES is seeking input on how we can improve our education and special education research programs, specifically around two of our five research goals—Efficacy and Replication (Goal 3), and Effectiveness (Goal 4). We want to know if these goals, as currently configured, are meeting the needs of the field and whether we should consider changes that would support more replication and effectiveness studies.

The request for feedback comes after IES convened a group of experts to discuss what should come after an efficacy study. This Technical Working Group met last fall and looked at the replication and effectiveness studies that IES has funded over the years and made suggestions on actions IES could take to increase the visibility and support of replication studies, encourage more effectiveness research, and further our understanding of causal mechanisms, variability in impacts, and implementation factors. We shared some of the findings and suggestions in a blog post earlier this year and posted a summary of the working group’s discussion on the IES website (PDF).

Please take a few moments to read the Invitation for Public Comment letter to see the specific questions we are seeking to answer, and send your input and ideas to Comments.Research@ed.gov. We ask that you respond by Monday, October 2, 2017.

What Works Clearinghouse Handbooks

IES is also seeking feedback on revisions to the What Works Clearinghouse (WWC) Procedures and Standards Handbooks. The handbooks describe how the WWC reviews effectiveness research to determine what works in public education.

The proposed Handbooks (WWC Standards and WWC Procedures will now have separate handbooks) have been developed by the WWC in consultation with experts and are made available to users in draft form as part of the process for updating WWC standards. The handbooks can be reviewed on the WWC website and any comments can be sent to contact.wwc@ed.gov. Feedback is requested by August 30. (SEPT. 5 UPDATE: The deadline for submitting feedback has passed although questions and ideas are welcome at the same address.) 

The revisions to WWC handbooks are part of the WWC's ongoing work to increase transparency, refine its processes, develop new standards, and create new products. In fact, the WWC recently launched a new product that was developed based on public input.

For the latest IES, follow us on Twitter and Facebook, and receive email updates through the IES News Flash.

Compiled by Dana Tofig, Communications Director, IES

Persistence Pays Off: Introducing the New WWC Website

By Ruth Curran Neild, Delegated Director, IES

Stick with it, teachers tell their students.  Don’t give up when the going gets tough. Important work is often difficult. Keep at it.

More than a decade ago, it took this kind of dogged determination to launch an entity that would review education research against rigorous standards—the What Works Clearinghouse (WWC).   There were some skeptics, but also many supporters. And year after year, the WWC persisted, building out new topic areas, adding systematic reviews and practice guides, and identifying more interventions with positive effects on student outcomes.

Over time, WWC standards began to influence the field, with researchers designing studies that would meet its rigorous expectations. At the same time, researchers provided input that helped us improve WWC and its standards. Now, there are about 1,000 effectiveness studies available that meet our high bar for research design.  

By 2014, there was so much content in the WWC and so many people using the WWC website that it was time to retool our online presence and make our resources easier to find.

So IES staff and WWC contractors undertook an extensive project to create a more flexible database and intuitive search engine that meets a variety of user needs. A new relational database was built from the ground up, using study review data previously kept in thousands of different spreadsheets. Focus groups and user testing allowed us to identify the most important functionalities and best design features.

After more than two years of work, we launched the new WWC website today (Sept. 13), with a much-improved Find What Works feature that makes it easier to identify interventions, programs, and policies have improved student outcomes. (The video below can help you learn how to navigate the new site). The new WWC site has something for all types of users:

Do you want just top-level information about a program’s effects?  We can give you that.

Do you want to dig into the details of a particular study, including outcome domains, population, geographic context, and implementation?  We can give you that, too.

Do you want a quick assessment of whether a study was conducted with students like yours?  Find What Works can do that.

Do you want an easy way to compare the research on interventions?  Yes, absolutely – that’s a new feature.  

Do you want to see a list of randomized controlled trials (RCTs) that meet WWC standards and found at least one statistically significant positive effect?  No problem.

This is a proud day indeed for the What Works Clearinghouse, but there’s more to come.  We are always working on new resources for our users. In the next few months, look for a new practice guide on teaching writing in grades 6-12, as well as online WWC reviewer training, which we hope will help to meet the high demand for this credential. And we are continuing to expand our topic areas, including reviews of studies of postsecondary education programs to identify those that are showing promise for improving student outcomes.  

A lot of hard work and persistence went into this project from many people, including IES staff and the teams from Mathematica Policy Research and Sanametrix that worked on the site. I want to say “thank you” to them all. I also want to thank the many people who have been a part of the WWC over the years; including IES team members, study reviewers, and contractors. They have helped get us to this point today.  

Most of all, I want to thank the hundreds of thousands of educators, researchers, and decision makers who visit and use the WWC each year. We are glad that the What Works Clearinghouse has been a part of your work and hope it will continue to be a go-to resource for many years to come. 

 

The What Works Clearinghouse Goes to College

By Vanessa Anderson, Research Scientist, NCEE

The What Works Clearinghouse (WWC) was founded in 2002 and, in its first decade, focused mainly on reviewing studies of programs, policies, products and practices—or interventions—for improving student outcomes in pre-K, elementary and secondary schools. But in 2012, the WWC broadened its focus and has been using rigorous standards to review studies of interventions designed to increase the success of students in postsecondary education.

This week, the WWC launches a new topic—Supporting Postsecondary Success—and it is a good time to look at the work we’re doing, and will do, in the postsecondary area. 

The WWC postsecondary topic area includes reviews of studies on a wide range of interventions, including learning communities, summer bridge programs, multi-faceted support programs, academic mentoring, and interventions that aim to reduce performance anxiety. As of today, 294 postsecondary studies have been reviewed by the WWC. Those reviews are summarized in six Intervention Reports, 25 Single Study Reviews, and four Quick Reviews. And there’s much more in the works!  For instance, a WWC Educator’s Practice Guide that includes strategies for supporting students in developmental education is planned for publication later this year. (Learn more about Practice Guides)

Identifying Studies for Review

In the postsecondary topic area, there are currently three main ways that studies are identified by the WWC for review.

The first is studies that are reviewed for WWC Intervention Reports. All WWC Intervention Reports use a systematic review process to summarize evidence from all available studies on a given intervention. The WWC conducts a broad search for all publicly available studies of interventions that are related to the topic. This process often identifies hundreds of studies for review. The effectiveness studies are then reviewed against WWC standards. Only the highest quality studies are summarized in an Intervention Report.

We released two new intervention reports this week as part of our new Supporting Postsecondary Success topic. You can view the new Intervention Reports on Summer Bridge programs and first-year experience courses on the WWC website.

The second way that studies are reviewed by the WWC is through Quick Reviews, which are performed on studies that have received a great deal of media attention. In these reports, the WWC provides a brief description of the study, the author-reported results, and a study rating. We like to think of Quick Reviews as a way to help people decide whether to fully believe the results of a study, based on the research design and how the study was conducted. For example, we released a quick review earlier this month that focused on a study of computer usage and student outcomes for a class at the U.S. Military Academy at West Point.

Finally, the WWC reviews postsecondary studies submitted as supporting evidence for discretionary grant competitions funded by the U.S. Department of Education, such as the Strengthening Institutions Program, First in the World and TRIO Student Support Services. These grant competitions require applicants to submit studies as evidence of the effectiveness of the interventions they propose to implement. The WWC reviews these studies and includes the results of those reviews in our database.

If you want to see all the studies on postsecondary interventions that have been reviewed by WWC you can check out—and download—the Reviewed Studies Database. In the “Topic Areas” dropdown menu, just select “Postsecondary,” and then easily customize the search by rating, publication type, and/or reasons for the review (such as a grant competition).  

For more information, visit the WWC postsecondary topic area on the website. To stay up-to-date on WWC news, information, and products, follow us on Facebook, Twitter and sign up for the WWC newsflash!