IES Blog

Institute of Education Sciences

The WWC Evidence Standards: A Valuable and Accessible Resource for Teaching Validity Assessment of Causal Inferences to Identify What Works

by Herbert Turner, Ph.D., President and Principal Scientist, ANALYTICA, Inc.

 

The WWC Evidence Standards (hereafter, the Standards) provide a detailed description of the criteria used by the WWC to review studies. The standards were first developed in 2002 by leading methodological researchers using initial concepts from the Study Design and Implementation Assessment Device (DIAD), an instrument for assessing the correspondence between the methodological characteristics and implementation of social science research and using this research to draw inferences about causal relationships (Boruch, 1997; Valentine and Cooper, 2008).  During the past 16 years, the Standards have gone through four iterations of improvement, to keep pace with advances in methodological practice, and have been through rigorous peer review. The most recent of these is now codified in the WWC Standards Handbook 4.0 (hereafter, the Handbook).

 

Across the different versions of the Handbook, the methodological characteristics of an internally valid study, designed to causally infer the effect of an intervention on an outcome, have stood the test of time. These characteristics can be summarized as follows: A strong design starts with how the study groups are formed. It continues with use of reliable and valid measures of outcomes, has low attrition if a randomized controlled trial (RCT), shows baseline equivalence (in the analysis sample) if a quasi-experimental design (QED), and has no confounds.

 

These elements are the critical components of any strong research design – and are the cornerstones of all versions of the WWC’s standards. That fact, along with the transparent description of their logical underpinning, is what motivated me to use Standards 4.0 (for Group Designs) as the organizing framework for understanding study validity in a graduate-level Program Evaluation II course I taught at Boston College’s Lynch School of Education.

 

In spring 2017, nine Master and four Doctoral students participated in this semester-long course. The primary goal was to teach students how to organize their thinking and logically derive internal validity criteria using Standards 4.0—augmented with additional readings from the methodological literature. Students used the Standards (along with the supplemental readings) to design, implement, analyze, and report impact evaluations to determine what interventions work, harm, or have no discernible effect (Mosteller and Boruch, 2002). The Standards Handbook 4.0 along with online course modules were excellent resources to augment the lectures and provide Lynch School students with hands on learning.

 

At the end of the course, students were offered the choice to complete the WWC Certification Exam for Group Design or take the instructor’s developed final exam. All thirteen students chose to complete the WWC Certification Exam. Approximately half of the students became certified. Many emailed me personally to express their appreciation for the (1) opportunity to learn a systematic approach to organizing their thinking about assessing the validity of causal inference using data generated by RCTs and QEDs, and (2) developing design skills that can be used in other graduate courses and beyond. The WWC Evidence Standards and related online resources are a valuable, accessible, and free resource that have been rigorously vetted for close to two decades. The Standards have few equals as a resource to help students think systematically, logically, and clearly about designing (and evaluating) a valid research study to make causal inferences about what interventions work in education and related fields.

 

References

Boruch, R. F. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage Publications.

Valentine, J.C., & Cooper, H. (2008), A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13(2), 130-149.

Mosteller, F., & Boruch, R. F. (2002). Evidence matters: Randomized trials in education research. Washington, D.C.: Brookings Institution Press.

Making the WWC Open to Everyone by Moving WWC Certification Online

In December 2016, the What Works Clearinghouse made a version of its online training publicly available through the WWC Website. This enabled everyone to be able to access the Version 3.0 Group Design Standards reviewer training to learn about the standards and methods that the WWC uses. While this was a great step to increase access to WWC resources, users still had to go through the 1 ½ day, in-person training to become a WWC certified reviewer.

To continue our efforts to promote access and transparency and make our resources available to everyone, the WWC has now moved all of its group design training to be online. Now everyone will have access to the same training and certification tests. This certification is available free of charge and is open to all users. It is our hope that this effort will increase the number of certified reviewers and help increase general awareness about the WWC.

Why did the WWC make these resources publicly available? As part of IES’s effort to increase access to high quality education research, we wanted to make it easier for researchers to use our standards. This meant opening up training opportunities and offering training online was a way to achieve this goal while using limited taxpayer resources most efficiently.

The online training consists of 9 modules. These videos feature an experienced WWC instructor and use the same materials that we used in our in-person courses, but adapted to Version 4.0 of the Group Design Standards. After completing the modules, users will have the opportunity to download a certificate of completion, take the online certification test, or go through the full certification exam.

Becoming a fully certified reviewer will require users to take a multiple choice online certification test and then use the new Online SRG application to conduct a full review using the same tools that the WWC team uses. The WWC team will then grade your exam to make sure you fully understand how to apply the Standards before certifying you to review for the Clearinghouse.

Not interested in becoming a certified reviewer? Online training still has several benefits. Educators can embed our videos in their course websites and use our training materials in their curricula. Researchers can use our Online SRG tool with their publications to determine a preliminary rating and understand what factors could cause their study to get the highest rating. They could also use the tool to use when conducting a systematic evidence review.

Have ideas for new resources we could make available? Email your ideas and suggestions to Contact.WWC@ed.gov!

by Erin Pollard, WWC Project Officer

 

Improving the WWC Standards and Procedures

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

Gathering Public Input to Help IES Improve

UPDATED SEPTEMBER 5, 2017 

The Institute of Education Sciences is committed to continuous improvement and this includes gathering public input on our work and our resources. Right now, we are seeking feedback on two important aspects of our work:

  • Two of our research goals, Efficacy and Replication and Effectiveness; and
  • Revisions to the What Works Clearinghouse Standards and Procedures handbooks.

Brief overviews of these opportunities are below, with links to where you can get more information and how to submit input. And, as always, if you have thoughts or ideas on how IES can better serve the field, please email us at Contact.IES@ed.gov.

IES Research Goals

IES is seeking input on how we can improve our education and special education research programs, specifically around two of our five research goals—Efficacy and Replication (Goal 3), and Effectiveness (Goal 4). We want to know if these goals, as currently configured, are meeting the needs of the field and whether we should consider changes that would support more replication and effectiveness studies.

The request for feedback comes after IES convened a group of experts to discuss what should come after an efficacy study. This Technical Working Group met last fall and looked at the replication and effectiveness studies that IES has funded over the years and made suggestions on actions IES could take to increase the visibility and support of replication studies, encourage more effectiveness research, and further our understanding of causal mechanisms, variability in impacts, and implementation factors. We shared some of the findings and suggestions in a blog post earlier this year and posted a summary of the working group’s discussion on the IES website (PDF).

Please take a few moments to read the Invitation for Public Comment letter to see the specific questions we are seeking to answer, and send your input and ideas to Comments.Research@ed.gov. We ask that you respond by Monday, October 2, 2017.

What Works Clearinghouse Handbooks

IES is also seeking feedback on revisions to the What Works Clearinghouse (WWC) Procedures and Standards Handbooks. The handbooks describe how the WWC reviews effectiveness research to determine what works in public education.

The proposed Handbooks (WWC Standards and WWC Procedures will now have separate handbooks) have been developed by the WWC in consultation with experts and are made available to users in draft form as part of the process for updating WWC standards. The handbooks can be reviewed on the WWC website and any comments can be sent to contact.wwc@ed.gov. Feedback is requested by August 30. (SEPT. 5 UPDATE: The deadline for submitting feedback has passed although questions and ideas are welcome at the same address.) 

The revisions to WWC handbooks are part of the WWC's ongoing work to increase transparency, refine its processes, develop new standards, and create new products. In fact, the WWC recently launched a new product that was developed based on public input.

For the latest IES, follow us on Twitter and Facebook, and receive email updates through the IES News Flash.

Compiled by Dana Tofig, Communications Director, IES

Persistence Pays Off: Introducing the New WWC Website

By Ruth Curran Neild, Delegated Director, IES

Stick with it, teachers tell their students.  Don’t give up when the going gets tough. Important work is often difficult. Keep at it.

More than a decade ago, it took this kind of dogged determination to launch an entity that would review education research against rigorous standards—the What Works Clearinghouse (WWC).   There were some skeptics, but also many supporters. And year after year, the WWC persisted, building out new topic areas, adding systematic reviews and practice guides, and identifying more interventions with positive effects on student outcomes.

Over time, WWC standards began to influence the field, with researchers designing studies that would meet its rigorous expectations. At the same time, researchers provided input that helped us improve WWC and its standards. Now, there are about 1,000 effectiveness studies available that meet our high bar for research design.  

By 2014, there was so much content in the WWC and so many people using the WWC website that it was time to retool our online presence and make our resources easier to find.

So IES staff and WWC contractors undertook an extensive project to create a more flexible database and intuitive search engine that meets a variety of user needs. A new relational database was built from the ground up, using study review data previously kept in thousands of different spreadsheets. Focus groups and user testing allowed us to identify the most important functionalities and best design features.

After more than two years of work, we launched the new WWC website today (Sept. 13), with a much-improved Find What Works feature that makes it easier to identify interventions, programs, and policies have improved student outcomes. (The video below can help you learn how to navigate the new site). The new WWC site has something for all types of users:

Do you want just top-level information about a program’s effects?  We can give you that.

Do you want to dig into the details of a particular study, including outcome domains, population, geographic context, and implementation?  We can give you that, too.

Do you want a quick assessment of whether a study was conducted with students like yours?  Find What Works can do that.

Do you want an easy way to compare the research on interventions?  Yes, absolutely – that’s a new feature.  

Do you want to see a list of randomized controlled trials (RCTs) that meet WWC standards and found at least one statistically significant positive effect?  No problem.

This is a proud day indeed for the What Works Clearinghouse, but there’s more to come.  We are always working on new resources for our users. In the next few months, look for a new practice guide on teaching writing in grades 6-12, as well as online WWC reviewer training, which we hope will help to meet the high demand for this credential. And we are continuing to expand our topic areas, including reviews of studies of postsecondary education programs to identify those that are showing promise for improving student outcomes.  

A lot of hard work and persistence went into this project from many people, including IES staff and the teams from Mathematica Policy Research and Sanametrix that worked on the site. I want to say “thank you” to them all. I also want to thank the many people who have been a part of the WWC over the years; including IES team members, study reviewers, and contractors. They have helped get us to this point today.  

Most of all, I want to thank the hundreds of thousands of educators, researchers, and decision makers who visit and use the WWC each year. We are glad that the What Works Clearinghouse has been a part of your work and hope it will continue to be a go-to resource for many years to come.