IES Blog

Institute of Education Sciences

Improving the WWC Standards and Procedures

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

Updating Our Recommendations to Prevent Dropping Out

By Dana Tofig, Communications Director, IES

Almost a decade ago, the What Works Clearinghouse (WWC) released Dropout Prevention, one of its first practice guides, which offered six recommendations for keeping students in school, based on the findings from high-quality research and best practices in the field. Since its release in August 2008, the practice guide has been downloaded thousands of times by practitioners across the country. In fact, the guide was downloaded more than 1,500 times in 2016, alone.

However, over the past decade, the research and knowledge base has grown in the area of dropout prevention, which is why the WWC decided to update this guide to reflect the latest evidence and information about keeping students in school and on track toward graduation.

This updated guide, Preventing Dropout in Secondary Schools, builds on the 2008 guide in two significant ways.

First, it reflects improvements in practices related to monitoring at-risk students, including advances in using early warning indicators to identify students at risk for dropping out. Secondly, it covers an additional nine years of research that were not a part of the previous guide. In fact, 15 of the 25 studies used to support the recommendations in this updated guide were published after the first guide was published. In addition, studies from the previous guide were reviewed again against current, more rigorous WWC evidence standards.

Preventing Dropout in Secondary Schools offers four evidence-based recommendations that can be used by schools and districts:

  • Monitor the progress of all students, and proactively intervene when students show early signs of attendance, behavior, or academic problems;
  • Provide intensive, individualized support to students who have fallen off track and face significant challenges to success;
  • Engage students by offering curricula and programs that connect schoolwork with college and career success and that improve students’ capacity to manage challenges in and out of school; and
  • For schools with many at-risk students, create small, personalized communities to facilitate monitoring and support.

Each of these recommendations includes specific strategies for implementation and examples of how this work is being done around the country (see one such example in the image to the right).

Like all of our practice guides, the recommendations were developed with a panel of educators, academics, and experts who brought a wealth of knowledge and experience to the process. Two of the panelists on this updated guide were also involved in the development of the first dropout prevention guide—Russell Rumberger, from University of California, Santa Barbara, and Mark Dynarksi, of Pemberton Research LLC. The other panelists for the new guide were Howard (Sandy) Addis, of the National Dropout Prevention Center and Network; Elaine Allensworth, from the University of Chicago; Robert Balfanz, from The Johns Hopkins University; and Debra Duardo, superintendent of the Los Angeles County Office of Education.

Please download this free guide today and let us know what you think. We would especially love to hear from people who are using the recommended strategies in schools. You can reach us through the WWC Help Desk or email us at Contact.IES@ed.gov

Putting Your Ideas into Action: Instructional Tips for Educators

By Christopher Weiss, Program Manager, What Works Clearinghouse

The What Works Clearinghouse (WWC) is always looking for ways to improve. We want it to be as easy as possible for our users to connect with the evidence they need, so they can make informed educational decisions.

Last year, we undertook a comprehensive, multi-faceted self-study. Through surveys, interviews, and focus groups, we asked a variety of different WWC users to tell us what we were doing well and, more importantly, what we could do better. (Click here if you’re interested in all the results.)

Some of the specific suggestions we received focused on the WWC Educator’s Practice Guides, which combine the best available research evidence and practitioner expertise on a topic to provide educators with strategies to use in their school or classroom. Based upon a review of the research literature and the guidance of a panel of nationally recognized experts, practice guides synthesize evidence and the wisdom of practitioners.

One particular suggestion that came from the self-study was to create a separate, stand-alone document with concise and specific information that a teacher or school would need to carry out some of a practice guide’s recommendations. It was a great suggestion – and we put it into action.

On July 25, we released our first Instructional Tips publication (PDF), which was created to help educators carry out the recommendations in the Improving Mathematical Problem Solving in Grades 4 through 8 practice guide. We provide tips for three of the Practice Guide’s five recommendations:

  • Assisting Students in Monitoring and Reflecting on the Problem-Solving Process;
  • Teaching Students to Use Visual Representations to Solve Problems; and
  • Helping Students Make Sense of Algebraic Notation.

As an example, for the recommendation on visual representations, we offer two instructional tips. First, we suggest that teachers demonstrate how to select the appropriate visual representation for the problem they are solving and we provide specific steps and examples for implementing this tip. Second, we suggest teachers use think-alouds and discussions to teach students how to represent problems visually and, again, provide specific steps and work examples. Here's one of the examples from the publication:

An accompanying document (PDF) to the Instructional Tips describes the evidence base that supports these recommended practices.

We are planning additional Instructional Tips publications down the road, but we want to hear from you first. If you have questions or ideas for how we can improve this resource, we’d love to hear them. Please send them through an email to the WWC Help Desk.

The Instructional Tips are just one of several ways we are working to improve the WWC. Over the past two years, we have redesigned our website and created a new Find What Works tool to make it easier for users to find the evidence they need. We have also increased our use of Facebook and Twitter to help us better connect with new audiences; published new briefs and held several webinars to explain WWC processes and resources; and have launched a new Reviews of Individual Studies database to give the field quicker access to the research we have reviewed. And all of this has been done while we continue to identify interventions, practices and programs that show evidence of improving student outcomes across a wide array of educational topics.

Stay up to date on new WWC products, events, and resources by signing up for the IES News Flash (under NCEE) and following us on Facebook and Twitter

Using the WWC as a Teaching Tool

EDITOR'S NOTE:The What Works Clearinghouse (WWC), a program of the Institute of Education Sciences, is a trusted source of scientific evidence on education programs, products, practices, and policies. The WWC also has many tools and resources for education researchers and students.  In this guest blog post, Jessaca Spybrook (pictured, below right), Associate Professor of Evaluation, Measurement and Research at Western Michigan University, discusses how she uses WWC procedures and standards as a teaching tool.


By Jessaca Spybrook, Western Michigan University

TraiJessaca Spybrookning the next generation of researchers so they are prepared to enter the world of education research is a critical part of my role as a faculty member in the Evaluation, Measurement, and Research program. I want to ensure that my students have important technical skills in a host of subject areas including, but not limited to, research design, statistics, and measurement. At the same time, I want to be sure they know how to apply the skills to design and analyze real-world studies. I often struggle to find resources for my classes that help me meet both goals.

One resource that has emerged as an important tool in meeting both goals is the What Works Clearinghouse website. I frequently integrate materials from the WWC into the graduate research design and statistics courses I teach.

For example, in a recent class I taught, Design of Experiments and Quasi-Experiments, I used the WWC Procedures and Standards Handbook Version 3.0 throughout (an image from the publication is pictured below). The Handbook met four important criteria as I was selecting resources for my class:

  1. Inclusion of important technical detail on design and analysis;
  2. Up-to-date and current thinking and “best practice” in design and analysis;
  3. Clear writing that is accessible for graduate students; and
  4. It was free (always a bonus when searching for class materials).Image from the What Works Clearinghouse Standards & Practices Guide 3.0

By no means did the Handbook replace classic and well-regarded textbooks in the class. Rather, it helped connect classic texts on design to both recent advances related to design, as well as real-life considerations and standards that designs are judged against.

At the end of my class, students may have been tired of hearing the question, “what is the highest potential rating for this study?” But I feel confident that using the WWC Handbook helped me not only prepare graduates with the technical know-how they need to design a rigorous experiment or quasi-experiment, but also raised awareness of current best practice and how to design a study that meets important standards set for the field.

 

Recommendations for Teaching Secondary Students to Write Effectively

EDITOR'S NOTE: Dr. Steve Graham was the head of a panel of experts that assisted the What Works Clearinghouse in developing recommendations for its practice guide on effective writing for secondary students. We invited Dr. Graham to author this blog about the guide and a January 18 webinar on its recommendations. 


By Steve Graham, Warner Professor in the Division of Leadership and Innovation, Arizona State University

Effective writing is a vital component of students’ literacy achievement and a life-long skill that plays a key role in postsecondary success. For more than 30 years, I’ve focused my research on how teachers can help students become strong writers, how writing develops, and how writing can be used to support reading and learning. Much progress has been made in the field of writing instruction, and summarizing and sharing these findings will help teachers implement evidence-based practices. Using effective instructional practices will help ensure our students become adept at using writing to support and extend learning, argue effectively and fairly, connect and communicate with others, tell captivating stories, and explore who they are as well as reflect on their experiences. 

Recently, the What Works Clearinghouse (WWC) released a new practice guide to address the challenges of teaching writing to secondary students. Teaching Secondary Students to Write Effectively offers three evidence-based recommendations for helping students in grades 6–12 develop effective writing skills. The first recommendation focuses on teaching students to use writing strategies to plan, think critically, and effectively convey their ideas. The second recommendation suggests integrating reading and writing to emphasize key features of text. Finally, the third recommendation describes how to use a formative assessment cycle to inform writing instruction.

The guide includes practical instructional tips and strategies for each recommendation that teachers can use to help students improve their writing. You’ll find over 30 examples to use in the classroom, including sample writing strategies and prompts and activities that incorporate writing and reading.

I’d like to invite teachers, administrators, and others to join me for a webinar on the recommendations in this practice guide, Wednesday, January 18, at 3 p.m. (ET). During the webinar, we will discuss the guide’s three recommendations and give teachers in all disciplines usable guidance on how to implement them in the classroom. We will also discuss potential challenges educators may face when implementing the recommended practices and provide advice on how to overcome those challenges.

Developing the Practice Guide

The WWC develops practice guides with the support of an expert panel. The panelists combine their expertise with the findings of rigorous research to produce specific recommendations. I was honored to chair this panel, which also included Jill Fitzgerald, from the University of North Carolina at Chapel Hill and MetaMetrics; Linda D. Friedrich, from the National Writing Project; Katie Greene, from Forsyth County (Ga.) Schools; James S. Kim, from Harvard University; and Carol Booth Olson, from the University of California, Irvine. 

For this practice guide, WWC staff conducted a systematic review of the research—a thorough literature search identified more than 3,700 relevant studies. After screening each study, 55 studies were found to use eligible research designs and examine the effec­tiveness of the practices found in this guide’s recommendations. The recommendations are based on the 15 studies that meet the WWC’s rigorous standards. For each of the recommendations, the WWC and the panel rate the strength of the evidence that supports it.  Appendix D in the guide presents a thorough summary of the evidence supporting each recommendation.