NCEE Blog

In December 2016, the What Works Clearinghouse made a version of its online training publicly available through the WWC Website. This enabled everyone to be able to access the Version 3.0 Group Design Standards reviewer training to learn about the standards and methods that the WWC uses. While this was a great step to increase access to WWC resources, users still had to go through the 1 ½ day, in-person training to become a WWC certified reviewer.

To continue our efforts to promote access and transparency and make our resources available to everyone, the WWC has now moved all of its group design training to be online. Now everyone will have access to the same training and certification tests. This certification is available free of charge and is open to all users. It is our hope that this effort will increase the number of certified reviewers and help increase general awareness about the WWC.

Why did the WWC make these resources publicly available? As part of IES’s effort to increase access to high quality education research, we wanted to make it easier for researchers to use our standards. This meant opening up training opportunities and offering training online was a way to achieve this goal while using limited taxpayer resources most efficiently.

The online training consists of 9 modules. These videos feature an experienced WWC instructor and use the same materials that we used in our in-person courses, but adapted to Version 4.0 of the Group Design Standards. After completing the modules, users will have the opportunity to download a certificate of completion, take the online certification test, or go through the full certification exam.

Becoming a fully certified reviewer will require users to take a multiple choice online certification test and then use the new Online SRG application to conduct a full review using the same tools that the WWC team uses. The WWC team will then grade your exam to make sure you fully understand how to apply the Standards before certifying you to review for the Clearinghouse.

Not interested in becoming a certified reviewer? Online training still has several benefits. Educators can embed our videos in their course websites and use our training materials in their curricula. Researchers can use our Online SRG tool with their publications to determine a preliminary rating and understand what factors could cause their study to get the highest rating. They could also use the tool to use when conducting a systematic evidence review.

Have ideas for new resources we could make available? Email your ideas and suggestions to Contact.WWC@ed.gov!

by Erin Pollard, WWC Project Officer

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

By Erin Pollard, ERIC Program Officer

The Education Resources Information Center (ERIC) provides the public with free, online access to a scholarly database of education research. We are frequently asked why the government sponsors such a tool when people can use Google or a subscription-based scholarly database.  

Commercial search engines and scholarly databases are important, but would not function as efficiently without ERIC’s metadata to power their search engines. Because of the costs associated with indexing, commercial and scholarly search engines would likely prioritize the work from major publishers, and may not index the work from small publishers on a regular basis.

But ERIC has built national and global relationships with key publishers, research centers, government entities, universities, education associations, and other organizations to disseminate their materials. We are currently under agreement with 1,020 different publishers, many of whom are small and only publish a single journal or report series.

For more than 50 years, ERIC has been acquiring grey literature (e.g., reports from the Institute of Education Sciences (IES) and other government reports, white papers, and conference papers) and making it centrally available and free-of-charge to the public. Therefore, an ERIC user is just as likely to find a relevant conference paper from a smaller publisher as they are to find a journal article from a major publisher. (See infographic (PDF) above to learn more about who uses ERIC)

ERIC also ensures that all records indexed meet a set of quality guidelines before indexing, and provides tools, such as a peer-review flag, that can help users evaluate the quality of the material. Underlying all of ERIC’s records are a set of metadata that helps guide users to the resources they are seeking. The metadata also includes descriptors from ERIC’s Thesaurus, a widely recognized, controlled vocabulary of subject-specific tags in the education field. Descriptors are added to each record and used by search engines to pinpoint results.

Lastly, and most importantly, ERIC provides access to more than 380,000 full-text resources, including journal articles and grey literature and makes these articles available for perpetuity. ERIC has been around for more than 50 years and has collected materials in hard copy, microfiche, and PDF. These materials are publicly available even after organizations or journals cease operations or redesign their website in a way that makes materials no longer available. In any given month, over 25% of ERIC’s new records are peer reviewed and provide free full text. Additionally, about 4% of journals provide peer-reviewed full text after an embargo. This includes work from IES grantees that normally appears in journals behind a paywall, but ERIC can make available through the IES Public Access Policy.

ERIC’s comprehensive collection, metadata, and access to full text articles make it an important resource for researchers, students, educators, policy makers and the general public. 

Want to learn more about ERIC? Watch this short video introduction or check out our multimedia page for access to other videos, infographics, and webinars.  

By Dana Tofig, Communications Director, IES

Almost a decade ago, the What Works Clearinghouse (WWC) released Dropout Prevention, one of its first practice guides, which offered six recommendations for keeping students in school, based on the findings from high-quality research and best practices in the field. Since its release in August 2008, the practice guide has been downloaded thousands of times by practitioners across the country. In fact, the guide was downloaded more than 1,500 times in 2016, alone.

However, over the past decade, the research and knowledge base has grown in the area of dropout prevention, which is why the WWC decided to update this guide to reflect the latest evidence and information about keeping students in school and on track toward graduation.

This updated guide, Preventing Dropout in Secondary Schools, builds on the 2008 guide in two significant ways.

First, it reflects improvements in practices related to monitoring at-risk students, including advances in using early warning indicators to identify students at risk for dropping out. Secondly, it covers an additional nine years of research that were not a part of the previous guide. In fact, 15 of the 25 studies used to support the recommendations in this updated guide were published after the first guide was published. In addition, studies from the previous guide were reviewed again against current, more rigorous WWC evidence standards.

Preventing Dropout in Secondary Schools offers four evidence-based recommendations that can be used by schools and districts:

• Monitor the progress of all students, and proactively intervene when students show early signs of attendance, behavior, or academic problems;
• Provide intensive, individualized support to students who have fallen off track and face significant challenges to success;
• Engage students by offering curricula and programs that connect schoolwork with college and career success and that improve students’ capacity to manage challenges in and out of school; and
• For schools with many at-risk students, create small, personalized communities to facilitate monitoring and support.

Each of these recommendations includes specific strategies for implementation and examples of how this work is being done around the country (see one such example in the image to the right).

Like all of our practice guides, the recommendations were developed with a panel of educators, academics, and experts who brought a wealth of knowledge and experience to the process. Two of the panelists on this updated guide were also involved in the development of the first dropout prevention guide—Russell Rumberger, from University of California, Santa Barbara, and Mark Dynarksi, of Pemberton Research LLC. The other panelists for the new guide were Howard (Sandy) Addis, of the National Dropout Prevention Center and Network; Elaine Allensworth, from the University of Chicago; Robert Balfanz, from The Johns Hopkins University; and Debra Duardo, superintendent of the Los Angeles County Office of Education.

Please download this free guide today and let us know what you think. We would especially love to hear from people who are using the recommended strategies in schools. You can reach us through the WWC Help Desk or email us at Contact.IES@ed.gov. 

By Christopher Weiss, Program Manager, What Works Clearinghouse

The What Works Clearinghouse (WWC) is always looking for ways to improve. We want it to be as easy as possible for our users to connect with the evidence they need, so they can make informed educational decisions.

Last year, we undertook a comprehensive, multi-faceted self-study. Through surveys, interviews, and focus groups, we asked a variety of different WWC users to tell us what we were doing well and, more importantly, what we could do better. (Click here if you’re interested in all the results.)

Some of the specific suggestions we received focused on the WWC Educator’s Practice Guides, which combine the best available research evidence and practitioner expertise on a topic to provide educators with strategies to use in their school or classroom. Based upon a review of the research literature and the guidance of a panel of nationally recognized experts, practice guides synthesize evidence and the wisdom of practitioners.

One particular suggestion that came from the self-study was to create a separate, stand-alone document with concise and specific information that a teacher or school would need to carry out some of a practice guide’s recommendations. It was a great suggestion – and we put it into action.

On July 25, we released our first Instructional Tips publication (PDF), which was created to help educators carry out the recommendations in the Improving Mathematical Problem Solving in Grades 4 through 8 practice guide. We provide tips for three of the Practice Guide’s five recommendations:

• Assisting Students in Monitoring and Reflecting on the Problem-Solving Process;
• Teaching Students to Use Visual Representations to Solve Problems; and
• Helping Students Make Sense of Algebraic Notation.

As an example, for the recommendation on visual representations, we offer two instructional tips. First, we suggest that teachers demonstrate how to select the appropriate visual representation for the problem they are solving and we provide specific steps and examples for implementing this tip. Second, we suggest teachers use think-alouds and discussions to teach students how to represent problems visually and, again, provide specific steps and work examples. Here's one of the examples from the publication:

An accompanying document (PDF) to the Instructional Tips describes the evidence base that supports these recommended practices.

We are planning additional Instructional Tips publications down the road, but we want to hear from you first. If you have questions or ideas for how we can improve this resource, we’d love to hear them. Please send them through an email to the WWC Help Desk.

The Instructional Tips are just one of several ways we are working to improve the WWC. Over the past two years, we have redesigned our website and created a new Find What Works tool to make it easier for users to find the evidence they need. We have also increased our use of Facebook and Twitter to help us better connect with new audiences; published new briefs and held several webinars to explain WWC processes and resources; and have launched a new Reviews of Individual Studies database to give the field quicker access to the research we have reviewed. And all of this has been done while we continue to identify interventions, practices and programs that show evidence of improving student outcomes across a wide array of educational topics.

Stay up to date on new WWC products, events, and resources by signing up for the IES News Flash (under NCEE) and following us on Facebook and Twitter. 

EDITOR'S NOTE:The What Works Clearinghouse (WWC), a program of the Institute of Education Sciences, is a trusted source of scientific evidence on education programs, products, practices, and policies. The WWC also has many tools and resources for education researchers and students.  In this guest blog post, Jessaca Spybrook (pictured, below right), Associate Professor of Evaluation, Measurement and Research at Western Michigan University, discusses how she uses WWC procedures and standards as a teaching tool.

By Jessaca Spybrook, Western Michigan University

Training the next generation of researchers so they are prepared to enter the world of education research is a critical part of my role as a faculty member in the Evaluation, Measurement, and Research program. I want to ensure that my students have important technical skills in a host of subject areas including, but not limited to, research design, statistics, and measurement. At the same time, I want to be sure they know how to apply the skills to design and analyze real-world studies. I often struggle to find resources for my classes that help me meet both goals.

One resource that has emerged as an important tool in meeting both goals is the What Works Clearinghouse website. I frequently integrate materials from the WWC into the graduate research design and statistics courses I teach.

For example, in a recent class I taught, Design of Experiments and Quasi-Experiments, I used the WWC Procedures and Standards Handbook Version 3.0 throughout (an image from the publication is pictured below). The Handbook met four important criteria as I was selecting resources for my class:

1. Inclusion of important technical detail on design and analysis;
2. Up-to-date and current thinking and “best practice” in design and analysis;
3. Clear writing that is accessible for graduate students; and
4. It was free (always a bonus when searching for class materials).

By no means did the Handbook replace classic and well-regarded textbooks in the class. Rather, it helped connect classic texts on design to both recent advances related to design, as well as real-life considerations and standards that designs are judged against.

At the end of my class, students may have been tired of hearing the question, “what is the highest potential rating for this study?” But I feel confident that using the WWC Handbook helped me not only prepare graduates with the technical know-how they need to design a rigorous experiment or quasi-experiment, but also raised awareness of current best practice and how to design a study that meets important standards set for the field.

ERIC has made recent improvements to help searchers find the education research they are looking for. One major enhancement relates to the ERIC identifiers, which have been improved to increase their usefulness as search tools. It is now easier than ever to refine searches to obtain specific resources in ERIC.

The identifier filters can be found on the search results page in three separate categories: (1) laws, policies, and programs, (2) assessments and surveys, and (3) location. After running a search on an education topic, users can scroll to the category on the left of the results page, select the desired identifier limiter within a category, and limit the results to only those materials tagged with that identifier.

We recently released a video that describes the enhanced identifiers, and walks through how to best use them to find materials in the ERIC collection. (We've embedded the video below.) 

Using the improved identifiers, searchers are now able to find materials related to specific locations, laws, or assessments no matter how the author referred to them in the article.

In other words, identifiers can now be used as an effective controlled vocabulary for ERIC, but this has not always been the case. While they have been part of ERIC since 1966, identifiers were not rigorously standardized, and they were often created "on the fly" by indexers. Also, the previous identifiers field had a character limit, meaning that some terms needed to be truncated to fit into the space allowed by the available technology. Therefore, over time, the identifiers proliferated with different spellings, abbreviations, and other variations, making them less useful as search aids.

To solve these issues, we launched a project in 2016 to review the lists of identifiers, and devise an approach for making them more user-friendly. Our solution was to streamline and standardize them, which eliminated redundancy and reduced their number from more than 7,800 to a more manageable 1,200. We also added the updated identifiers to the website’s search limiters to make them easier to use.

In addition to our new video, which demonstrates the best ways to use identifiers in your search, we also have a new infographic (pictured above) that depicts what identifiers are. You can use these companion pieces to learn more about identifiers, and begin putting them to work in your research. 

EDITOR'S NOTE: Dr. Steve Graham was the head of a panel of experts that assisted the What Works Clearinghouse in developing recommendations for its practice guide on effective writing for secondary students. We invited Dr. Graham to author this blog about the guide and a January 18 webinar on its recommendations. 

By Steve Graham, Warner Professor in the Division of Leadership and Innovation, Arizona State University

Effective writing is a vital component of students’ literacy achievement and a life-long skill that plays a key role in postsecondary success. For more than 30 years, I’ve focused my research on how teachers can help students become strong writers, how writing develops, and how writing can be used to support reading and learning. Much progress has been made in the field of writing instruction, and summarizing and sharing these findings will help teachers implement evidence-based practices. Using effective instructional practices will help ensure our students become adept at using writing to support and extend learning, argue effectively and fairly, connect and communicate with others, tell captivating stories, and explore who they are as well as reflect on their experiences. 

Recently, the What Works Clearinghouse (WWC) released a new practice guide to address the challenges of teaching writing to secondary students. Teaching Secondary Students to Write Effectively offers three evidence-based recommendations for helping students in grades 6–12 develop effective writing skills. The first recommendation focuses on teaching students to use writing strategies to plan, think critically, and effectively convey their ideas. The second recommendation suggests integrating reading and writing to emphasize key features of text. Finally, the third recommendation describes how to use a formative assessment cycle to inform writing instruction.

The guide includes practical instructional tips and strategies for each recommendation that teachers can use to help students improve their writing. You’ll find over 30 examples to use in the classroom, including sample writing strategies and prompts and activities that incorporate writing and reading.

I’d like to invite teachers, administrators, and others to join me for a webinar on the recommendations in this practice guide, Wednesday, January 18, at 3 p.m. (ET). During the webinar, we will discuss the guide’s three recommendations and give teachers in all disciplines usable guidance on how to implement them in the classroom. We will also discuss potential challenges educators may face when implementing the recommended practices and provide advice on how to overcome those challenges.

Developing the Practice Guide

The WWC develops practice guides with the support of an expert panel. The panelists combine their expertise with the findings of rigorous research to produce specific recommendations. I was honored to chair this panel, which also included Jill Fitzgerald, from the University of North Carolina at Chapel Hill and MetaMetrics; Linda D. Friedrich, from the National Writing Project; Katie Greene, from Forsyth County (Ga.) Schools; James S. Kim, from Harvard University; and Carol Booth Olson, from the University of California, Irvine. 

For this practice guide, WWC staff conducted a systematic review of the research—a thorough literature search identified more than 3,700 relevant studies. After screening each study, 55 studies were found to use eligible research designs and examine the effec­tiveness of the practices found in this guide’s recommendations. The recommendations are based on the 15 studies that meet the WWC’s rigorous standards. For each of the recommendations, the WWC and the panel rate the strength of the evidence that supports it.  Appendix D in the guide presents a thorough summary of the evidence supporting each recommendation. 

By Christopher Weiss, Senior Education Research Scientist, WWC

Many individuals and organizations have special ways of doing things, specific procedures that make them unique —Coca-Cola has its formula; sports teams have their playbooks; and grandparents have their secret recipes for biscuits, barbecue, and other family favorites.

It’s the same for the What Works Clearinghouse (WWC). Our “special sauce” is in how we review effectiveness research to help determine what is working in education. But unlike Coke, coaches, and grandma, the WWC doesn’t keep it a secret.

On December 15, the Institute of Education Sciences (IES) launched a set of video training modules – the WWC Group Design Standards Online Training – to share our procedures. These modules are designed to help you learn more about the elements that go into a WWC rating and the features of a research study that WWC examines during evaluation.  The online training will help education decision-makers and researchers better understand key elements of the WWC review process. These modules describe and explain key topics and concepts of the WWC’s Group Design standards and how the WWC uses these standards to identify and evaluate high quality, rigorous research.

The series is designed to address the needs of both consumers and future producers of the WWC’s reviews of educational effectiveness research. Whether you’re a researcher who’s hoping your study will meet the WWC’s standards or someone trying to make an evidence-based decision related to education, this training series will help! And no background in research is needed –we’ve also developed an extensive set of materials to support you as you learn.

Each of the five modules follows a similar structure, including an overview of module objectives, detailed information about the topic, examples, and knowledge checks to reinforce what you’ve learned. (We've embedded the first video in the series at the end of this post, but if you are going to take the training, start it through the WWC website.)  

Each module focuses on a specific aspect of the standards.

• Group designs – or overall research designs – and the types of research that can be reviewed using the WWC Group Design Standards;
• Attrition, or loss of participants in a research study, and why this is important;
• Baseline Equivalence, which assesses how similar two groups are at the beginning of a research study;
• Confounding Factors, which are study components that make it difficult or impossible to distinguish the effect of an educational intervention from the effect of that component; and
• Outcome Measures, or what is measured to assess the effectiveness of an intervention.

If you view all five of the training modules, you will earn a certificate of completion. Details about how to view the session and earn this certificate are available on the What Works Clearinghouse website.

The online training takes about seven hours to complete, but the modules are designed so that you can complete them at your own pace. We’ve included a feature that allows you to take a break from the training at any point – then pick it up again where you left off when you’re ready to continue.

These modules cover the same material that WWC reviewers learn through their in-person certification training – and completion of the online training course is one step toward becoming a certified reviewer in WWC Group Design Standards. Certification also requires completing WWC Procedures training and successful completion of a certification exam. We expect to be able to offer online versions of the WWC Procedures training and the certification exam later in 2017.

We hope this online training brings more transparency and understanding to the WWC review process. Then we can work on that secret biscuit recipe. 

By Erin Pollard, ERIC Project Officer, NCEE

ERIC has recently added several new fields to our database that will make it easier for researchers to find relevant studies. These are changes we've been working on for a while and we are excited that they are finally live.