IES Blog

Institute of Education Sciences

Improving the WWC Standards and Procedures

By Chris Weiss and Jon Jacobson

For the What Works Clearinghouse (WWC), standards and procedures are at the foundation of the WWC’s work to provide scientific evidence for what works in education. They guide how studies are selected for review, what elements of an effectiveness study are examined, and how systematic reviews are conducted. The WWC’s standards and procedures are designed to be rigorous and reflective of best practices in research and statistics, while also being aspirational to help point the field of education effectiveness research toward an ever-higher quality of study design and analysis.

To keep pace with new advances in methodological research and provide necessary clarifications for both education researchers and decision makers, the WWC regularly updates its procedures and standards and shares them with the field. We recently released Version 4.0 of the Procedures and Standards Handbooks, which describes the five steps of the WWC’s systematic review process.

For this newest version, we have divided information into two separate documents (see graphic below).  The Procedures Handbook describes how the WWC decides which studies to review and how it reports on study findings. The Standards Handbook describes how the WWC rates the evidence from studies.

The new Standards Handbook includes several improvements, including updated and overhauled standards for cluster-level assignment of students; a new approach for reviewing studies that have some missing baseline or outcome data; and revised standards for regression discontinuity designs. The new Procedures Handbook includes a revised discussion of how the WWC defines a study.  All of the changes are summarized on the WWC website (PDF).

Making the Revisions

These updates were developed in a careful, collaborative manner that included experts in the field, external peer review, and input from the public.

Staff from the Institute of Education Sciences oversaw the process with the WWC’s Statistical, Technical, and Analysis Team (STAT), a panel of highly experienced researchers who revise and develop the WWC standards. In addition, the WWC sought and received input from experts on specific research topics, including regression discontinuity designs, cluster-level assignment, missing data, and complier average causal effects. Based on this information, drafts of the standards and procedures handbooks were developed.

External peer reviewers then provided input that led to additional revisions and, in the summer, the WWC posted drafts and gathered feedback from the public. The WWC’s response to some of the comments is available on its website (PDF).   

Version 4.0 of the Handbooks was released on October 26. This update focused on a few key areas of the standards, and updated and clarified some procedures. However, the WWC strives for continuous improvement and as the field of education research continues to evolve and improve, we expect that there will be new techniques and new tools incorporated into future versions the Handbooks.

Your thoughts, ideas, and suggestions are welcome and can be submitted through the WWC help desk.

Why Can’t You Just Use Google Instead of ERIC?

By Erin Pollard, ERIC Program Officer

The Education Resources Information Center (ERIC) provides the public with free, online access to a scholarly database of education research. We are frequently asked why the government sponsors such a tool when people can use Google or a subscription-based scholarly database.  

Commercial search engines and scholarly databases are important, but would not function as efficiently without ERIC’s metadata to power their search engines. Because of the costs associated with indexing, commercial and scholarly search engines would likely prioritize the work from major publishers, and may not index the work from small publishers on a regular basis.

But ERIC has built national and global relationships with key publishers, research centers, government entities, universities, education associations, and other organizations to disseminate their materials. We are currently under agreement with 1,020 different publishers, many of whom are small and only publish a single journal or report series.

For more than 50 years, ERIC has been acquiring grey literature (e.g., reports from the Institute of Education Sciences (IES) and other government reports, white papers, and conference papers) and making it centrally available and free-of-charge to the public. Therefore, an ERIC user is just as likely to find a relevant conference paper from a smaller publisher as they are to find a journal article from a major publisher. (See infographic (PDF) above to learn more about who uses ERIC)

ERIC also ensures that all records indexed meet a set of quality guidelines before indexing, and provides tools, such as a peer-review flag, that can help users evaluate the quality of the material. Underlying all of ERIC’s records are a set of metadata that helps guide users to the resources they are seeking. The metadata also includes descriptors from ERIC’s Thesaurus, a widely recognized, controlled vocabulary of subject-specific tags in the education field. Descriptors are added to each record and used by search engines to pinpoint results.

Lastly, and most importantly, ERIC provides access to more than 380,000 full-text resources, including journal articles and grey literature and makes these articles available for perpetuity. ERIC has been around for more than 50 years and has collected materials in hard copy, microfiche, and PDF. These materials are publicly available even after organizations or journals cease operations or redesign their website in a way that makes materials no longer available. In any given month, over 25% of ERIC’s new records are peer reviewed and provide free full text. Additionally, about 4% of journals provide peer-reviewed full text after an embargo. This includes work from IES grantees that normally appears in journals behind a paywall, but ERIC can make available through the IES Public Access Policy.

ERIC’s comprehensive collection, metadata, and access to full text articles make it an important resource for researchers, students, educators, policy makers and the general public. 

Want to learn more about ERIC? Watch this short video introduction or check out our multimedia page for access to other videos, infographics, and webinars.  

Using the WWC as a Teaching Tool

EDITOR'S NOTE:The What Works Clearinghouse (WWC), a program of the Institute of Education Sciences, is a trusted source of scientific evidence on education programs, products, practices, and policies. The WWC also has many tools and resources for education researchers and students.  In this guest blog post, Jessaca Spybrook (pictured, below right), Associate Professor of Evaluation, Measurement and Research at Western Michigan University, discusses how she uses WWC procedures and standards as a teaching tool.


By Jessaca Spybrook, Western Michigan University

TraiJessaca Spybrookning the next generation of researchers so they are prepared to enter the world of education research is a critical part of my role as a faculty member in the Evaluation, Measurement, and Research program. I want to ensure that my students have important technical skills in a host of subject areas including, but not limited to, research design, statistics, and measurement. At the same time, I want to be sure they know how to apply the skills to design and analyze real-world studies. I often struggle to find resources for my classes that help me meet both goals.

One resource that has emerged as an important tool in meeting both goals is the What Works Clearinghouse website. I frequently integrate materials from the WWC into the graduate research design and statistics courses I teach.

For example, in a recent class I taught, Design of Experiments and Quasi-Experiments, I used the WWC Procedures and Standards Handbook Version 3.0 throughout (an image from the publication is pictured below). The Handbook met four important criteria as I was selecting resources for my class:

  1. Inclusion of important technical detail on design and analysis;
  2. Up-to-date and current thinking and “best practice” in design and analysis;
  3. Clear writing that is accessible for graduate students; and
  4. It was free (always a bonus when searching for class materials).Image from the What Works Clearinghouse Standards & Practices Guide 3.0

By no means did the Handbook replace classic and well-regarded textbooks in the class. Rather, it helped connect classic texts on design to both recent advances related to design, as well as real-life considerations and standards that designs are judged against.

At the end of my class, students may have been tired of hearing the question, “what is the highest potential rating for this study?” But I feel confident that using the WWC Handbook helped me not only prepare graduates with the technical know-how they need to design a rigorous experiment or quasi-experiment, but also raised awareness of current best practice and how to design a study that meets important standards set for the field.

 

How to Use the Improved ERIC Identifiers

ERIC has made recent improvements to help searchers find the education research they are looking for. One major enhancement relates to the ERIC identifiers, which have been improved to increase their usefulness as search tools. It is now easier than ever to refine searches to obtain specific resources in ERIC.

The identifier filters can be found on the search results page in three separate categories: (1) laws, policies, and programs, (2) assessments and surveys, and (3) location. After running a search on an education topic, users can scroll to the category on the left of the results page, select the desired identifier limiter within a category, and limit the results to only those materials tagged with that identifier.

We recently released a video that describes the enhanced identifiers, and walks through how to best use them to find materials in the ERIC collection. (We've embedded the video below.) 

Using the improved identifiers, searchers are now able to find materials related to specific locations, laws, or assessments no matter how the author referred to them in the article.

In other words, identifiers can now be used as an effective controlled vocabulary for ERIC, but this has not always been the case. While they have been part of ERIC since 1966, identifiers were not rigorously standardized, and they were often created "on the fly" by indexers. Also, the previous identifiers field had a character limit, meaning that some terms needed to be truncated to fit into the space allowed by the available technology. Therefore, over time, the identifiers proliferated with different spellings, abbreviations, and other variations, making them less useful as search aids.

To solve these issues, we launched a project in 2016 to review the lists of identifiers, and devise an approach for making them more user-friendly. Our solution was to streamline and standardize them, which eliminated redundancy and reduced their number from more than 7,800 to a more manageable 1,200. We also added the updated identifiers to the website’s search limiters to make them easier to use.

In addition to our new video, which demonstrates the best ways to use identifiers in your search, we also have a new infographic (pictured above) that depicts what identifiers are. You can use these companion pieces to learn more about identifiers, and begin putting them to work in your research. 

New Fields in ERIC

By Erin Pollard, ERIC Project Officer, NCEE

ERIC has recently added several new fields to our database that will make it easier for researchers to find relevant studies. These are changes we've been working on for a while and we are excited that they are finally live. 

Below is an overview of the changes, but you can learn more about our new fields during a webinar on January 18, 2017 at 2 p.m. (ET)

New Links to IES

The first fields that we introduced were designed to connect ERIC users with additional relevant information available on the Institute of Education Sciences (IES) website. Because ERIC sits on a separate website, we found many ERIC users never visited the IES website and did not take advantage of the high-quality content that is available. So, we added several fields to help connect you to places of interest on the IES website. First, we added links from the ERIC website to each publication page on the IES website. These links will help you find related videos and companion products for IES reports, if they are available. Second, for any work funded by an IES grant, we added a link to the grant abstract. This provides information about the overall body of work funded by the grant and any accompanying publications. Lastly, the What Works Clearinghouse has recently redesigned its website, and one aspect of the redesign is that there are now study pages that provide detailed information on specific studies that the Clearinghouse has reviewed. ERIC is linking to these pages so that our users can benefit from the in-depth, user-friendly information provided by the Clearinghouse.

New “Identifiers”

The second set of new fields was designed to clean up the previous “identifiers” field and make them more useful for searchers.

The identifiers field was a hodgepodge of proper nouns that mainly contained information on laws, tests and measures, and geographic locations. We separated this into three new fields—laws, measures, and location. We also standardized the language that we used to make these a controlled vocabulary that users could filter on. This change will enable you to find all work done in Alabama or any work that used the National Assessment of Educational Progress (for example).

New Author Identification Numbers

The third new field adds links to author’s biosketch pages. It can get confusing when several authors have the same name, and when the same author can publishes under different names. For example, the same individual could publish under “John Young,” “John P. Young,” “J.P. Young,” and “Jack Young.” ERIC does not have the ability to determine if these are all the same people, but we were able to add hyperlinks to those authors that have an Orchid ID or a SciENcv  page set up. If these numbers are available when we are indexing the record, we will be able to link to authors’ pages so that users can see the other work they have published. IES is encouraging grantees to use SciencCV, so we expect to see a large increase in the use of these fields.

If you have any questions about the new fields, please contact the ERIC help desk or join us for our webinar.