NCEE Blog

National Center for Education Evaluation and Regional Assistance

Using the WWC as a Teaching Tool

EDITOR'S NOTE:The What Works Clearinghouse (WWC), a program of the Institute of Education Sciences, is a trusted source of scientific evidence on education programs, products, practices, and policies. The WWC also has many tools and resources for education researchers and students.  In this guest blog post, Jessaca Spybrook (pictured, below right), Associate Professor of Evaluation, Measurement and Research at Western Michigan University, discusses how she uses WWC procedures and standards as a teaching tool.


By Jessaca Spybrook, Western Michigan University

TraiJessaca Spybrookning the next generation of researchers so they are prepared to enter the world of education research is a critical part of my role as a faculty member in the Evaluation, Measurement, and Research program. I want to ensure that my students have important technical skills in a host of subject areas including, but not limited to, research design, statistics, and measurement. At the same time, I want to be sure they know how to apply the skills to design and analyze real-world studies. I often struggle to find resources for my classes that help me meet both goals.

One resource that has emerged as an important tool in meeting both goals is the What Works Clearinghouse website. I frequently integrate materials from the WWC into the graduate research design and statistics courses I teach.

For example, in a recent class I taught, Design of Experiments and Quasi-Experiments, I used the WWC Procedures and Standards Handbook Version 3.0 throughout (an image from the publication is pictured below). The Handbook met four important criteria as I was selecting resources for my class:

  1. Inclusion of important technical detail on design and analysis;
  2. Up-to-date and current thinking and “best practice” in design and analysis;
  3. Clear writing that is accessible for graduate students; and
  4. It was free (always a bonus when searching for class materials).Image from the What Works Clearinghouse Standards & Practices Guide 3.0

By no means did the Handbook replace classic and well-regarded textbooks in the class. Rather, it helped connect classic texts on design to both recent advances related to design, as well as real-life considerations and standards that designs are judged against.

At the end of my class, students may have been tired of hearing the question, “what is the highest potential rating for this study?” But I feel confident that using the WWC Handbook helped me not only prepare graduates with the technical know-how they need to design a rigorous experiment or quasi-experiment, but also raised awareness of current best practice and how to design a study that meets important standards set for the field.

 

How to Use the Improved ERIC Identifiers

ERIC has made recent improvements to help searchers find the education research they are looking for. One major enhancement relates to the ERIC identifiers, which have been improved to increase their usefulness as search tools. It is now easier than ever to refine searches to obtain specific resources in ERIC.

The identifier filters can be found on the search results page in three separate categories: (1) laws, policies, and programs, (2) assessments and surveys, and (3) location. After running a search on an education topic, users can scroll to the category on the left of the results page, select the desired identifier limiter within a category, and limit the results to only those materials tagged with that identifier.

We recently released a video that describes the enhanced identifiers, and walks through how to best use them to find materials in the ERIC collection. (We've embedded the video below.) 

Using the improved identifiers, searchers are now able to find materials related to specific locations, laws, or assessments no matter how the author referred to them in the article.

In other words, identifiers can now be used as an effective controlled vocabulary for ERIC, but this has not always been the case. While they have been part of ERIC since 1966, identifiers were not rigorously standardized, and they were often created "on the fly" by indexers. Also, the previous identifiers field had a character limit, meaning that some terms needed to be truncated to fit into the space allowed by the available technology. Therefore, over time, the identifiers proliferated with different spellings, abbreviations, and other variations, making them less useful as search aids.

To solve these issues, we launched a project in 2016 to review the lists of identifiers, and devise an approach for making them more user-friendly. Our solution was to streamline and standardize them, which eliminated redundancy and reduced their number from more than 7,800 to a more manageable 1,200. We also added the updated identifiers to the website’s search limiters to make them easier to use.

In addition to our new video, which demonstrates the best ways to use identifiers in your search, we also have a new infographic (pictured above) that depicts what identifiers are. You can use these companion pieces to learn more about identifiers, and begin putting them to work in your research. 

Recommendations for Teaching Secondary Students to Write Effectively

EDITOR'S NOTE: Dr. Steve Graham was the head of a panel of experts that assisted the What Works Clearinghouse in developing recommendations for its practice guide on effective writing for secondary students. We invited Dr. Graham to author this blog about the guide and a January 18 webinar on its recommendations. 


By Steve Graham, Warner Professor in the Division of Leadership and Innovation, Arizona State University

Effective writing is a vital component of students’ literacy achievement and a life-long skill that plays a key role in postsecondary success. For more than 30 years, I’ve focused my research on how teachers can help students become strong writers, how writing develops, and how writing can be used to support reading and learning. Much progress has been made in the field of writing instruction, and summarizing and sharing these findings will help teachers implement evidence-based practices. Using effective instructional practices will help ensure our students become adept at using writing to support and extend learning, argue effectively and fairly, connect and communicate with others, tell captivating stories, and explore who they are as well as reflect on their experiences. 

Recently, the What Works Clearinghouse (WWC) released a new practice guide to address the challenges of teaching writing to secondary students. Teaching Secondary Students to Write Effectively offers three evidence-based recommendations for helping students in grades 6–12 develop effective writing skills. The first recommendation focuses on teaching students to use writing strategies to plan, think critically, and effectively convey their ideas. The second recommendation suggests integrating reading and writing to emphasize key features of text. Finally, the third recommendation describes how to use a formative assessment cycle to inform writing instruction.

The guide includes practical instructional tips and strategies for each recommendation that teachers can use to help students improve their writing. You’ll find over 30 examples to use in the classroom, including sample writing strategies and prompts and activities that incorporate writing and reading.

I’d like to invite teachers, administrators, and others to join me for a webinar on the recommendations in this practice guide, Wednesday, January 18, at 3 p.m. (ET). During the webinar, we will discuss the guide’s three recommendations and give teachers in all disciplines usable guidance on how to implement them in the classroom. We will also discuss potential challenges educators may face when implementing the recommended practices and provide advice on how to overcome those challenges.

Developing the Practice Guide

The WWC develops practice guides with the support of an expert panel. The panelists combine their expertise with the findings of rigorous research to produce specific recommendations. I was honored to chair this panel, which also included Jill Fitzgerald, from the University of North Carolina at Chapel Hill and MetaMetrics; Linda D. Friedrich, from the National Writing Project; Katie Greene, from Forsyth County (Ga.) Schools; James S. Kim, from Harvard University; and Carol Booth Olson, from the University of California, Irvine. 

For this practice guide, WWC staff conducted a systematic review of the research—a thorough literature search identified more than 3,700 relevant studies. After screening each study, 55 studies were found to use eligible research designs and examine the effec­tiveness of the practices found in this guide’s recommendations. The recommendations are based on the 15 studies that meet the WWC’s rigorous standards. For each of the recommendations, the WWC and the panel rate the strength of the evidence that supports it.  Appendix D in the guide presents a thorough summary of the evidence supporting each recommendation. 

Sharing our Recipe: Online Training in WWC Standards

By Christopher Weiss, Senior Education Research Scientist, WWC

Many individuals and organizations have special ways of doing things, specific procedures that make them unique —Coca-Cola has its formula; sports teams have their playbooks; and grandparents have their secret recipes for biscuits, barbecue, and other family favorites.

It’s the same for the What Works Clearinghouse (WWC). Our “special sauce” is in how we review effectiveness research to help determine what is working in education. But unlike Coke, coaches, and grandma, the WWC doesn’t keep it a secret.

On December 15, the Institute of Education Sciences (IES) launched a set of video training modules – the WWC Group Design Standards Online Training – to share our procedures. These modules are designed to help you learn more about the elements that go into a WWC rating and the features of a research study that WWC examines during evaluation.  The online training will help education decision-makers and researchers better understand key elements of the WWC review process. These modules describe and explain key topics and concepts of the WWC’s Group Design standards and how the WWC uses these standards to identify and evaluate high quality, rigorous research.

The series is designed to address the needs of both consumers and future producers of the WWC’s reviews of educational effectiveness research. Whether you’re a researcher who’s hoping your study will meet the WWC’s standards or someone trying to make an evidence-based decision related to education, this training series will help! And no background in research is needed –we’ve also developed an extensive set of materials to support you as you learn.

Each of the five modules follows a similar structure, including an overview of module objectives, detailed information about the topic, examples, and knowledge checks to reinforce what you’ve learned. (We've embedded the first video in the series at the end of this post, but if you are going to take the training, start it through the WWC website.)  

Each module focuses on a specific aspect of the standards.

  • Group designs – or overall research designs – and the types of research that can be reviewed using the WWC Group Design Standards;
  • Attrition, or loss of participants in a research study, and why this is important;
  • Baseline Equivalence, which assesses how similar two groups are at the beginning of a research study;
  • Confounding Factors, which are study components that make it difficult or impossible to distinguish the effect of an educational intervention from the effect of that component; and
  • Outcome Measures, or what is measured to assess the effectiveness of an intervention.

If you view all five of the training modules, you will earn a certificate of completion. Details about how to view the session and earn this certificate are available on the What Works Clearinghouse website.

The online training takes about seven hours to complete, but the modules are designed so that you can complete them at your own pace. We’ve included a feature that allows you to take a break from the training at any point – then pick it up again where you left off when you’re ready to continue.

These modules cover the same material that WWC reviewers learn through their in-person certification training – and completion of the online training course is one step toward becoming a certified reviewer in WWC Group Design Standards. Certification also requires completing WWC Procedures training and successful completion of a certification exam. We expect to be able to offer online versions of the WWC Procedures training and the certification exam later in 2017.

We hope this online training brings more transparency and understanding to the WWC review process. Then we can work on that secret biscuit recipe. 

 

New Fields in ERIC

By Erin Pollard, ERIC Project Officer, NCEE

ERIC has recently added several new fields to our database that will make it easier for researchers to find relevant studies. These are changes we've been working on for a while and we are excited that they are finally live. 

Below is an overview of the changes, but you can learn more about our new fields during a webinar on January 18, 2017 at 2 p.m. (ET)

New Links to IES

The first fields that we introduced were designed to connect ERIC users with additional relevant information available on the Institute of Education Sciences (IES) website. Because ERIC sits on a separate website, we found many ERIC users never visited the IES website and did not take advantage of the high-quality content that is available. So, we added several fields to help connect you to places of interest on the IES website. First, we added links from the ERIC website to each publication page on the IES website. These links will help you find related videos and companion products for IES reports, if they are available. Second, for any work funded by an IES grant, we added a link to the grant abstract. This provides information about the overall body of work funded by the grant and any accompanying publications. Lastly, the What Works Clearinghouse has recently redesigned its website, and one aspect of the redesign is that there are now study pages that provide detailed information on specific studies that the Clearinghouse has reviewed. ERIC is linking to these pages so that our users can benefit from the in-depth, user-friendly information provided by the Clearinghouse.

New “Identifiers”

The second set of new fields was designed to clean up the previous “identifiers” field and make them more useful for searchers.

The identifiers field was a hodgepodge of proper nouns that mainly contained information on laws, tests and measures, and geographic locations. We separated this into three new fields—laws, measures, and location. We also standardized the language that we used to make these a controlled vocabulary that users could filter on. This change will enable you to find all work done in Alabama or any work that used the National Assessment of Educational Progress (for example).

New Author Identification Numbers

The third new field adds links to author’s biosketch pages. It can get confusing when several authors have the same name, and when the same author can publishes under different names. For example, the same individual could publish under “John Young,” “John P. Young,” “J.P. Young,” and “Jack Young.” ERIC does not have the ability to determine if these are all the same people, but we were able to add hyperlinks to those authors that have an Orchid ID or a SciENcv  page set up. If these numbers are available when we are indexing the record, we will be able to link to authors’ pages so that users can see the other work they have published. IES is encouraging grantees to use SciencCV, so we expect to see a large increase in the use of these fields.

If you have any questions about the new fields, please contact the ERIC help desk or join us for our webinar.

Why We Still Can Learn from Evaluations that Find No Effects

By Thomas Wei, Evaluation Team Leader

As researchers, we take little pleasure when the programs and policies we study do not find positive effects on student outcomes. But as J.K. Rowling, author of the Harry Potter series, put it: there are “fringe benefits of failure.” In education research, studies that find no effects can still reveal important lessons and inspire new ideas that drive scientific progress.

On November 2, The Institute of Education Sciences (IES) released a new brief synthesizing three recent large-scale random assignment studies of teacher professional development (PD). As a nation we invest billions of dollars in PD every year, so it is important to assess the impact of those dollars on teaching and learning. These studies are part of an evaluation agenda that IES has developed to advance understanding of how to help teachers improve.

One of the studies focused on second-grade reading teachers, one on fourth-grade math teachers, and one on seventh-grade math teachers. The PD programs in each study emphasized building teachers’ knowledge of content or content-specific pedagogy. The programs combined summer institutes with teacher meetings and coaching during the school year. These programs were compared to the substantially less intensive PD that teachers typically received in study districts.

All three studies found that the PD did not have positive impacts on student achievement. Disappointing? Certainly. But have we at least learned something useful? Absolutely.

For example, the studies found that the PD did have positive impacts on teachers’ knowledge and some instructional practices. This tells us that intensive summer institutes with periodic meetings and coaching during the school year may be a promising format for this kind of professional development. (See the graphic above and chart below, which are from a snapshot of one of the studies.)

But why didn’t the improved knowledge and practice translate to improved student achievement? Educators and researchers have long argued that effective teachers need to have strong knowledge of the content they teach and know how best to convey the content to their students. The basic logic behind the content-focused PD we studied is to boost both of these skills, which were expected to translate to better student outcomes. But the findings suggest that this translation is actually very complex. For example, at what level do teachers need to know their subjects? Does a third-grade math teacher need to be a mathematician, just really good at third-grade math, or somewhere in between? What knowledge and practices are most important for conveying the content to students? How do we structure PD to ensure that teachers can master the knowledge and practice they need?

When we looked at correlational data from these three studies, we consistently found that most of the measured aspects of teachers’ knowledge and practice were not strongly related to student achievement. This reinforces the idea that we may need to find formats for PD that can boost knowledge and practice to an even larger degree, or we may need to find other aspects of knowledge and practice for PD to focus on that are more strongly related to student achievement. This is a critical lesson that we hope will inspire researchers, developers, and providers to think more carefully about the logic and design of content-focused PD.

Scientific progress is often incremental and painstaking. It requires continual testing and re-testing of interventions, which sometimes will not have impacts on the outcomes we care most about. But if we are willing to step back and try to connect the dots from various studies, we can still learn a great deal that will help drive progress forward.

 

Bringing Evidence-based Practices to the Field

By Dr. Barbara Foorman, Director Emeritus, Florida Center for Reading Research, Florida State University

The Institute of Education Sciences recently released a What Works Clearinghouse (WWC) educator’s practice guide that has four recommendations to support the development of foundational reading skills that are critically important to every student’s success. The recommendations in Foundational Skills to Support Reading for Understanding in Kindergarten Through 3rd Grade are based on a comprehensive review of 15 years of research on reading, and guidance from a national panel of reading experts, of which I was the chair.

Recently, the Regional Educational Laboratory (REL) Southeast at Florida State University has developed a set of professional learning community (PLC) materials and videos to help teachers and other practitioners implement the guide’s recommendations in classrooms.

Over the past few months, REL Southeast has shared the practice guide and PLC materials with practitioners and policymakers in two states – North Carolina and Mississippi, which both have K-3 reading initiatives and reading coaches who assist with implementation. I’m excited by the feedback we are getting.

During these presentations, we shared the format of the ten 75-min PLC sessions and accompanying videos that demonstrate the recommendations and action steps in actual classrooms. We filmed the videos in partnership with Dr. Lynda Hayes, Director of the PK Yonge Developmental Research School at the University of Florida, and her primary grade teachers.

In North Carolina, we trained K–3 regional literacy consultants, elementary teachers and reading coaches, and higher education faculty on the PLC Facilitator’s Guide in Charlotte and Raleigh. The K-3 regional literacy consultants are organized by the North Carolina Department of Public Instruction.

In Mississippi, we trained the 90 Mississippi Department of Education reading coaches and district-supported special education specialists in Jackson. In turn, the state coaches will train the K–3 teachers who are a part of the reading initiative in the practice guide recommendations and action steps. Additionally, the coaches will work with the primary grade teachers in each of their assigned schools to implement the PLC. Having the state coaches oversee the implementation of the PLC will help ensure commitment and instill depth to the PLC sessions.

Also present at the training in Mississippi were faculty members from the University of Mississippi and Belhaven University. I accepted an invitation from the Mississippi Institutions of Higher Learning Literacy Council to speak to higher education faculty about the guide and PLC materials. The invitation is timely because Mississippi recently completed a study of teacher preparation for early literacy instruction.

I hope you will download the practice guide and PLC materials. If you have any thoughts, comments, or questions, please email Contact.IES@ed.gov. You can learn more about the work of the Regional Educational Laboratories program and REL Southeast on the IES website.  

Dr. Foorman is the Director of REL Southeast, located at Florida State University

How ERIC is Helping Students

By Erin Pollard, ERIC Project Officer, NCEE

Over the past several years, the staff of ERIC has worked hard to get to know its users—we want to understand who uses ERIC and how they are using it. We found that the most common users are college undergraduates using the ERIC electronic library to write research papers as part of their school work. We also learned that they often weren’t searching for the best resources to meet their needs.

For example, some users told the ERIC help desk they were writing a paper for an introductory “contemporary issues in education” course, but they were requesting documents from over 20 years ago. Also, some of the documents they wanted were not peer-reviewed or particularly relevant to their topic. It became clear that we could do a better job addressing our most frequent users’ needs.

First, we redesigned the ERIC website to be more friendly and intuitive for those who are not familiar with traditional search methods, making it more likely they would get the best results (learn more about searching ERIC). Second, we worked to develop tools to help novice users, including a new series of how-to videos.

The first video in this new series is aimed at students—Using ERIC to Write a Research Paper. The ERIC team worked with undergraduate faculty members to outline a step-by-step approach to describe how a student should begin a search process, refine their results, and use different fields to find the resources they need. The video describes how to use ERIC to find full-text resources and how users can know that they found right resources.

We asked school of education faculty and librarians for their thoughts on this video and received a lot of positive feedback. Some professors said that they would link to this video on their syllabi and post it on their course management systems, while librarians have posted it on their content management system, called LibGuides. Take a look and see what you think.  And if you have feedback on this or any other ERIC videos, or suggestions for new videos and tools, let us know through the ERIC Help Desk.

 

Why We Need Large-Scale Evaluations in Education

By Elizabeth Warner, Team Leader for Teacher Evaluations, NCEE

These days, people want answers to their questions faster than ever, and education research and evaluation is no exception. In an ideal world, evaluating the impact of a program or policy would be done quickly and at a minimal expense. There is increasing interest in quicker-turnaround, low-cost studies – and IES offers grants specifically for these types of evaluations.

But in education research, quicker isn’t always better. It depends, in part, on the nature of the program you want to study and what you want to learn.

Consider the case of complex, multi-faceted education programs.  By their very nature, complex programs may require several years for all of the pieces to be fully implemented. Some programs also may take time to influence behavior and desired outcomes.  A careful and thorough assessment of these programs may require an evaluation that draws on a large amount of data, often from multiple sources over an extended period of time. Though such studies can require substantial resources, they are important for understanding whether and why an investment had an intended effect. This is especially true for Federal programs that involved millions of dollars of taxpayer money.

On August 24, IES plans to release a new report from a large-scale, multi-year study of a complex Federal initiative: the Teacher Incentive Fund (TIF) [1] This report – the third from this evaluation -- will provide estimated program impacts on student achievement after three years of implementation. The Impact Evaluation of the Teacher Incentive Fund is a $13.7 million study that stretches for more than six years and has reported interim findings annually since 2014. (The graphic above is from the a study snapshot of the first TIF evaluation report.) 

A “big study” of TIF was important to do, and here’s why.

Learning from the Teacher Incentive Fund

TIF is a Federal program that provides grants to districts that want to implement performance pay with the goal of improving teacher quality.  TIF grants awarded in 2010 included the requirement that performance pay be based on an educator evaluation system with multiple performance measures consistent with recent research. The grantees were also expected to use the performance measures to guide educator improvement.

The TIF evaluation provides an opportunity not only to learn about impacts on student achievement over time as the grant activities mature but also to get good answers to implementation questions such as:

  • How do districts structure the pay-for-performance bonus component of TIF?
  • Are educators even aware of the TIF pay-for-performance bonuses?
  • Do educators report having opportunities for professional development to learn about the measures and to improve their performance?
  • Do educators change their practice in ways that improve their performance measures? 
  • Are principals able to use their ability to offer pay-for-performance bonuses to hire or retain more effective teachers?  

Analyses to address these questions can suggest avenues for program improvement that a small-scale impact evaluation with limited data collection would miss. 

Studying the Initiative Over Time

With four years of data collection, the TIF evaluation is longer than most evaluations conducted by IES. But the characteristics of the TIF program made that extensive data collection important for a number of reasons.

 

First, TIF’s approach to educator compensation differs enough from the traditional pay structure that it might take time for educators to fully comprehend how it works. They likely need to experience the new performance measures to see how they might score and how that translates in terms of additional money earned. Second, educators might need to receive a performance bonus or see others receive one in order to fully believe it is possible to earn a bonus. (The chart pictured here is from the second report and compares teachers’ understanding with the actual size of the maximum bonuses that they could receive. This chart will be updated in the third report.)

That also might help them better understand what behaviors are needed to earn a bonus.  Finally, time might be needed for educators to respond to all of the intended policy levers of the program, particularly related to recruitment and hiring.

The TIF evaluation is designed to estimate an impact over the full length of the grants as well as provide rich information to improve the program. Sometimes, even after a number of years, some aspects of a program are never fully implemented. Thus, it may take time to see if whether it is even possible to implement a complex policy like TIF with fidelity.  An evaluation that only looks at initial implementation of a complex program may miss important program components that are incorporated or refined with time. Also, it may take several years to determine if the intended educator behaviors and desired outcomes of the policy are realized.

Learning how and why a policy does or does not work is central to program improvement.  For large, complex programs like TIF, this assessment is only possible with data-rich study over an extended period of time.     

 

[1] Under the newly reauthorized Elementary and Secondary Education Act, this program is now called the Teacher and School Leader Incentive Program, but for this blog, we will call it by its original name.

Gearing up for the Fall Semester? See How ERIC Can Help!

By Erin Pollard, ERIC Project Officer, NCEE

The ERIC Help Desk frequently receives questions from academic users asking if we have resources to help students and faculty use the Institute of Education Sciences' free online library of education research. In response, ERIC has created several videos that can be embedded in LibGuides (a content management system used by libraries), linked to in syllabi, and posted on course websites. These videos can also be accessed from ERIC’s Multimedia Page.

Our newest videos include:

About ERIC: The About ERIC video is an introduction to our digital library and answers basic questions about who our website is designed for and what types of information you’ll find there. Users can view this video to determine if ERIC is the right resource to use for their research.

Using ERIC to Write a Research Paper: The ERIC Help Desk has received a lot  of requests from students asking how to use ERIC to write a research paper.  This video was specifically created to be a resource that librarians and faculty members could direct students to as a starting point for research projects. For more information on this video, stay tuned for next week’s blog post!

Searching eric.ed.gov: ERIC was redesigned in 2013 using intuitive search technology, as opposed to more traditional searching mechanisms. This video explains how our search engine works and offers tips to help users get the best results.

How ERIC Selects New Sources: Users frequently ask how sources are selected to be indexed in ERIC. This video describes the process laid out in the ERIC Selection Policy and explains how users can recommend new resources for our collection.

IES Public Access Policy: How Grantees and Contractors Meet Requirements by Submitting Work to ERIC: Are you conducting research funded by IES? If so, then you are likely subject to the IES Public Access Policy (learn more about the policy for publications and data).  This video explains how researchers can meet their compliance requirements by submitting their work to ERIC.

Tips for Using the ERIC Online Submission System:  ERIC encourages authors to submit their work through our Online Submission System. Learn what types of materials are eligible to be included and how to prepare publications for submission to ERIC.

Have suggestions for other videos ERIC should make? Email our Help Desk! To learn more about ERIC, sign up for our Newsflash and follow us on Facebook and Twitter.