NCEE Blog

By Dr. Barbara Foorman, Director Emeritus, Florida Center for Reading Research, Florida State University

The Institute of Education Sciences recently released a What Works Clearinghouse (WWC) educator’s practice guide that has four recommendations to support the development of foundational reading skills that are critically important to every student’s success. The recommendations in Foundational Skills to Support Reading for Understanding in Kindergarten Through 3^rd Grade are based on a comprehensive review of 15 years of research on reading, and guidance from a national panel of reading experts, of which I was the chair.

Recently, the Regional Educational Laboratory (REL) Southeast at Florida State University has developed a set of professional learning community (PLC) materials and videos to help teachers and other practitioners implement the guide’s recommendations in classrooms.

Over the past few months, REL Southeast has shared the practice guide and PLC materials with practitioners and policymakers in two states – North Carolina and Mississippi, which both have K-3 reading initiatives and reading coaches who assist with implementation. I’m excited by the feedback we are getting.

During these presentations, we shared the format of the ten 75-min PLC sessions and accompanying videos that demonstrate the recommendations and action steps in actual classrooms. We filmed the videos in partnership with Dr. Lynda Hayes, Director of the PK Yonge Developmental Research School at the University of Florida, and her primary grade teachers.

In North Carolina, we trained K–3 regional literacy consultants, elementary teachers and reading coaches, and higher education faculty on the PLC Facilitator’s Guide in Charlotte and Raleigh. The K-3 regional literacy consultants are organized by the North Carolina Department of Public Instruction.

In Mississippi, we trained the 90 Mississippi Department of Education reading coaches and district-supported special education specialists in Jackson. In turn, the state coaches will train the K–3 teachers who are a part of the reading initiative in the practice guide recommendations and action steps. Additionally, the coaches will work with the primary grade teachers in each of their assigned schools to implement the PLC. Having the state coaches oversee the implementation of the PLC will help ensure commitment and instill depth to the PLC sessions.

Also present at the training in Mississippi were faculty members from the University of Mississippi and Belhaven University. I accepted an invitation from the Mississippi Institutions of Higher Learning Literacy Council to speak to higher education faculty about the guide and PLC materials. The invitation is timely because Mississippi recently completed a study of teacher preparation for early literacy instruction.

I hope you will download the practice guide and PLC materials. If you have any thoughts, comments, or questions, please email Contact.IES@ed.gov. You can learn more about the work of the Regional Educational Laboratories program and REL Southeast on the IES website.  

Dr. Foorman is the Director of REL Southeast, located at Florida State University

By Erin Pollard, ERIC Project Officer, NCEE

Over the past several years, the staff of ERIC has worked hard to get to know its users—we want to understand who uses ERIC and how they are using it. We found that the most common users are college undergraduates using the ERIC electronic library to write research papers as part of their school work. We also learned that they often weren’t searching for the best resources to meet their needs.

For example, some users told the ERIC help desk they were writing a paper for an introductory “contemporary issues in education” course, but they were requesting documents from over 20 years ago. Also, some of the documents they wanted were not peer-reviewed or particularly relevant to their topic. It became clear that we could do a better job addressing our most frequent users’ needs.

First, we redesigned the ERIC website to be more friendly and intuitive for those who are not familiar with traditional search methods, making it more likely they would get the best results (learn more about searching ERIC). Second, we worked to develop tools to help novice users, including a new series of how-to videos.

The first video in this new series is aimed at students—Using ERIC to Write a Research Paper. The ERIC team worked with undergraduate faculty members to outline a step-by-step approach to describe how a student should begin a search process, refine their results, and use different fields to find the resources they need. The video describes how to use ERIC to find full-text resources and how users can know that they found right resources.

We asked school of education faculty and librarians for their thoughts on this video and received a lot of positive feedback. Some professors said that they would link to this video on their syllabi and post it on their course management systems, while librarians have posted it on their content management system, called LibGuides. Take a look and see what you think.  And if you have feedback on this or any other ERIC videos, or suggestions for new videos and tools, let us know through the ERIC Help Desk.

By Elizabeth Warner, Team Leader for Teacher Evaluations, NCEE

These days, people want answers to their questions faster than ever, and education research and evaluation is no exception. In an ideal world, evaluating the impact of a program or policy would be done quickly and at a minimal expense. There is increasing interest in quicker-turnaround, low-cost studies – and IES offers grants specifically for these types of evaluations.

But in education research, quicker isn’t always better. It depends, in part, on the nature of the program you want to study and what you want to learn.

Consider the case of complex, multi-faceted education programs.  By their very nature, complex programs may require several years for all of the pieces to be fully implemented. Some programs also may take time to influence behavior and desired outcomes.  A careful and thorough assessment of these programs may require an evaluation that draws on a large amount of data, often from multiple sources over an extended period of time. Though such studies can require substantial resources, they are important for understanding whether and why an investment had an intended effect. This is especially true for Federal programs that involved millions of dollars of taxpayer money.

On August 24, IES plans to release a new report from a large-scale, multi-year study of a complex Federal initiative: the Teacher Incentive Fund (TIF) [1] This report – the third from this evaluation -- will provide estimated program impacts on student achievement after three years of implementation. The Impact Evaluation of the Teacher Incentive Fund is a $13.7 million study that stretches for more than six years and has reported interim findings annually since 2014. (The graphic above is from the a study snapshot of the first TIF evaluation report.) 

A “big study” of TIF was important to do, and here’s why.

Learning from the Teacher Incentive Fund

TIF is a Federal program that provides grants to districts that want to implement performance pay with the goal of improving teacher quality.  TIF grants awarded in 2010 included the requirement that performance pay be based on an educator evaluation system with multiple performance measures consistent with recent research. The grantees were also expected to use the performance measures to guide educator improvement.

The TIF evaluation provides an opportunity not only to learn about impacts on student achievement over time as the grant activities mature but also to get good answers to implementation questions such as:

• How do districts structure the pay-for-performance bonus component of TIF?
• Are educators even aware of the TIF pay-for-performance bonuses?
• Do educators report having opportunities for professional development to learn about the measures and to improve their performance?
• Do educators change their practice in ways that improve their performance measures? 
• Are principals able to use their ability to offer pay-for-performance bonuses to hire or retain more effective teachers?  

Analyses to address these questions can suggest avenues for program improvement that a small-scale impact evaluation with limited data collection would miss. 

Studying the Initiative Over Time

With four years of data collection, the TIF evaluation is longer than most evaluations conducted by IES. But the characteristics of the TIF program made that extensive data collection important for a number of reasons.

First, TIF’s approach to educator compensation differs enough from the traditional pay structure that it might take time for educators to fully comprehend how it works. They likely need to experience the new performance measures to see how they might score and how that translates in terms of additional money earned. Second, educators might need to receive a performance bonus or see others receive one in order to fully believe it is possible to earn a bonus. (The chart pictured here is from the second report and compares teachers’ understanding with the actual size of the maximum bonuses that they could receive. This chart will be updated in the third report.)

That also might help them better understand what behaviors are needed to earn a bonus.  Finally, time might be needed for educators to respond to all of the intended policy levers of the program, particularly related to recruitment and hiring.

By Erin Pollard, ERIC Project Officer, NCEE

The ERIC Help Desk frequently receives questions from academic users asking if we have resources to help students and faculty use the Institute of Education Sciences' free online library of education research. In response, ERIC has created several videos that can be embedded in LibGuides (a content management system used by libraries), linked to in syllabi, and posted on course websites. These videos can also be accessed from ERIC’s Multimedia Page.

Our newest videos include:

About ERIC: The About ERIC video is an introduction to our digital library and answers basic questions about who our website is designed for and what types of information you’ll find there. Users can view this video to determine if ERIC is the right resource to use for their research.

Using ERIC to Write a Research Paper: The ERIC Help Desk has received a lot  of requests from students asking how to use ERIC to write a research paper.  This video was specifically created to be a resource that librarians and faculty members could direct students to as a starting point for research projects. For more information on this video, stay tuned for next week’s blog post!

Searching eric.ed.gov: ERIC was redesigned in 2013 using intuitive search technology, as opposed to more traditional searching mechanisms. This video explains how our search engine works and offers tips to help users get the best results.

How ERIC Selects New Sources: Users frequently ask how sources are selected to be indexed in ERIC. This video describes the process laid out in the ERIC Selection Policy and explains how users can recommend new resources for our collection.

IES Public Access Policy: How Grantees and Contractors Meet Requirements by Submitting Work to ERIC: Are you conducting research funded by IES? If so, then you are likely subject to the IES Public Access Policy (learn more about the policy for publications and data).  This video explains how researchers can meet their compliance requirements by submitting their work to ERIC.

Tips for Using the ERIC Online Submission System:  ERIC encourages authors to submit their work through our Online Submission System. Learn what types of materials are eligible to be included and how to prepare publications for submission to ERIC.

Have suggestions for other videos ERIC should make? Email our Help Desk! To learn more about ERIC, sign up for our Newsflash and follow us on Facebook and Twitter.

By Vanessa Anderson, Research Scientist, NCEE

The What Works Clearinghouse (WWC) was founded in 2002 and, in its first decade, focused mainly on reviewing studies of programs, policies, products and practices—or interventions—for improving student outcomes in pre-K, elementary and secondary schools. But in 2012, the WWC broadened its focus and has been using rigorous standards to review studies of interventions designed to increase the success of students in postsecondary education.

This week, the WWC launches a new topic—Supporting Postsecondary Success—and it is a good time to look at the work we’re doing, and will do, in the postsecondary area. 

The WWC postsecondary topic area includes reviews of studies on a wide range of interventions, including learning communities, summer bridge programs, multi-faceted support programs, academic mentoring, and interventions that aim to reduce performance anxiety. As of today, 294 postsecondary studies have been reviewed by the WWC. Those reviews are summarized in six Intervention Reports, 25 Single Study Reviews, and four Quick Reviews. And there’s much more in the works!  For instance, a WWC Educator’s Practice Guide that includes strategies for supporting students in developmental education is planned for publication later this year. (Learn more about Practice Guides)

Identifying Studies for Review

In the postsecondary topic area, there are currently three main ways that studies are identified by the WWC for review.

The first is studies that are reviewed for WWC Intervention Reports. All WWC Intervention Reports use a systematic review process to summarize evidence from all available studies on a given intervention. The WWC conducts a broad search for all publicly available studies of interventions that are related to the topic. This process often identifies hundreds of studies for review. The effectiveness studies are then reviewed against WWC standards. Only the highest quality studies are summarized in an Intervention Report.

We released two new intervention reports this week as part of our new Supporting Postsecondary Success topic. You can view the new Intervention Reports on Summer Bridge programs and first-year experience courses on the WWC website.

The second way that studies are reviewed by the WWC is through Quick Reviews, which are performed on studies that have received a great deal of media attention. In these reports, the WWC provides a brief description of the study, the author-reported results, and a study rating. We like to think of Quick Reviews as a way to help people decide whether to fully believe the results of a study, based on the research design and how the study was conducted. For example, we released a quick review earlier this month that focused on a study of computer usage and student outcomes for a class at the U.S. Military Academy at West Point.

Finally, the WWC reviews postsecondary studies submitted as supporting evidence for discretionary grant competitions funded by the U.S. Department of Education, such as the Strengthening Institutions Program, First in the World and TRIO Student Support Services. These grant competitions require applicants to submit studies as evidence of the effectiveness of the interventions they propose to implement. The WWC reviews these studies and includes the results of those reviews in our database.

If you want to see all the studies on postsecondary interventions that have been reviewed by WWC you can check out—and download—the Reviewed Studies Database. In the “Topic Areas” dropdown menu, just select “Postsecondary,” and then easily customize the search by rating, publication type, and/or reasons for the review (such as a grant competition).  

For more information, visit the WWC postsecondary topic area on the website. To stay up-to-date on WWC news, information, and products, follow us on Facebook, Twitter and sign up for the WWC newsflash!

By Diana McCallum, Education Research Analyst, What Works Clearinghouse

It’s been more than a decade since the first What Works Clearinghouse reports were released and we have a wealth of information and resources that can help educators and leaders make evidence-based decisions about teaching and learning. Since 2005, the WWC has assessed more than 11,500 education studies using rigorous standards and has published hundreds of resources and guides across many content areas. (View the full version of the graphic to the right.) 

The WWC website has already received more than 1.7 million page views this year, but if you haven’t visited whatworks.ed.gov lately, here are five reasons you might want to click over:

1) We are always adding new and updated reviews. Multiple claims about programs that work can be overwhelming and people often lack time to sift through piles of research. That’s where the WWC comes in. We provide an independent, objective assessment of education research. For example, we have intervention reports that provide summaries of all of the existing research on a given program or practice that educators can use to help inform their choices.  In addition, when a new education study grabs headlines, the WWC develops a quick review that provides our take on the evidence presented to let you know whether the study is credible. In 2015, we added 43 publications to WWC and we’re adding more every month this year.

2) We’ve expanded our reach into the Postsecondary area. In late 2012, the WWC expanded its focus to include reviews of studies within the Postsecondary area to capture the emerging research on studies on a range of topics, from the transition to college to those that focus on postsecondary success.  To date, the WWC has reviewed over 200 studies on postsecondary programs and interventions, and this area continues grow rapidly. In fact, several Office of Postsecondary Education grant competitions add competitive priority preference points for applicants that submit studies that meet WWC standards. (Keep an eye out for a blog post on the postsecondary topic coming soon!)

3) You can find what works using our online tool. Wondering how to get started with so many resources at your fingertips? Find What Works lets you do a quick comparison of interventions for different subjects, grades, and student populations. Want to know more about a specific intervention? We’ve produced more than 400 intervention reports to provide you the evidence about a curriculum, program, software product, or other intervention for your classroom before you choose it.  Recently, we’ve added a feature that allows a user to search for interventions that have worked for different populations of students and in different geographic locations. As we mentioned in a recent blog post, the Find What Works tool is undergoing an even bigger transformation this September, so keep visiting!

4) We identify evidence-based practices to use in the classroom. The WWC has produced 19 practice guides that feature practical recommendations and instructional tips to help educators address common challenges. Practice guides (now available for download as ebooks) provide quick, actionable guidance for educators that are supported by evidence and expert knowledge within key areas.  Some of our guides now feature accompanying videos and brief summaries that demonstrate recommended practices and highlight the meaning behind the levels of evidence. The work of practice guides are also actively disseminated during Regional Educational Laboratory (REL) Bridge events. For instance, REL Southwest held a webinar on Teaching Math to Young Children, which was based on a WWC practice guide. For more information, read a previously published blog post on practice guides.

5) We compile information by topic. Our “Special Features” pages focus on common themes in education, such as tips for college readiness, information for heading back to school, and guidance for what works in early childhood education. These Special Features provide a starting point to access a variety of WWC resources related to a topic.

In the coming months, we’ll post other blogs that will explore different parts of the WWC and tell you about ongoing improvements. So keep visiting the What Works website or signup to receive emails when we release new reports or resources. You can also follow us on Facebook and Twitter.

The What Works Clearinghouse is a part of the National Center for Education Evaluation and Regional Assistance in the Institute of Education Sciences (IES), the independent research, evaluation, and statistics arm of the U.S. Department of Education. You can learn more about IES’ other work on its website or follow IES on Twitter and Facebook. 

By Erin Pollard, ERIC Project Officer, NCEE

ERIC builds a strong education research collection by continuously seeking out new sources of rigorous content and adding them to the collection. But how does ERIC select publications for the online library?

A new video (embedded below) provides the answer to how ERIC selects new sources, including education-focused journals, grey literature reports, and conference papers. The video was developed to help answer one of the most frequently asked questions by ERIC users and to help publishers and organizations producing materials in the field of education understand what ERIC considers when evaluating potential new sources. Watch this video if you want to learn about the types of resources ERIC will and will not index, the source selection process, and how to recommend a new resource.

Twice a year, in the spring and fall, ERIC reviews journals and producers of conference papers, reports, and books as potential candidates for inclusion in ERIC, using a revised selection policy as a guide when evaluating recommended content. The revised policy was released in January 2016 to clarify the types of materials ERIC is seeking for the collection. ERIC considers resources that are education research focused and include citations, orginal analyses of data, and well-formed arguments. ERIC also considers collection priorities, such as peer- reviewed and full-text materials.

We are continuously working to build a strong education research collection that includes the latest and very best resources in the field. If you are a publisher of high-quality education research, or if you have a favorite journal, or know a source of conference papers or reports not currently in ERIC, please send us your recommendations.

To stay up-to-date on ERIC, follow us on Facebook, YouTube, or Twitter and check out the Notes area of the ERIC website at eric.ed.gov. 

By Thomas Wei, Evaluation Team Leader, NCEE

NOTE: In an effort to turn around the nation’s chronically low-performing schools, the Department of Education injected more than $6 billion into the federal School Improvement Grants (SIG) program over the past several years. SIG schools received a lot of money for a short period of time—up to $6 million over three years—to implement a number of prescribed improvement practices.

What is the prognosis for low-performing schools now that many federal School Improvement Grants (SIG) are winding down? This is an important question that the National Center for Education Evaluation and Regional Assistance (NCEE) addressed through its Study of School Turnaround. 

The second and final report from this study was released on April 14 and describes the experiences of 12 low-performing schools as they implemented SIG from 2010 to 2013 (Read a blog post on the first report). Findings are based on analyses of teacher surveys and numerous interviews with other school stakeholders, such as district administrators, principals, assistant principals, members of the school improvement team, instructional coaches, and parents.

After three years trying a diverse array of improvement activities ranging from replacing teachers to extending learning time to installing behavioral support systems, most of the 12 schools felt they had changed in primarily positive ways (see chart below from report).

The report also found that schools with lower organizational capacity in the first year of SIG appeared to boost their capacity by the final year of SIG. At the same time, schools with higher capacity appeared generally able to maintain that capacity.

Many experts believe that organizational capacity is an important indicator of whether a low-performing school can improve (see chart below showing schools with higher organizational capacity also appeared more likely to sustain improvements). Organizational capacity is indicated by for example, how strong a leader the principal is, how consistent school policies are with school goals, how much school leaders and staff share clear goals, how much collaboration and trust there is among teachers, and how safe and orderly the school climate is.

Despite these promising results, the report found that the overall prospects for sustaining any improvements appeared to be fragile in most of these 12 schools. The report identified four major risk factors, including (1) anticipated turnover or loss of staff; (2) leadership instability; (3) lack of district support, particularly with regard to retaining principals and teachers; and (4) loss of specific interventions such as professional learning or extended day programs. Most of the case study schools had at least one of these major risk factors, and a number of schools had multiple risk factors.

It is important to note that this study cannot draw any causal conclusions and that it is based on surveys and interviews at a small number of schools that do not necessarily reflect the experiences of all low-performing schools. Still, it raises interesting questions for policymakers as they consider how best to deploy limited public resources in support of future school improvement efforts that will hopefully be long-lasting.

NCEE has a larger-scale study of SIG underway that is using rigorous methods to estimate the impact of SIG on student outcomes. The findings from the case studies report released last week may yield important contextual insights for interpreting the overall impact findings. These impact findings are due out later this year, so stay tuned.

By Thomas Wei, Evaluation Team Leader

NOTE: Since 2009, the Department of Education has invested more than $6 billion in School Improvement Grants (SIG). SIG provided funds to the nation’s persistently lowest-achieving schools to implement one of four improvement models. Each model prescribed a set of practices, for example: replacing the principal, replacing at least 50 percent of teachers, increasing learning time, instituting data-driven instruction, and using “value-added” teacher evaluations.

Other than outcomes, how similar are our nation’s low-performing schools? The answers to this question could have important implications for how best to improve these, and other, schools. If schools share similar contexts, it may be more sensible to prescribe similar improvement practices than if they have very different contexts.

This is one of the central questions the National Center for Education Evaluation and Regional Assistance is exploring through its Study of School Turnaround. The first report (released in May 2014) described the experiences of 25 case study schools in 2010-2011, which was their first year implementing federal School Improvement Grants (SIG).

The report found that even though the 25 SIG schools all struggled with a history of low performance, they were actually quite different in their community and fiscal contexts, their reform histories, and the root causes of their performance problems. Some schools were situated in what the study termed “traumatic” contexts, with high crime, incarceration, abuse, and severe urban poverty. Other schools were situated in comparatively “benign” contexts with high poverty but limited crime, homes in good repair, and little family instability. All schools reported facing challenges with funding and resources, but some felt it was a major barrier to improvement while others felt it was merely a nuisance. Some schools felt their problems were driven by student behavior, others by poor instruction or teacher quality, and still others by the school’s external context such as crime or poverty.

Given how diverse low-performing schools appear to be, it is worth wondering whether they need an equally diverse slate of strategies to improve. Indeed, the report found that the 25 case study schools varied in their improvement actions even with the prescriptive nature of the SIG models (see the chart above, showing school improvement actions used by sample schools).

It is important to note that this study cannot draw any causal conclusions and that it is based on a small number of schools that do not necessarily reflect the experiences of all low-performing schools. Still, policymakers may wish to keep this finding in mind as they consider how to structure future school improvement efforts.

The first report also found that all but one of the 25 case study schools felt they made improvements in at least some areas after the first year of implementing SIG. Among the issues studied in the second report, released April 14, 2016, is whether these schools were able to build on their improvements in the second and third year of the grant. Read a blog post on the second report.

UPDATED APRIL 18 to reflect release of second report.

By Chris Boccanfuso Education Research Analyst, NCEE

One of the most commonly asked questions about the Regional Educational Laboratory, or REL, program is how we choose the applied research and evaluation studies, analytic technical assistance, and dissemination activities we provide for free to stakeholders every year. The answer is simple – we don’t!

Instead, the REL staff at the Institute of Education Sciences (IES) and our contractors who run the nation’s ten regional labs listen to the voices of teachers, administrators, policymakers, and students to identify and address high-leverage problems of practice and build the capacity of stakeholders. In other words, these groups determine where the RELs should use their resources to make the largest, most lasting impact possible on practice, policy and, ultimately, student achievement.

How do the RELs do this? Through a variety of activities we collectively refer to as needs sensing. The following are a few examples of how the RELs engage in needs sensing:

Research Alliances: Research Alliances are a type of researcher–practitioner partnership where a group of education stakeholders convene around a specific topic of concern to work collaboratively to investigate a problem and build capacity to address it. Alliances can be made up of many types of stakeholders, such as teachers, administrators, researchers and members of community organizations. Alliances can vary in size and scope and address a variety of topics. For instance, alliances have formed to address topics as broad as dropout prevention and as specific as Hispanic students’ STEM performance. The vast majority of the RELs’ work is driven by these research alliances.

While the RELs’ 79 research alliances are incredibly diverse, one thing each alliance has in common is that they collectively develop a research agenda. These agendas can change, as the alliance continually weighs the questions and needs of various groups against the types of services and the resources available to address these needs. Not every need has to be addressed through a multi-year research study. Sometimes, it can be addressed through a workshop, a literature review, or a “Bridge Event”, where national experts on certain topic work with practitioners to provide the information that alliance members need, when they need it. Sometimes, a need is state or district-specific, is related to the impact of a specific program, or covers a topic where the existing research literature is thin. In these cases, a research study may be most appropriate.