Inside IES Research

Notes from NCER & NCSER

Celebrating the Launch of the Registry of Efficacy and Effectiveness Studies (REES)

The Registry of Efficacy and Effectiveness Studies (REES) is now ready for use! REES is a registry of causal impact studies in education developed with a grant from the National Center for Education Research (NCER). REES will increase transparency, improve the replicability of studies, and provide easy access to information about completed and ongoing studies.

The release of REES aligns with recent IES efforts to promote study registration. In the FY 2019 Requests for Applications (RFAs) for the Education Research and Special Education Research Grants Programs, IES recommended that applicants describe a plan for pre-registering their studies in both the project narrative (as part of the research) as well as the data management plan. IES is also developing the Standards for Excellence in Education Research (SEER) and has identified study registration as an important dimension of high value education research.

We asked the REES team to tell us more about how the registry works.

How can researchers access REES? REES can be accessed through the SREE website at www.sreereg.org. Over the next year, REES will transition to a permanent home at the Inter-university Consortium for Political and Social Research (ICPSR) at the University of Michigan, but it will still be accessible through the SREE link.

Screenshot of REESWhat kinds of studies can be registered? REES is a registry of causal impact studies. It accommodates a range of study designs including randomized controlled trials, quasi-experimental designs, regression discontinuity designs, and single case designs.

What information should be included in a study entry? A REES entry includes basic study information and a pre-analysis plan. The checklist of required information for a registry entry provides detailed information for each of the different design options. All of the information for a REES entry should be easily found in a grant application.

How long does it take to register a study? For a study with a complete grant application, completing a REES entry should be straightforward and take approximately one hour.

What if a study entry needs to be changed? Principal investigators (PI) or other authorized research team members should update a REES entry as changes occur. All updates to an entry will be time-stamped. Original entries and updated entries will be publically available.

Are registered studies searchable by the public? Yes! When a PI or authorized research team member is ready to make the study available in the public domain, they click on the publish option. This will time stamp the entry and make it publically available. REES entries that are published are available on the search page. A pdf of individual entries can be downloaded from the search page or an Excel file of multiple entries can be exported.

What will happen to studies that were entered in the pilot phase of REES? A REES entry that was started and/or completed during the pilot phase is a part of the REES database. To make the study publically available and a part of the searchable database, the PI or other authorized research team member needs to click on the publish option for the entry.

Over the next two years, the REES team will be working to ensure the sustainability and visibility of REES with a grant from the National Center for Special Education Research (NCSER). To do this, the team will transfer REES to its permanent location on the ICPSR website and disseminate information about REES within the education research community, as well as with funders, publishers, and users of education research, through meetings, conferences, websites, social media, and targeted outreach.

So, what are you waiting for? Go check it out!

If you have questions about REES, please email contact@sreereg.org.

 

 

Informing Future Research in Career and Technical Education (CTE)

Career and Technical Education (CTE) has been evolving and expanding at a rapid pace in recent years as industry and education leaders focus on students’ readiness for college and careers. While some studies have shown positive effects of CTE on students, the evidence base is thin. To learn more about the research needs of the CTE field, the National Center for Special Education Research (NCSER) and the National Center for Education Research (NCER) at the Institute of Education Sciences (IES) convened a group of experts in policy, practice and research related to CTE.  The discussion held by the Technical Working Group (TWG) led both NCER and NCSER to increase their investments in CTE for fiscal year 2019: NCSER included a CTE special topic, and NCER changed its CTE topic from a special topic to a standing topic. Applications to both are due August 23, 2018.  Both research centers hope to fund more studies that will help us better understand this growing aspect of education.

The TWG focused on the following four questions:

  1. Who is served by CTE and who is left behind? From national CTE statistics, we know that 82% of all public high schools offer CTE. And, 85% of students earn at least one credit in CTE with the average high school student earning 2.5 CTE credits. However, TWG members noted that research is lacking on specific subpopulations in CTE, such as students from various demographic backgrounds and students with disabilities. Disaggregated data on these dimensions are needed to better understand the CTE experiences of the range of students being served. Such data may help educators improve equity of access to high quality programs for all students.
  2. What do we know―and need to know―about CTE policies, programs, and practices at the secondary and postsecondary levels?  TWG experts discussed the need to know more about industry-recognized credentials and about business and industry engagement in CTE at the secondary level. They argued that we do not know if credentials align with industry requirements, nor do we understand the impact of different types of credentials on student outcomes and wage trajectories. TWG members also noted that the higher the perceived quality or prestige of the CTE program, the more exclusive it becomes, and the more difficult it is for disadvantaged students to obtain access. TWG members also expressed concerns about CTE teacher training, particularly for experts who are recruited from industry without prior teacher preparation. As the experts discussed postsecondary CTE, they suggested that the field would be best served by framing the conversation about secondary to postsecondary pathways as a continuum that enables transparent and sequential transitions from secondary to 2-year and then to 4-year programs or to training or employment, with guidance for students to understand possible sequences.
  3. What are the critical methodological issues in CTE?  TWG members noted that, with a few notable exceptions (e.g.,  a 2008  MDRC study on career academies in New York and a recent study of CTE high schools in Massachusetts), few causal studies on CTE have been conducted. There is an urgent need for more high quality, causal research on CTE policies and programs. In addition, the experts noted that there is almost no research on students with disabilities in CTE. TWG members concluded that the field needs to re-conceptualize CTE research – including better defining CTE students, instructors, programs, and measures – and identify the critical research questions in order to encourage more research in this field.
  4. What is needed to advance CTE research?  State CTE administrators want to know how to identify quality CTE programs so they know how to spend their dollars most effectively on programs that best meet the needs of students. Policymakers also want to know what “works” and what the benefits are of such investments. The TWG members encouraged studies that examine the educational benefits of particular instructional approaches. They also highlighted the importance of collaborative cross-institutional and cross-agency efforts to advance CTE research.

Readers are invited to read the summary of the TWG discussion.

By Corinne Alfeld (NCER program officer) and Kimberley Sprague (former NCSER program officer)

Building Evidence: Changes to the IES Goal Structure for FY 2019

The IES Goal Structure was created to support a continuum of education research that divides the research process into stages for both theoretical and practical purposes. Individually, the five goals – Exploration (Goal 1), Development and Innovation (Goal 2), Efficacy and Replication (Goal 3), Effectiveness (Goal 4), and Measurement (Goal 5) – were intended to help focus the work of researchers, while collectively they were intended to cover the range of activities needed to build evidence-based solutions to the most pressing education problems in our nation. Implicit in the goal structure is the idea that over time, researchers will identify possible strategies to improve student outcomes (Goal 1), develop and pilot-test interventions (Goal 2), and evaluate the effects of interventions with increasing rigor (Goals 3 and 4).

Over the years, IES has received many applications and funded a large number of projects under Goals 1-3.  In contrast, IES has received relatively few applications and awarded only a small number of grants under Goal 4. To find out why – and to see if there were steps IES could take to move more intervention studies through the evaluation pipeline – IES hosted a Technical Working Group (TWG) meeting in 2016 to hear views from experts on what should come after an efficacy study (see the relevant summary and blog post). IES also issued a request for public comment on this question in July 2017 (see summary).

The feedback we received was wide-ranging, but there was general agreement that IES could do more to encourage high-quality replications of interventions that show prior evidence of efficacy. One recommendation was to place more emphasis on understanding “what works for whom” under various conditions.  Another comment was that IES could provide support for a continuum of replication studies.  In particular, some commenters felt that the requirements in Goal 4 to use an independent evaluator and to carry out an evaluation under routine conditions may not be practical or feasible in all cases, and may discourage some researchers from going beyond Goal 3.   

In response to this feedback, IES revised its FY 2019 RFAs for Education Research Grants (84.305A) and Special Education Research Grants (84.324A) to make clear its interest in building more and better evidence on the efficacy and effectiveness of interventions. Among the major changes are the following:

  • Starting in FY 2019, Goal 3 will continue to support initial efficacy evaluations of interventions that have not been rigorously tested before, in addition to follow-up and retrospective studies.
  • Goal 4 will now support all replication studies of interventions that show prior evidence of efficacy, including but not limited to effectiveness studies.
  • The maximum amount of funding that may be requested under Goal 4 is higher to support more in-depth work on implementation and analysis of factors that moderate or mediate program effects.

The table below summarizes the major changes. We strongly encourage potential applicants to carefully read the RFAs (Education Research, 84.305A and Special Education Research, 84.324A) for more details and guidance, and to contact the relevant program officers with questions (contact information is in the RFA).

Applications are due August 23, 2018 by 4:30:00 pm Washington DC time.

 

Name Change

Focus Change

Requirements Change

Award Amount Change

Goal 3

Formerly “Efficacy and Replication;” in FY2019, “Efficacy and Follow-Up.”

Will continue to support initial efficacy evaluations of interventions in addition to follow-up and retrospective studies.

No new requirements.

No change.

Goal 4

Formerly “Effectiveness;” in FY2019, “Replication: Efficacy and Effectiveness.”

Will now support all replications evaluating the impact of an intervention. Will also support Efficacy Replication studies and Re-analysis studies.

Now contains a requirement to describe plans to conduct analyses related to implementation and analysis of key moderators and/or mediators. (These were previously recommended.)

Efficacy Replication studies maximum amount: $3,600,000.

Effectiveness studies maximum amount: $4,000,000.

Re-analysis studies maximum amount: $700,000.

 

 

By Thomas Brock (NCER Commissioner) and Joan McLaughlin (NCSER Commissioner)

Announcing ED/IES SBIR’s 2018 Awards: Funding the Next Generation of Education Technology

In recent years, thousands of schools around the country have used technologies developed through the Small Business Innovation Research program at the U.S. Department of Education’s Institute of Education Sciences (ED/IES SBIR). The program emphasizes a rapid research and development (R&D) process, with rigorous research informing iterative development and evaluating the promise of products for improving the intended outcomes. ED/IES SBIR also focuses on the commercialization after development is complete so that products can reach schools and be sustained over time.


This month, IES announced 21 new awards for 2018. Of these, 15 are Phase I projects to develop and test a prototype, and six are Phase II projects to fully develop and evaluate an education technology product for students, teachers, or administrators to use in classrooms and schools. A playlist of videos from the Phase II projects is available below. 

Many of the new projects continue trends that have emerged across the portfolio in recent years, including creating learning games and dashboards that present data to inform learning and instruction in core subjects like reading and math. Several other awards are for projects that promote learning in new areas, such as computer science and Career and Technical Education.

Trend #1: Learning Games

Games are increasingly being used to engage students in learning by presenting content in new ways. For the eighth straight year, many new ED/IES SBIR awardees will be developing game-based learning products.

  • Phase II awardee Electric Funstuff and Phase I awardee Schell Games are developing virtual reality (VR) games to immerse students in history in 360-degree environments, and Phase I awardee Gigantic Mechanic is developing a role-playing game on civic discourse facilitated by tablet-based computers.
  • With Phase I funding, Fablevision and Sirius Thinking are creating interventions that employ game mechanics to improve reading.
  • Several project teams are embedding storylines within learning games, including Phase II awardee MidSchoolMath for algebra, and Phase I awardees Codespark for computer science and Immersed Games for ecosystems science.
  • The 3C Institute is creating a game-based assessment of early grade science learning.

Trend #2: Dashboards for Students, Teachers and Administrators

Modern technologies provide the opportunity to organize and present data in real-time to students, teachers, and administrators to inform learning and decision-making. Several new awards are developing data dashboards.

  • Phase II awardees are fully developing dashboards across several areas. LiveSchool will generate reports on students’ behavior across classes with a recommendation engine for administrators and teachers to address challenges, StoryWorld will provide teachers of English Learners insights into students’ language acquisition, and Simbulus and Myriad Sensors will present real-time information to enrich classroom discussions on math and science topics.
  • Phase I projects by Appendis and Graspable are creating adaptive learning technologies which include teacher dashboards to present results on student performance to guide instruction. VidCode is creating a dashboard for teachers to monitor student progress in learning coding. Education Modified is developing a dashboard to provide special education teachers guidance on student IEPs, or individual educational plans.

New Areas of Focus

Along with continuing to support projects in the areas above, a series of Phase I 2018 awards are focusing on areas new to ED/IES SBIR.  VidCode, CodeSpark, and Zyante are focusing on computer science learning and Core Learning is seeking to build capacity in Career and Technical Education (CTE). Language Learning Partners is developing an automated avatar tutor to support English Learners through conversation. And in special education, Attainment Company is developing an app for supporting student writing.

Stay tuned for updates on Twitter and Facebook as IES continues to support innovative forms of technology.

Written by Edward Metz, Program Manager, ED/IES SBIR

Computerized Preschool Language Assessment Extends to Toddlers

Identifying young children with language delays can improve later outcomes

Language is a core ability that children must master for success both in and out of the classroom. Extensive studies have shown that many tasks, including math, depend on linguistic skill, and that early language skills are predictive of school readiness and academic success. Being able to quickly identify children at early ages with language delays is crucial for targeting effective interventions.

Enter the QUILS.

In 2011, the National Center for Education Research (NCER) at IES funded a 4-year grant to Dr. Roberta Golinkoff (University of Delaware) and Drs. Kathy Hirsh-Pasek (Temple University) and Jill de Villiers (Smith College) to develop a valid and reliable computer-based language assessment for preschoolers aged 3-5 years old. The resulting product was the Quick Interactive Language Screener (QUILS), a computerized tool to measure vocabulary, syntax, and language acquisition skills. The assessment ultimately measures what a child knows about language and how a child learns, and automatically provides results and reports to the teacher.

The preschool version of QUILS is now being used by early childhood educators, administrators, reading specialists, speech-language pathologists, and other early childhood professionals working with young children to identify language delays. The QUILS is also being utilized in other learning domains. For example, a new study relied on the QUILS, among other measures, to examine links between approaches to learning and science readiness in over 300 Head Start students aged 3 to 5 years.

QUILS is now being revised for use with toddlers. In 2016, the National Center for Special Education Research (NCSER) funded a 3-year study to revise the QUILS for use with children aged 24-36 months. The researchers have been testing the tool in both laboratory and natural (child care centers, homes, and Early Head Start programs) settings to determine which assessment items to use in the toddler version of QUILS. Ultimately, these researchers aim to develop a valid and reliable assessment to identify children with language delays so that appropriate interventions can begin early.

By Amanda M. Dettmer, AAAS Science & Technology Policy Fellow Sponsored by the American Psychological Association Executive Branch Science Fellowship