Inside IES Research

Notes from NCER & NCSER

Building Evidence: Changes to the IES Goal Structure for FY 2019

The IES Goal Structure was created to support a continuum of education research that divides the research process into stages for both theoretical and practical purposes. Individually, the five goals – Exploration (Goal 1), Development and Innovation (Goal 2), Efficacy and Replication (Goal 3), Effectiveness (Goal 4), and Measurement (Goal 5) – were intended to help focus the work of researchers, while collectively they were intended to cover the range of activities needed to build evidence-based solutions to the most pressing education problems in our nation. Implicit in the goal structure is the idea that over time, researchers will identify possible strategies to improve student outcomes (Goal 1), develop and pilot-test interventions (Goal 2), and evaluate the effects of interventions with increasing rigor (Goals 3 and 4).

Over the years, IES has received many applications and funded a large number of projects under Goals 1-3.  In contrast, IES has received relatively few applications and awarded only a small number of grants under Goal 4. To find out why – and to see if there were steps IES could take to move more intervention studies through the evaluation pipeline – IES hosted a Technical Working Group (TWG) meeting in 2016 to hear views from experts on what should come after an efficacy study (see the relevant summary and blog post). IES also issued a request for public comment on this question in July 2017 (see summary).

The feedback we received was wide-ranging, but there was general agreement that IES could do more to encourage high-quality replications of interventions that show prior evidence of efficacy. One recommendation was to place more emphasis on understanding “what works for whom” under various conditions.  Another comment was that IES could provide support for a continuum of replication studies.  In particular, some commenters felt that the requirements in Goal 4 to use an independent evaluator and to carry out an evaluation under routine conditions may not be practical or feasible in all cases, and may discourage some researchers from going beyond Goal 3.   

In response to this feedback, IES revised its FY 2019 RFAs for Education Research Grants (84.305A) and Special Education Research Grants (84.324A) to make clear its interest in building more and better evidence on the efficacy and effectiveness of interventions. Among the major changes are the following:

  • Starting in FY 2019, Goal 3 will continue to support initial efficacy evaluations of interventions that have not been rigorously tested before, in addition to follow-up and retrospective studies.
  • Goal 4 will now support all replication studies of interventions that show prior evidence of efficacy, including but not limited to effectiveness studies.
  • The maximum amount of funding that may be requested under Goal 4 is higher to support more in-depth work on implementation and analysis of factors that moderate or mediate program effects.

The table below summarizes the major changes. We strongly encourage potential applicants to carefully read the RFAs (Education Research, 84.305A and Special Education Research, 84.324A) for more details and guidance, and to contact the relevant program officers with questions (contact information is in the RFA).

Applications are due August 23, 2018 by 4:30:00 pm Washington DC time.

 

Name Change

Focus Change

Requirements Change

Award Amount Change

Goal 3

Formerly “Efficacy and Replication;” in FY2019, “Efficacy and Follow-Up.”

Will continue to support initial efficacy evaluations of interventions in addition to follow-up and retrospective studies.

No new requirements.

No change.

Goal 4

Formerly “Effectiveness;” in FY2019, “Replication: Efficacy and Effectiveness.”

Will now support all replications evaluating the impact of an intervention. Will also support Efficacy Replication studies and Re-analysis studies.

Now contains a requirement to describe plans to conduct analyses related to implementation and analysis of key moderators and/or mediators. (These were previously recommended.)

Efficacy Replication studies maximum amount: $3,600,000.

Effectiveness studies maximum amount: $4,000,000.

Re-analysis studies maximum amount: $700,000.

 

 

By Thomas Brock (NCER Commissioner) and Joan McLaughlin (NCSER Commissioner)

Announcing ED/IES SBIR’s 2018 Awards: Funding the Next Generation of Education Technology

In recent years, thousands of schools around the country have used technologies developed through the Small Business Innovation Research program at the U.S. Department of Education’s Institute of Education Sciences (ED/IES SBIR). The program emphasizes a rapid research and development (R&D) process, with rigorous research informing iterative development and evaluating the promise of products for improving the intended outcomes. ED/IES SBIR also focuses on the commercialization after development is complete so that products can reach schools and be sustained over time.


This month, IES announced 21 new awards for 2018. Of these, 15 are Phase I projects to develop and test a prototype, and six are Phase II projects to fully develop and evaluate an education technology product for students, teachers, or administrators to use in classrooms and schools. A playlist of videos from the Phase II projects is available below. 

Many of the new projects continue trends that have emerged across the portfolio in recent years, including creating learning games and dashboards that present data to inform learning and instruction in core subjects like reading and math. Several other awards are for projects that promote learning in new areas, such as computer science and Career and Technical Education.

Trend #1: Learning Games

Games are increasingly being used to engage students in learning by presenting content in new ways. For the eighth straight year, many new ED/IES SBIR awardees will be developing game-based learning products.

  • Phase II awardee Electric Funstuff and Phase I awardee Schell Games are developing virtual reality (VR) games to immerse students in history in 360-degree environments, and Phase I awardee Gigantic Mechanic is developing a role-playing game on civic discourse facilitated by tablet-based computers.
  • With Phase I funding, Fablevision and Sirius Thinking are creating interventions that employ game mechanics to improve reading.
  • Several project teams are embedding storylines within learning games, including Phase II awardee MidSchoolMath for algebra, and Phase I awardees Codespark for computer science and Immersed Games for ecosystems science.
  • The 3C Institute is creating a game-based assessment of early grade science learning.

Trend #2: Dashboards for Students, Teachers and Administrators

Modern technologies provide the opportunity to organize and present data in real-time to students, teachers, and administrators to inform learning and decision-making. Several new awards are developing data dashboards.

  • Phase II awardees are fully developing dashboards across several areas. LiveSchool will generate reports on students’ behavior across classes with a recommendation engine for administrators and teachers to address challenges, StoryWorld will provide teachers of English Learners insights into students’ language acquisition, and Simbulus and Myriad Sensors will present real-time information to enrich classroom discussions on math and science topics.
  • Phase I projects by Appendis and Graspable are creating adaptive learning technologies which include teacher dashboards to present results on student performance to guide instruction. VidCode is creating a dashboard for teachers to monitor student progress in learning coding. Education Modified is developing a dashboard to provide special education teachers guidance on student IEPs, or individual educational plans.

New Areas of Focus

Along with continuing to support projects in the areas above, a series of Phase I 2018 awards are focusing on areas new to ED/IES SBIR.  VidCode, CodeSpark, and Zyante are focusing on computer science learning and Core Learning is seeking to build capacity in Career and Technical Education (CTE). Language Learning Partners is developing an automated avatar tutor to support English Learners through conversation. And in special education, Attainment Company is developing an app for supporting student writing.

Stay tuned for updates on Twitter and Facebook as IES continues to support innovative forms of technology.

Written by Edward Metz, Program Manager, ED/IES SBIR

Computerized Preschool Language Assessment Extends to Toddlers

Identifying young children with language delays can improve later outcomes

Language is a core ability that children must master for success both in and out of the classroom. Extensive studies have shown that many tasks, including math, depend on linguistic skill, and that early language skills are predictive of school readiness and academic success. Being able to quickly identify children at early ages with language delays is crucial for targeting effective interventions.

Enter the QUILS.

In 2011, the National Center for Education Research (NCER) at IES funded a 4-year grant to Dr. Roberta Golinkoff (University of Delaware) and Drs. Kathy Hirsh-Pasek (Temple University) and Jill de Villiers (Smith College) to develop a valid and reliable computer-based language assessment for preschoolers aged 3-5 years old. The resulting product was the Quick Interactive Language Screener (QUILS), a computerized tool to measure vocabulary, syntax, and language acquisition skills. The assessment ultimately measures what a child knows about language and how a child learns, and automatically provides results and reports to the teacher.

The preschool version of QUILS is now being used by early childhood educators, administrators, reading specialists, speech-language pathologists, and other early childhood professionals working with young children to identify language delays. The QUILS is also being utilized in other learning domains. For example, a new study relied on the QUILS, among other measures, to examine links between approaches to learning and science readiness in over 300 Head Start students aged 3 to 5 years.

QUILS is now being revised for use with toddlers. In 2016, the National Center for Special Education Research (NCSER) funded a 3-year study to revise the QUILS for use with children aged 24-36 months. The researchers have been testing the tool in both laboratory and natural (child care centers, homes, and Early Head Start programs) settings to determine which assessment items to use in the toddler version of QUILS. Ultimately, these researchers aim to develop a valid and reliable assessment to identify children with language delays so that appropriate interventions can begin early.

By Amanda M. Dettmer, AAAS Science & Technology Policy Fellow Sponsored by the American Psychological Association Executive Branch Science Fellowship

NCSER Grantees Recognized by Council for Exceptional Children

Two NCSER-funded researchers were recently recognized for their contributions to the field of special education by the Council for Exceptional Children (CEC). They were honored at the CEC Convention and Expo earlier this month.

Headshot of Diane Browder

Diane Browder received the 2018 CEC Special Education Research Award, which recognizes an individual whose research has significantly advanced the education of children and youth with exceptionalities. Browder, Distinguished Professor of Special Education at the University of North Carolina at Charlotte, has served as Principal Investigator on multiple NCSER-funded grants. Through Project RAISE (Reading and Accommodations Interventions for Students with Emergent Literacy), she formed a partnership with local schools and community services to evaluate interventions designed to teach reading to students with moderate and severe mental retardation in Grades K – 3. Browder and her colleagues found that their Early Literacy Skills Builder intervention improves phonological awareness and phonics skills and that comprehensive reading instruction produces better reading outcomes when compared to instruction that provides sight words alone for students with intellectual disabilities in special education classrooms. Browder also developed math and science instruction for students with significant cognitive disabilities in Grades 3 – 10 who participate in alternate achievement assessments as well as instructional materials for teaching mathematical problem solving to students with moderate and severe intellectual disabilities in Grades 4 – 8. This research has shown that students with moderate and severe intellectual disabilities are capable of learning grade-level content in math and science, challenging long held assumptions about the academic potential of these populations.

 

Headshot of Sarah Powell

Sarah Powell received the 2018 Distinguished Early Career Research Award from CEC’s Division of Research, which recognizes individuals who have made outstanding scientific contributions to research in special education within the first 10 years after receiving their doctoral degrees. Powell is an Assistant Professor in the Department of Special Education at University of Texas at Austin.She is currently the Principal Investigator of a NCSER-funded grant evaluating the efficacy of equation-solving instruction, within the context of tutoring, for improving word-problem solving outcomes for students in Grade 3 with mathematics difficulties.

 

These NCSER-supported researchers have been contributing to the advancement of the field by investigating instruction in a variety of subjects for students with disabilities. Congratulations to the CEC award recipients!

 

By Amy Sussman, NCSER Program Officer

Resolve to Study Effectively in 2018

Students of all ages want to study more effectively and efficiently, while teachers want to improve their students’ understanding and retention of important concepts. The Institute’s Cognition and Student Learning (CASL) program has invested in several research projects that test the effectiveness of different strategies for improving learning and provide resources for teachers who want to implement these strategies in their classrooms.  Two strategies that are easy for teachers and students to implement and do not require a lot of time or money are retrieval practice and interleaving.

 

Retrieval practice (also known as test-enhanced learning) involves recalling information that has been previously learned. Research has shown that when students actively retrieve information from memory (e.g., through low-stakes quizzes), their ability to retain that information in the future improves when compared to other common study strategies like re-reading and highlighting key concepts while reading class texts.

Interleaving is the process of mixing up different types of problems during practice. Unlike blocking, where a student practices the same type of problem over and over again, interleaved practice involves multiple types of problems that require different strategies to solve. In mathematics, where interleaving and blocking have been studied most, blocking is frequently employed at the end of each chapter in a textbook. With interleaved practice, students must choose a strategy to solve the problem and apply that strategy successfully. Research has shown that students learn more when engaged in interleaved practice relative to blocked practice.

For decades, research has shown the benefits of both retrieval practice and interleaving for learning; however, until this point, there were no easily accessible resources that practitioners could turn to for concrete details and suggestions for how to implement these strategies in their classrooms. Two of the IES CASL research projects that have focused on these study strategies (Developing a Manual for Test-Enhanced Learning in the Classroom, PI: Henry Roediger and Interleaved Mathematics Practice, PI: Douglas Rohrer) resulted in guides for teachers that provide information on how to implement these strategies in their classrooms. These guides are freely available through the website www.retrievalpractice.org (download the Retrieval Practice Guide here and the Interleaving Guide here). With these guides, teachers can learn more about how to use retrieval practice and interleaving to improve their students’ understanding and retention of the concepts they are learning. Happy studying!

By Erin Higgins, NCER Program Officer, Cognition and Student Learning