Inside IES Research

Notes from NCER & NCSER

IES Announces Forthcoming Funding Opportunity For the R&D of an “ROI Tool” to Inform Students’ Postsecondary Education and Career Decision Making

Students with electronic devices sitting against a wall.

Overview

On or about February 15, 2019, the Small Business Innovation Research Program at the US Department of Education’s Institute of Education Sciences (ED/IES SBIR) anticipates releasing a Special Topic Solicitation #91990019R0016 in Postsecondary Education. The solicitation will be announced through an IES Newsflash and will be posted here. It will request Phase I proposals for awards of up to $200,000 for 8 months to develop a prototype of a "ROI tool.” The tool will be designed to measure the costs versus benefits (the return on investment) of different postsecondary education and training programs to help students make well-informed choices about options to pursue after they complete high school.

Applicants must be a for-profit business 500 employees or less, and U.S. owned and operated. Applicants may partner with entities or organizations working on related initiatives in the field of postsecondary education, or may subcontract to non-profit researchers or individuals with specialized expertise as needed. The due date for submission for proposals will likely be on or about April 15, 2019, with awards in mid-June, and projects beginning shortly thereafter. All Phase I awardees will be eligible to apply for a Phase II award in 2020, for $900,000 for full scale development and research to test and validate the ROI tool.

Background

While many websites provide ways for students to explore colleges or careers and occupations of interest (e.g., such as College Scorecard and CareerOneStop), there is currently no tool that helps students understand the costs and benefits of individual postsecondary programs in an integrated, customizable, and user-friendly manner.  An ROI tool would likely combine information on individual program’s tuition and fees, time needed to complete, and expected earnings. Because these characteristics can vary significantly across programs and institutions, creating a single estimated measure of ROI would allow students to more easily compare postsecondary program options. If it helps students make better choices, it could lead to improved program completion rates, higher levels of employment and earnings, less education-related debt, and more satisfaction with their selected education and career paths. 

The ED/IES SBIR Special Topic intends to fund up to five (5) Phase I projects to (a) develop and research a prototype of an ROI tool, and (b) conduct planning and concept testing for a fully developed ROI tool that provides a user-friendly experience for students. The prototype of the ROI tool developed in Phase I shall integrate with one or more existing technology systems, data sets, data standards, or resources (such as CareerOneStop or College Scorecard), and add new data elements provided by an end-user.  After a successful Phase I project, it is anticipated that small businesses that win Phase II awards will complete the full-scale development of the ROI tool that was started in Phase I, including developing an interface to improve the experience of students using the ROI tool.

Because data for ROI at the program level may only be available from some states, regions, or sets of institutions at this time, it is expected that the scope of the ROI tool developed in Phase I & II would be limited and would not be an attempt to calculate ROI for every program and institution in the country. Applicants must propose a project scope that appropriately reflects the datasets that are to be integrated within the new ROI tool, and the amount of funding and time allotted for development and research of the SBIR awards in Phase I and II.  Small businesses that are interested in this solicitation must have expertise with related efforts in the field to enhance student choices by linking education and workforce information.

Potential applicants may submit questions to ED’s Contracting Specialist Kevin.Wade@ed.gov. All questions and responses will be posted publically on the same website where the solicitation is posted as Amendments to the Solicitation.

 

Building Evidence: Changes to the IES Goal Structure for FY 2019

The IES Goal Structure was created to support a continuum of education research that divides the research process into stages for both theoretical and practical purposes. Individually, the five goals – Exploration (Goal 1), Development and Innovation (Goal 2), Efficacy and Replication (Goal 3), Effectiveness (Goal 4), and Measurement (Goal 5) – were intended to help focus the work of researchers, while collectively they were intended to cover the range of activities needed to build evidence-based solutions to the most pressing education problems in our nation. Implicit in the goal structure is the idea that over time, researchers will identify possible strategies to improve student outcomes (Goal 1), develop and pilot-test interventions (Goal 2), and evaluate the effects of interventions with increasing rigor (Goals 3 and 4).

Over the years, IES has received many applications and funded a large number of projects under Goals 1-3.  In contrast, IES has received relatively few applications and awarded only a small number of grants under Goal 4. To find out why – and to see if there were steps IES could take to move more intervention studies through the evaluation pipeline – IES hosted a Technical Working Group (TWG) meeting in 2016 to hear views from experts on what should come after an efficacy study (see the relevant summary and blog post). IES also issued a request for public comment on this question in July 2017 (see summary).

The feedback we received was wide-ranging, but there was general agreement that IES could do more to encourage high-quality replications of interventions that show prior evidence of efficacy. One recommendation was to place more emphasis on understanding “what works for whom” under various conditions.  Another comment was that IES could provide support for a continuum of replication studies.  In particular, some commenters felt that the requirements in Goal 4 to use an independent evaluator and to carry out an evaluation under routine conditions may not be practical or feasible in all cases, and may discourage some researchers from going beyond Goal 3.   

In response to this feedback, IES revised its FY 2019 RFAs for Education Research Grants (84.305A) and Special Education Research Grants (84.324A) to make clear its interest in building more and better evidence on the efficacy and effectiveness of interventions. Among the major changes are the following:

  • Starting in FY 2019, Goal 3 will continue to support initial efficacy evaluations of interventions that have not been rigorously tested before, in addition to follow-up and retrospective studies.
  • Goal 4 will now support all replication studies of interventions that show prior evidence of efficacy, including but not limited to effectiveness studies.
  • The maximum amount of funding that may be requested under Goal 4 is higher to support more in-depth work on implementation and analysis of factors that moderate or mediate program effects.

The table below summarizes the major changes. We strongly encourage potential applicants to carefully read the RFAs (Education Research, 84.305A and Special Education Research, 84.324A) for more details and guidance, and to contact the relevant program officers with questions (contact information is in the RFA).

Applications are due August 23, 2018 by 4:30:00 pm Washington DC time.

 

Name Change

Focus Change

Requirements Change

Award Amount Change

Goal 3

Formerly “Efficacy and Replication;” in FY2019, “Efficacy and Follow-Up.”

Will continue to support initial efficacy evaluations of interventions in addition to follow-up and retrospective studies.

No new requirements.

No change.

Goal 4

Formerly “Effectiveness;” in FY2019, “Replication: Efficacy and Effectiveness.”

Will now support all replications evaluating the impact of an intervention. Will also support Efficacy Replication studies and Re-analysis studies.

Now contains a requirement to describe plans to conduct analyses related to implementation and analysis of key moderators and/or mediators. (These were previously recommended.)

Efficacy Replication studies maximum amount: $3,600,000.

Effectiveness studies maximum amount: $4,000,000.

Re-analysis studies maximum amount: $700,000.

 

 

By Thomas Brock (NCER Commissioner) and Joan McLaughlin (NCSER Commissioner)

What’s Next for the Reading for Understanding Research Initiative?

After years of intense collaboration and research, the Reading for Understanding (RfU) Research Initiative is coming to an end. But the initiative’s work continues through recently announced IES-funded grants.

Over six years, research teams in the RfU network designed and tested new interventions that aim to improve reading comprehension in students in all grade levels and developed new measures of reading comprehension and component skills that support it. The initiative led to several new and important findings. In the coming years, several teams will build on that work through new research projects funded by IES’ National Center for Education Research (NCER) and National Center for Special Education Research (NCSER).  

During the RfU initiative, the Promoting Adolescents’ Comprehension of Text (PACT) team found positive effects in improving the content-area reading comprehension of middle school students. The PACT intervention uses social studies content to engage students and teach them to build coherent representations of the ideas in texts. Through a new grant from NCER, the PACT team will be testing the effectiveness of the intervention in middle school social studies classrooms in eight states.

Another group of researchers from the PACT team are starting a new project with funding from NCSER to design and test a technology-based intervention aimed at improving how middle school students with reading disabilities make inferences while reading.  

The RfU assessment team is also launching a new NCER-funded project to develop a digital assessment appropriate for adults, in particular those reading between the 3rd- to 8th-grade levels. Building on the Global, Integrated Scenario-Based Assessment (GISA) developed in RfU, the team intends for this new assessment to help determine an adult reader's strengths and weaknesses, inform instruction, and improve programs and institutional accountability. In addition, this team is using assessment items developed with RfU funding to explore the relationship between high school students' background knowledge and their reading comprehension.  

Finally, the Florida State University RfU team is continuing to explore which combination of interventions will improve the early language skills that are foundational to mastery of reading. In this new project, the researchers will examine the relative efficacy and sustained impacts of a language and vocabulary intervention for prekindergarten and kindergarten students, with variations on when and how long the intervention is used.

Written by Elizabeth Albro, Associate Commissioner, NCER

Supporting Early Career Researchers

An important part of the mission at the Institute of Education Sciences (IES) is to support the work of scholars who are early in their independent research careers. An example of that commitment can be seen in the latest round of grants from the National Center for Education Research (NCER), which includes eight grants being led by early career scholars.  These principal investigators are all individuals who began their independent research careers within the last five years.

Under the Education Research Grants competition, there are four NCER-funded grants that were awarded to early career scholars.

Abigail Gray, Senior Research Specialist at the University of Pennsylvania, who was an IES predoctoral fellow, is now leading a grant to study the efficacy of Zoology One, an integrated science and literacy intervention for kindergarten students. At the Cincinnati Children’s Hospital Medical Center, Assistant Professor Stephen Becker is examining the academic, social, and emotional problems experienced by children with sluggish cognitive tempo—specific attention-related symptoms such as excessive daydreaming, mental confusion, and slowed thinking or behavior. 

       

(From l to r: Abigail Gray, Stephen Becker, Shaun Dougherty, and Sonia Cabell)

Shaun Dougherty, Assistant Professor at the University of Connecticut, is looking at whether attending a high school in the Connecticut Technical High School System has an impact on achievement, high school graduation, and college enrollment. And Sonia Cabell, a Research Scientist from the University of Virginia, is leading a team that will study the efficacy of the Core Knowledge Language Arts Listening and Learning intervention for children in kindergarten through second grade. This program is designed to guide teachers in their reading out loud to their students in classrooms.

NCER awarded another set of Early Career grants through its Statistics and Research Methodology in Education grant program. This year, NCER made three awards under this competition.

      

(From l to r: Chun Wang, James Pustejovsky​, Nathan VanHoudnos, and Benjamin Castleman)

Finally, Benjamin Castleman, Assistant Professor at the University of Virginia, is leading a team in the Scalable Strategies to Support College Completion Network. Research Networks involve a number of research teams working together to address a critical education problem or issue. Castleman’s team will examine whether text messages that provide personalized information will help students at open- and broad-access enrollment institutions to complete their degrees.

Written by Becky McGill-Wilkinson, Education Research Analyst, NCER

Low-Cost, Short-Duration Evaluations: Helping States and School Districts Make Evidence-based Decisions

Last year, IES launched two new grant programs to help state and district education agencies obtain rigorous and timely evidence they can use to inform decisions about education interventions of high importance to the agency: Low-Cost, Short-Duration Evaluation of Education Interventions (CFDA 84.305L) and Low-Cost, Short-Duration Evaluation of Special Education Interventions (CFDA 84.324L). These programs are intended to support relatively quick (completed within 2 years) and inexpensive ($250,000 or less) evaluations to test the effects of education interventions that schools and other education institutions may want to scale up or redesign to improve student education outcomes.

IES developed the Low-Cost, Short Duration Evaluation grant programs to help states and school districts plan and launch rigorous evaluations when they roll out new interventions. Such interest is likely to intensify under the Every Student Succeeds Act, which requires states to set accountability goals and encourages use of evidence-based programs to close achievement gaps and improve low-performing schools. 

Advances in technology and administrative data systems have also led some policymakers and researchers to suggest that rigorous evaluations ought to be less expensive to perform today than in the past, when evaluators had to do much of their own data collection.

To meet the goals of relevance, rigor, and affordability, the Low-Cost, Short-Duration Evaluation grant programs set out specific requirements and expectations for applicants:

  • To ensure the relevance, the evaluations must be focused on an intervention that the education agency identifies as a priority and will be implemented in Year 1 of the grant, and must be carried out as a partnership between a research institution and a state and local education agency. 
  • To ensure rigor, evaluation designs must either be randomized controlled trials or regression discontinuity designs that have the potential to meet the What Works Clearinghouse evidence standards without reservation. Beginning in FY 2017, single-case designs may also be used in evaluations of special education interventions. 
  • To ensure lower-cost, the evaluations are expected to rely mainly on agency administrative datasets or other secondary sources rather than on new data collection. 

The short-term nature of the grants leads to a focus on interventions that are expected to improve student education outcomes within a summer session, quarter, semester or academic year. The low cost also means that researchers are not expected to conduct in-depth research into program implementation, comparison group practices, and costs and benefits of interventions unless the data exist in the administrative datasets. Other IES funding opportunities – including Education Research Grants, Special Education Research Grants, and Evaluation of State and Local Education Policies and Programs – are available to support longer-term and/or more in-depth evaluations. 

In sum, the Low-Cost, Short-Duration Evaluation grant programs fill a special niche within the IES portfolio by providing results on specific types of interventions that education agencies can use in making important decisions on programs, policies, and spending. They can also lay the groundwork for future grant-funded research to investigate implementation, underlying mechanisms, and longer-term impacts.

The results from last year’s competitions will be released soon and both programs are being re-competed this year with an application deadline of August 4, 2016.  On June 9, 2016, 2:00-3:30 p.m. (EDT), IES will host a webinar on the two grant competitions. Please register in advance. An archive of the webinar will also be available on the IES funding webinar page.

Written by Allen Ruby, Associate Commissioner, National Center for Education Research