Skip Navigation

Recap: The 2022 Annual Principal Investigators Meeting

Mark Schneider, Director of IES | February 2, 2022

IES held its annual Principal Investigators Meeting from January 25–27. I gave opening remarks; I thought that I would share the key points of that talk with a wider audience.

Standards for Excellence in Education Research

Most of my talk was focused on the Standards for Excellence in Education Research, including some of the steps IES took to promote SEER in 2021 and laying out a few of the developments IES anticipates making in 2022. As a reminder, SEER encourages researchers to—

  1. Pre-register studies
  2. Make findings, methods, and data open
  3. Identify interventions' components
  4. Document treatment implementation and contrast
  5. Analyze interventions' costs
  6. Use high-quality outcome measures
  7. Facilitate generalization of study findings
  8. Support scaling of promising interventions

At the PI meeting, I also unveiled ongoing IES work on a ninth principle focused on equity. The working language calls for researchers to "Address inequities in societal resources and outcomes."

As with other SEER components, we will begin with a high-level statement of principle and then work to operationalize that principle with standards to guide the work we support. The creation of these standards is often a lengthy process but is required to shape rigorous education research that is transparent, actionable, and focused on outcomes with the potential to dramatically improve learner success across the lifespan.

We have incorporated SEER into our requests for applications as well as other IES work. We are committed to continue supporting, developing, and improving SEER to guide the education sciences, and have made or considered adjustments to the standards and their implementation. I noted these examples in my talk:

  • IES developed Sharing Study Data: A Guide for Education Researchers, which focuses on steps researchers can take to make their data available in accordance with the IES Public Access Policy (Principle 2: Making findings, methods, and data open). This guide will be published later this year.
  • We developed a guide called Enhancing the Generalizability of Study Findings in Education, which outlines steps PIs can take to increase the likelihood that findings from impact studies generalize to the populations of students they hope to serve (Principle 7: Facilitate generalization of study findings). This guide will be published within the next few months.
  • We supported SRI International in developing a guide, From Research to Market: Development of a Transition Process to Integrate Sustainable Scaling Methodologies into Education Innovation Research Design and Development (Principle 8: Support scaling of promising interventions). That report, available now, advances a framework for scaling innovations and includes a series of questions that PIs should consider prior to the development of an intervention to increase the likelihood of its adoption by educators.
  • IES will soon publish The BASIE Framework for Interpreting Findings from Impact Evaluations: A Practice Guide for Education Researchers. This guide will outline how researchers can augment traditional approaches to reporting impact estimates, which typically include effect sizes and p-values, with more user-friendly Bayesian estimates of the likelihood that impacts meet or exceed thresholds (for example, the likelihood that the effect size is 0.20 or greater). After decades of using the bright line of p=.05 (and after decades of misunderstanding what a p-value connotes), we hope that we can help the field to transition to better and more useful ways to report outcomes.
  • We are working with Mathematica to develop an initial framework for identifying intervention components and an approach to developing topically focused "nomenclatures" in foundational literacy and postsecondary developmental math education. We continue to test these nomenclatures and approaches to coding components to learn more about their fit and ease of use for the purpose of deciphering the impact and cost of individual components constituting any given intervention. Public-facing materials on these developments will be available this summer.

IES is committed to further developing standards operationalizing SEER principles, funding more resources to help the field adhere to SEER, and incorporating more and more of SEER into our funding decisions. We will also be increasing our monitoring of work we fund, with the goal of increasing adherence to and implementation of SEER.

Building infrastructure for the education sciences

John Whitmer, a fellow at IES who specializes in artificial intelligence (he possesses plenty of the human kind too), noted that much of AI (and other research) follows an "80–20" time allocation rule, in which researchers spend 80 percent of their time finding data and only 20 percent of their time analyzing it. One of IES' goals is to flip that division of labor to 20–80. We are investing in infrastructure that will (hopefully) allow more researchers to do their own work faster, cheaper, and better (yes, I know the old "pick two" saw, but we hope that strong infrastructure will allow us to move on all three fronts!).

This is the motivation behind our support for platforms. I have often talked about our two biggest platform pushes to date: the XPRIZE and our support for the SEERNet platform network. I have also noted our joint AI projects with NSF that should help many researchers move their work along. Two other infrastructure efforts described below are in the early stages of development.

IES data library

Machine learning, natural language processing (NLP), and AI are all frontiers that IES needs to explore in education research. But these approaches require large data sets to train and test models. IES has been exploring the possibility of creating a data library designed to facilitate machine learning, NLP, and AI. Building this library may start with a network approach in which IES funds experts in, say, five important education science topic areas to assemble large data sets that are privacy protected and address the racial bias that is evident in too many data sets and training models. The data libraries we are envisioning will be open to anyone wishing to train/test their models.

Researcher and practitioner matching

Many, many IES researchers complain that they spend too much time recruiting schools/districts into their research projects—another 80–20 situation. For this reason, IES is exploring the possibility of supporting the building of something like the system that matches medical school graduates to hospitals for medical internships. The algorithm for doing the match is known, so we wouldn't have to invent it, but the question is whether we could match schools that want help in choosing and/or evaluating interventions with researchers who want access to schools. Our ability to operate such a program would depend on the extent to which we are able to build the trust and relationships that we know are essential to a smooth partnership. Any thoughts on the desirability and feasibility of this idea are welcome.

As noted, the ideas regarding the data library and this matching facility are nascent, but I hope to have an RFA supporting the data library ready for this year's grant cycle.

A final note

Look for a rerun of the transformative RFA, which will incorporate several lessons from last year's competition. This will likely be competed along with the other RFAs that we will release later this year.

I have laid out several ideas that require some back and forth with the field. So, please take seriously my usual sign-off and contact me at about your thoughts and experiences with any of these efforts.