Message from IES Director:
Changes Are Coming to Research Competitions
(Since the publication of this message, IES has continued to refine the LEED concept discussed below. Now referred to as IES's Standards for Excellence in Education Research, you can learn more by clicking here.)
The 2018 "proposal season" just ended and IES received over 900 applications. As many of you know, we instituted several changes in this year's RFAs. These changes were important but were mostly on the margin, since there was not enough time after I arrived at IES to make broader changes. In the next few months, IES will more systematically examine the existing RFAs with the goal of more extensive updates for next year. Each new Director at IES brings their own experience and expertise to the process, but we all build on the good work of the past.
I want to share some of our thinking about next steps.
I will begin with what may seem like a digression about the What Works Clearinghouse, one of IES' most important products. We are in the midst of designing a rating system for work included in the WWC. While we are not changing WWC evidence standards, we are working on an additional system modeled on the well-known "LEED" green building certification system. Following LEED, we intend to award research in the WWC different "certification" levels, such as platinum, gold, or silver. We are still working out the dimensions of the ratings, but they will include things such as registration, cost-effectiveness analysis, and the measurement of long term outcomes. To achieve the highest rating, studies will have to meet these dimensions.
Here's the connection to the RFAs: one purpose of the proposed WWC rating system is to recognize and incentivize more work using what IES views as the most valuable dimensions of education research to support evidence-based decision making.
Our RFAs should emphasize the same dimensions.
Clearly, not all aspects of the proposed certification process will fit with all types of research we fund, but, in general, more explicitly aligning WWC standards and the RFAs should create a more coherent and coordinated IES research agenda.
We moved in this direction this year by requiring registration of research designs and cost-effectiveness analysis, as appropriate. Our goal is to increase the pipeline of research that will eventually find its way into the WWC with more dimensions of high quality research. We will continue working on this synchronization over the coming year.
Simplification of RFAs
We also plan to simplify our RFAs. Presently, they all too often read like an accretion of planks and phrases, many of which were added to deal with a problem that we encountered 5 or 10 years ago. As a result, the RFAs have become overly complex and difficult to read.
We are vigorously pursuing a "plain language" initiative across all of IES, including our RFAs. Malcolm Gladwell's Revisionist History podcast devoted an episode to a particularly unpleasant event in the politics of science. The hero of the story is Bernadine Healy, the first woman to run the National Institutes of Health. While Healy demonstrated many admirable characteristics, her commitment to clear writing is central to the podcast. Healy summarized her approach to writing in simple, powerful guidance: "Strong verbs. Short sentences."
Now, consider the introduction to Goal 4 (Replication: Efficacy and Effectiveness) in the current Education and Special Education Research RFAs:
"The Replication: Efficacy and Effectiveness goal supports replication research under two broad categories: Efficacy Replications and Effectiveness Studies. Under this goal, the Institute supports Effectiveness studies, which carry out the independent evaluation of fully developed education interventions with prior evidence of efficacy to determine whether they produce a beneficial impact on student education outcomes relative to a counterfactual when they are implemented by the end user under routine conditions in authentic education settings. In addition, under this Goal the Institute will now also support Efficacy Replications and Re-analysis Studies..."
The key sentence is 55 words, and, in oh so many ways, violates "Short sentences. Strong verbs." It never gets to an easily understandable statement of what we are trying to accomplish with the tens of millions of dollars we will allocate to support Goal 4 research.
Similarly, IES support for replication research (part of Goal 4) and the identification of moderators and mediators should all be understood in the context of identifying the conditions under which interventions work and for whom. A unifying principle gets buried in far too much verbiage.
None of this is new or radical. In other places in the RFA, we clearly state that we are interested in identifying what works for whom and why and we also state that we want to fund work that meets the WWC evidence standards. To be clear, we will not change the link between our research and WWC evidence standards. Our goal is to use the new WWC ratings system to create a consistently strong statement about what we stand for and what we are trying to accomplish across the Institute.
We are exploring other changes in the lineup of research topics, such as how to encourage apprenticeships between pre-doc training programs and education agencies, how to best support new investigators, how to make sure that the research we support is used, and how to encourage the exploration of long term outcomes in research we support.
More to come. As always, we welcome feedback.
A coda:
I will be sending out updates as we continue working on the rating standards. But here's a task on which we could use your help (some of you may even find that the challenge passes for fun):
LEED stands for "Leadership in Energy and Environmental Design." We need something like that for the WWC set of standards. Here are a few we have considered:
- Leadership in Education Research (LIER—too close to LIAR?).
- Leadership in Evidence-Based Education Research (LEBER).
Dropping Leadership in favor of Excellence:
- Excellence in Education Research
- Excellence in Evidence-Based Education Research (EEBER).
Suggestions welcome!
Mark Schneider
Director