Here’s an update on several months of work trying to reimagine IES’ research topics.
I thought my focus on changing topic areas was going to be a gangbuster reform; however, few people shared my enthusiasm. I thank all of you who took the time to send their thoughts and opinions about the topic areas—but the consensus was “Why bother?”
This does not mean that this effort was a failure—indeed, there were important lessons to be had. And even if existing topic areas remain for now, we will change and clarify the RFAs in response to comments we received.
Most importantly, we will continue to emphasize the questions that any proposal’s research plan must address. To do this, we will continue to emphasize SEER principles. For example, under “meaningful outcomes,” we are considering expanding our expectations that researchers justify the practical importance of the outcome they are trying to improve and give us their best estimate of the size of the effect (and, of course, the justification for that estimate). Research plans will be expected to highlight core components and clearly demonstrate how they will be identified and tested. We will also expect researchers to address long-term outcomes—which are critically important but largely neglected—as part of the research plan. Indeed, we are experimenting with giving additional money to extend grants that are in their last year but whose results have great promise.
I understand that we are making research plans more complicated and expensive to execute. We are likely therefore to increase the size of grants. Depending on appropriations, that may mean we award fewer grants.
Many of these ideas are already part of the RFA, but we will be even more clear to researchers and reviewers about what should be emphasized in the proposal and in peer review.
Finally, a word about off-cycle competitions: I will continue to work with people in and out of IES to identify special topics to compete off-cycle. I view these off-cycle grants as experiments for future work.
Of the three off-cycle ideas I floated previously—using state longitudinal data systems (SLDS) to measure long-term outcomes, NAEP process data, the next wave of systematic replications—we are moving ahead with the SLDS outcomes and systematic replication competitions. Unfortunately, the NAEP process data will not be ready in time, but we plan to hold a competition as soon as it is available.
As part of the topics discussion, we also explored project types. As you know, like many other funding agencies, IES project types connote a progression from exploration to development to efficacy testing to effectiveness trials. And, as you all know, this is a tortuous path where very few projects get to effectiveness trials (and only a few of those show evidence of effectiveness).
We have been looking at the pattern of movement “up the ladder” of testing (indeed, we considered changing the name of the project types to “step 1, 2, 3 and 4”). Our data show not a neat, linear progression but a series of twisting, turning pathways. Projects that move up the steps often involve people who were not involved in earlier ones. But even a twisting road is heading toward someplace, and we intend to monitor our grants more carefully to ensure they move toward more rigorous testing.
My biggest struggle is with development grants. During one of our deliberations, a project officer noted that we have funded 10 different development projects from the same researcher, who never pursued subsequent steps of testing. Research Commissioner Albro assures me that this is an anomaly—but only in the number of development grants this person received.
I have heard several times that “our developers” do not necessarily have the skills or interests in going to the next step of testing their projects and bringing them to market. Given that the world is full of developers who have that goal, should IES be in the development business at all?
Commissioner Albro has been forceful in arguing that we need to continue to invest in development projects, at least in some topic areas. I am willing to defer to her (at least for now) with the following proviso: IES RFAs will require a clear plan for moving development projects and products into more rigorous trials and for scaling up and/or commercialization. This plan will be a bigger part of the review process and incorporated into any judgment regarding merit. This change will make our development grants look more like our highly successful SBIR program, with the obvious difference that organizations other than small businesses can apply. By the way, many of our most successful SBIR grants are built around innovative partnerships between researchers and people with backgrounds in business and product development. To be sure, scaling up and commercialization are not new ideas, but they will be strengthened in our next round of RFAs.
Just as we will ask researchers to be more explicit about how to move to scaling up and testing, we will hold ourselves more accountable for achieving that same goal. We are exploring ways to modify the research centers’ annual performance indicators to include measures of how well we are moving successful projects into scale-up and testing. Core to our mission is to increase the use of our products in order to change outcomes; hence, we need to create incentives inside of IES (as we are doing with our grantees) to do this.
Our mission is to improve the condition of education using research, statistics, and evaluation. To fulfill that mission, IES is one of the world’s biggest investors in education research, and—combining NCER and NCSER—we manage a research portfolio well over $200 million per year. This is a substantial investment of taxpayer money. We take responsibilities that go with that money seriously and will continue to experiment with ways of making our work faster, better, and cheaper. IES will inevitably have failures along the way—but failure shows that we are trying.As always, I welcome your comments. Please know that I read these comments and that they don’t just disappear into some black hole. Email me directly at firstname.lastname@example.org