The crisp September mornings we've been having in Washington, DC of late evoke happy childhood memories for many of us. Of course, they signal the start of a new school year. But for me, they signal nothing so much as the return of a fall ritual . . . the Kansas State Fair! Naturally, you can find corndogs, pig races, and butter cows at the Fair. But if you search just a bit, you can find a favorite of fairgoers everywhere: Whac-a-Mole. (It's probably right next to a stand selling something fried that, cholesterol-wise, shouldn't be.)
Whac-a-Mole is refreshingly straightforward. As electric moles pop up from holes on the board, the player "whacks" them on the head with a rubber mallet. Simple, you say? It is—at first. At first, only one mole is coming up from one hole. Easy to whack. But then more moles start popping up from more holes. The pace picks up, inconveniently erratic. Before long, you're whacking mole-less holes on the one part of the board on which you've focused while you're missing the moles popping up and down elsewhere. In the end, you're left with the sick feeling that at least as many moles got away as were cleanly dispatched.
As is turns out, the challenges of Whac-a-Mole bear an uncanny resemblance to those faced by the What Works Clearinghouse (WWC). (This had to turn to the education sciences at some point, right?) In this analogy, the Whac-a-Mole board is the totality of researchable education policy and practice, and each hole is a pressing issue someone cares desperately about. Each mole is the latest pre-print, journal article, or research report on a topic of interest. And the mallet? That's the WWC. Here's how WWC Whac-a-Mole plays out in real life.
For the bulk of its history, the WWC has organized much of its work around the publications—that is, the Practice Guides and Intervention Reports —that it intends to produce. Imagine WWC needs-sensing suggests there's a need for evidence-based guidance on improving secondary writing. In response, IES would award a contract to author a Practice Guide on that specific topic and assign a WWC staff member to oversee the work.
Once awarded, work on the Practice Guide begins in earnest. For three years, give or take, some share of the WWC's bandwidth would be taken up by the review and synthesis of high-quality causal research on the focal topic. Given staffing and other resource levels, four to five such efforts are typically ongoing at one time, with each at a different stage of the research process (e.g., protocol development, literature review, evidence synthesis, recommendation generation). As each contract draws to a close and its publications are released, the WWC turns its attention to the next priority topic on the docket. Perhaps not surprisingly, several—sometimes many—years can pass by before the WWC can return to a topic it has already explored.
Metaphorically speaking, this way of working leaves the WWC intensely focused on whacking moles in a small number of holes for an extended period (think: Practice Guide topics), while leaving other holes unmonitored (think: everything else). When the WWC shifts its attention from one part of the Whac-a-Mole board to another (think: beginning a new Practice Guide), it is blind both to new moles popping up in recently monitored holes (think: potentially game-changing research) and to new holes that spontaneously appear on the board (think: emergent topics, like AI or chronic absenteeism). If only there was a better way! Turns out, there may be.
This past year, the WWC has been lucky to participate in a project led by the Federation of American Scientists and the Future Evidence Foundation to explore an emergent approach to evidence synthesis: the living evidence model. As its name implies, living evidence is a dynamic process of evidence synthesis. Rather than commissioning systematic evidence reviews and meta-analyses that address discrete topics over finite timeframes, the living evidence model calls us to constantly monitor the flow of new research across a wide range of issues, consider our existing understandings in the light of new findings, and update practice recommendations when the weight of the most up-to-date evidence demands it. (To learn more about how this model has been successfully applied elsewhere, consider the experience of Australia during the COVID-19 pandemic.)
Adopting the living evidence model wouldn't require the WWC to change its standards for rigorous research or its technical approach to synthesizing study findings. But it would provide an opportunity to think about how the WWC organizes its work. We refer to one approach under consideration as "Evidence Hubs."
The basic notion of "Evidence Hubs" is that IES would segment the education research landscape into several broad topical areas, such as "language and literacy" or "STEM." Instead of entering the work with a predefined interest in a specific topic and conducting a systematic literature review in service of a related Practice Guide, the Hub would broadly review all the most recent causal research within its domain. Hubs would generate evidence maps to better understand the supply of high-quality research, and through enhanced engagement with communities of educators and policymakers, better understand where demand for evidence is greatest. IES would then focus part of the Hub's effort on the places where supply and demand align.
Importantly, though, Evidence Hubs must be designed as multi-taskers. Even when one part of a Hub is focused on creating the next WWC Practice Guide, for example, another continues to monitor the rest of its part of the Whac-a-Mole board, to review new studies, and to update the systematic reviews and meta-analyses that are the foundation for WWC Intervention Reports and Practice Guide recommendations. This approach has two benefits. First, it gives confidence to educators and policymakers that the WWC is keeping up with the latest research and that our current publications still reflect the field's best understanding of "what works." Second, if up-to-date systematic reviews and meta-analyses suggest that the evidence base is shifting, the WWC can release new publications on relevant topics sooner rather than later. It feels—to me at least—a win-win.
We're eager to begin piloting this new approach to the WWC's work. Whether you find this approach promising or perilous, I'd love to hear your feedback! Please send questions, comments, or concerns my way. I can be reached at matthew.soldner@ed.gov.
