The Leaning Tower of PISA
Mark Schneider, Director of IES | March 3, 2019
I was recently involved in an exchange about PISA, the Program for International Student Assessment, in which I argued that PISA is stuck in a three-year testing cycle that is no longer necessary. It’s not just that the three-year cycle is too short and too expensive—which it is—it’s that the current cycle ultimately detracts from the quality of PISA.
The three-year cycle means that critical R&D work—including fundamentals, such as correcting errors, confirming the accuracy of translations, and validating item constructs through cognitive labs and think aloud protocols—cannot be done before the preparation must start for the next testing cycle. I argued that, given today’s technology, a five-year cycle would allow for the three foundational core domains of PISA (reading, math, and science) to be measured in depth with high quality.
To be clear, more R&D is needed in the main assessment of the three core domains. But PISA’s practices in pursuit of new “innovative” domains are even more problematic. These domains are supposed to assess new skills required in the ever-changing global economy.
Simply put, PISA is not doing a good job in developing instruments that assess those skills. For example, in 2018 PISA, 40 countries chose not to participate in the “Global Competencies” assessment. The U.S. has decided not to participate in the 2021 Creative Thinking assessment—and we have already heard from several other countries that they too will not participate. Our testing and content experts believe that the tasks in these questionnaires are inconsistent with the competencies these innovative domains are supposed to assess—that is, the tests lack validity. Maybe just as bad, these innovative assessments are essentially “one-offs.” Rather than building on what is learned and perfecting the instrument, PISA tests the innovative domain and then moves on to the next shiny object.
In short, in its core assessment and its innovative domains, PISA is rushing to develop assessments for ideas that are complex and difficult to measure. This rushed practice results in measurements that are bad and potentially harmful. The solution is to slow down the PISA cycle.
The U.S. has been a strong supporter of PISA from its inception. We continue to support PISA. But we want to see PISA live up to the standards that made it the world’s most prominent international education assessment.
Dr. Michele Bruniges, the chair of the OECD PISA Governing Board, published a rejoinder in Ed Week that implies that my commentary called for less testing for political reasons (it doesn’t) and makes no reference to my argument that we should change PISA’s periodicity to ensure sufficient time for proper research and development. Dr. Bruniges’ comment ends with the following paragraph: “… it may be wiser to accelerate our efforts to learn from the world’s most rapidly improving education systems, rather than slowing down the rate at which we measure educational progress. We compare the rate of economic growth among our countries every three months, and the evolution of unemployment every six months, so seeing how well we prepare our students for their future every three years doesn’t seem exaggerated.”
Had I called for changing PISA’s periodicity simply to reduce testing, this might be on point. Yet, what does “accelerate our efforts” mean? Administering PISA every year? Every six months? Every quarter? And how would more frequent and, with even less time between cycles, less accurate assessments help anyone?
My core concern is that if we do not improve the quality of PISA’s measurement and create sufficient time to ensure the quality of its data, then we will not be accelerating our learning about the world’s top education systems even if we were to administer PISA every month. PISA needs to take more time between testing to get things right if it is to be a high-quality measurement of educational progress.
Righting PISA
Slowing down, investing in needed R&D, and linking assessment items to the core ideas in PISA will make PISA stronger and provide a more solid foundation that will stop PISA from leaning further and further off center.
Mark Schneider
Director of IES
Mark.Schneider@ed.gov