IES Centers: A Reassessment

Mark Schneider, Director of IES | May 12, 2020

At the end of March, the American Statistical Association (ASA) and the American Education Research Association (AERA) sent a joint letter to Congress focused on shortfalls in the budget and staffing levels for the National Center of Education Statistics (NCES). In that letter, ASA and AERA also dismissed the proposal that a new center focused on education assessments be created within IES, moving NAEP and the international assessments (such as TIMSS, PISA, and PIAAC) out of NCES into this center. I want to explain why I support this move.

Let me begin by saying that I proudly served as Commissioner of NCES in the mid-2000s. I have great respect for the staff and the mission of NCES. Further, NCES is one of the constituent centers of IES and, as Institute Director, my support for NCES' statistical work and its commitment to the collection and dissemination of education statistics continues. But education sciences change and how the IES centers are organized needs to keep pace.

Prior to 2002, NCES was an independent organization within the Department of Education with a long and proud history. When IES was created in 2002, Congress moved NCES into IES, and NAEP was moved as part of NCES. But from the very creation of IES, Congress has kept appropriations for NAEP separate from NCES' statistical activities, a de facto recognition that statistical data collection and NAEP assessments are separate and separable activities. Indeed, the very fact that NAEP has its own governing board established by Congress reinforces the differences between statistics and assessment.

Congressional support for these activities has taken separate paths, with a radical expansion of assessment activities since IES' creation. When NAEP first moved into IES, it was roughly a $50 million a year operation. But NAEP grew rapidly, given the testing requirements of No Child Left Behind (NCLB) and the Every Student Succeeds Act (ESSA). Under federal education laws, NAEP has grown to occupy a central role in the discussion of national and state policies and its budget has tripled.

International assessments have also expanded in budget and impact since 2002. When IES was created, PISA had been given only once (in 2000) and no one could have predicted its impact on worldwide and national education discussions. Other prominent OECD international assessments of adult competencies (PIAAC) and teachers (TALIS) did not even exist. Not surprisingly, given the increasingly global environment, the budget and the prominence of international assessments have increased dramatically. The new center would further strengthen these assessment activities.

Let me expand my argument in favor of creating a new center with a thought experiment.

Let's say in 2002 when IES was created, only two centers were constituted—a research center (NCER) and the statistics center (NCES). Let's further say that evaluation activities, which include both statistical and research work, were placed in NCES. Finally, let's say that evaluation activities continued to grow, so that eventually evaluation was funded at a level equal to NCES' statistical data collections.

Given these circumstances, would a decision to separate evaluation activities from NCES to create a new evaluation center be seen as somehow reducing the role of NCES? Or would creating NCEE be viewed as a reasonable response of the growth in budget and scope of evaluation activities? In this scenario, the creation of an evaluation center would recognize the differences between the function, roles, and expertise of evaluation and statistical data collections. It would also build on the fact that evaluation activities should be less constrained by the strictures of a statistical agency. In short, in this scenario, the evaluation mission that is part of IES' congressional mandate would be more fully realized in a separate center rather than as a part of NCES.

Moving from this hypothetical case to the case in favor of an assessment center, the same logic applies.

To be clear, my argument in favor of an assessment center is more than just bureaucratic juggling. Rather it is based on the reality of the size and importance of assessment activities—and that assessment activities and responsibilities are different than purely statistical activities rightfully housed in NCES. Paralleling the NCEE thought experiment, I believe it is time for assessment to be housed in a separate center to better pursue its unique responsibilities and mission. I believe that a new assessment center would have more freedom to engage in the translation of the data and findings of NAEP and international assessments that could help improve practice and translate data into actionable lessons.

The joint standards on testing and assessment issued in 2014 by AERA with the American Psychological Association (APA) and the National Council on Measurement in Education (NCME) support this idea. These standards define validity as a function of how well test scores are interpreted for their intended users and not just of the technical quality of the test itself. This "unitary validity" requires a richer interpretation of the results that can easily transcend the comfort level of a statistical agency. For example, it could mean presenting NAEP results in the context of the multiple international and state assessments in which the U.S. participates. Under this unitary validity standard, we would want the performance data of our students so carefully measured by NAEP to be put in context by comparing those scores against PISA and examining in detail the meaning of that comparison.

The assessment center would not have carte blanche to pontificate on policies based on assessments. Rather, freed from the extraordinary (and valid) constraints of a statistical agency, the new center would have more latitude to describe the meaning of assessment data in a way that is more accessible to the wide range of audiences that use NAEP and other assessments. In short, the new center would be able to use plain language to translate the lessons of NAEP and international assessments into terms that could be accessed by more people, while still holding to high IES standards.

Second, the assessment division has an important role in developing content and tests, which is not a good fit within a statistical agency. Developing frameworks, items, and scaling instruments builds on knowledge and skills that most statisticians do not have. (To get into the weeds, assessments need psychometricians; NCES needs statisticians.) Indeed, most assessment companies (such as ACT and ETS) do not think of their work as statistical in the way that federal statistical agencies such as NCES think about their data. These companies, which are our partners in creating and administering large-scale assessments, do not consider themselves only responsible for producing statistics. Rather, they are partners with IES' assessment teams in conceptualizing complicated education- and skills-related constructs and then designing assessment items to test them. This construction and interpretative role of assessment goes beyond "just the facts", which is NCES' core function.

Since the creation of IES, Congress has funded assessment and statistics separately and the separability of these two functions has grown even greater. I understand—and appreciate—the effort of ASA and AERA to safeguard NCES. But hiving off an activity that is tangential to NCES' core mission in order to better serve the nation seems to me a justifiable strategy to protect taxpayer investment in education assessment and research. It is time to create a new assessment center within IES.