|Title:||Evaluating the Impact of the Choice of Test Score Scale on the Measurement of Individual Student Growth|
|Principal Investigator:||Ho, Andrew||Awardee:||University of Iowa|
|Program:||Statistical and Research Methodology in Education [Program Details]|
|Award Period:||2 years||Award Amount:||$273,844|
|Type:||Methodological Innovation||Award Number:||R305U070008|
Co-Principal Investigator: Stephen Dunbar
Purpose: State and federal educational policies are focusing increasingly on school accountability for individual student growth. As statistical models and policy approaches proliferate, little attention is paid to the serious dependency of growth statistics on the choice of the test score scale. A different score scale, defined by a nonlinear transformation of the original score scale, can theoretically alter growth statistics, reverse aggregate trends, and distort interpretations from so called value-added models. However, the practical consequences of scale choice have not been well described, leaving growth-based education policies with statistics subject to undocumented scale-dependency. The researchers will develop a framework for evaluating the impact of scale choice on large-scale, policy-relevant growth statistics. This conceptual and mathematical framework considers families of gentle transformations for vertically scaled, longitudinal data that allow for the quantification of the scale dependency of target statistics.
Project Activities: The researchers will develop a framework for evaluating the impact of scale choice on large-scale, policy-relevant growth statistics. Moreover, they will determine if this framework has "scale-neutral" properties and can thus be applied to vertically scaled data from any state or local testing program. Theoretical developments will be anchored by applications using a state-level, 5-year longitudinal dataset from the state of Iowa. Iowa is currently one of eight states to have approval of its growth model under the Growth Model Pilot Program. The researchers will investigate families of plausible transformations whose well-established properties will allow for the standardized characterization of the dependency of growth statistics on the choice of vertical scale. Transformations from these families will emphasize different regions of the score scale in a systematic and predictable fashion while keeping the degree of emphasis modest. Under this framework, any dataset may be rescaled according to these plausible transformations, and descriptive statistics and growth model parameter estimates may be recalculated according to these transformed data to establish the limits of pliability.
Products: The outcomes of this research will include the development of the theoretical framework, implementation of the framework on Iowa testing data, and, where appropriate data are available, analysis of the scale dependency of growth statistics—including gains, gaps, and "value added"—for the states selected to implement growth models under the Growth Model Pilot Program. Reports of this research will also be prepared and published.
Purpose: The purpose of this project is to develop a systematic, scale-neutral, broadly applicable framework for understanding and describing scale dependency.
Research Design and Methods: A primary research task in this project is the development of families of reference distributions representing plausible transformations. Three approaches to developing reference families will be considered. The established score scale will be divided into regions, and different relative linear weights will be given to these regions. Three broad categories of advanced issues will be considered to fully explicate the framework: multiple time points, discrete data structures, and measurement error. The full treatment of the framework will investigate methods of addressing measurement error in observed score distributions. The researchers will evaluate the impact of scale choice on both widely reported statistics and decisions of consequence such as adequate yearly progress (AYP), using a longitudinal database of student scores into the determination of AYP for Iowa schools. Large-scale data from other state programs will be gathered, and an analysis of cross-state comparisons of pliability (stability across scale choice) will be made in order to illustrate the full potential of this framework: cross-test, cross program, and cross-state comparison of the pliability of widely reported, high-stakes growth statistics.
Data Analytic Strategy: The researchers will subject state student achievement data to families of possible scale transformations and produce statistical and evaluative reports about the scale dependency of all major achievement growth statistics. The results will inform researchers and policymakers about the stability of both school-level and state-level measures of student achievement growth for states participating in the No Child Left Behind Growth Model Pilot Program.
Journal article, monograph, or newsletter
Furgol, K.E., Ho, A.D., and Zimmerman, D.L. (2010). Estimating Trends From Censored Assessment Data Under No Child Left Behind. Educational and Psychological Measurement, 70(5): 760–776.
Ho, A.D., Lewis, D.M., and Farris, J.L.M. (2009). The Dependence of Growth-Model Results on Proficiency Cut Scores. Educational Measurement: Issues and Practice, 28(4): 15–26.
** This project was submitted to and funded as an Unsolicited application in FY 2007.