Project Activities
Vertical scales aim to capture growth across time by linking test forms appropriate for students of different ages, placing scores onto a common scale. This scale design is integral to several applications in education, ranging from the measurement of individual students' growth across years using interim assessments, to the estimation of how much a typical student grows academically in a year, to screening students at different ages for a common developmental feature such as early number sense. Vertical scales are typically created by administering overlapping sets of items to students in adjacent grades, then estimating a linkage from differences in how students in the upper and lower grade perform on those items. A key assumption of this endeavor is that the common items reflect the target construct equivalently for students in both grades, despite the possible differential influence of learning opportunities in the upper grade on the ways that students approach those items. Existing methods for validating this assumption are ad hoc and rarely applied, and this project will use the relatively new MNLFA framework to approach this "common construct" assumption in a way that is both more statistically interpretable than existing methods and more useful for pragmatic assessment of the impact of violations of the assumption on the single most important question a vertical scale exists to answer: how much students appear to grow.
Research plan
The research plan is centered around developing a detailed methodology for the application of MNLFA to vertical scaling, complete with a real data example using item response data from the 1999 Early Childhood Longitudinal Study, Kindergarten (ECLS-K 1999). This promises to be a contribution on its own, as well as the beginning of a larger research program for the (Principal Investigator) PI consisting of simulation studies evaluating the performance of MNLFA-based methods in vertical scaling and beyond.
The first phase of the project will be mainly conceptual; the PI's aim during this phase will be to develop a clear, step-by-step methodology for how one can apply the MNLFA framework to the task of linking a vertical scale, validating the common construct assumption underlying the linking process, and pragmatically assessing the impact of any violations for estimates of student growth.
The second phase of the project will involve analysis of a real dataset, the ECLS-K 1999 item response dataset. The dataset will be analyzed using the methodology developed during the first phase, and the results will be compared to the results of linking the vertical scale using traditional methods. The main question that this phase will answer is: how different does student growth from one testing occasion to the next look, depending on whether one uses MNLFA or a traditional item response theory model? In consultation with Dr. Hancock, a simulation study to demonstrate the application of MNLFA to vertical scaling will also be developed.
The third phase of the project will focus on dissemination in the form of a conference paper/presentation and subsequent manuscript for submission to a prominent journal in the field of education measurement. During the writing and dissemination phase of the project, the PI will also begin formulating future research plans, based on a larger research agenda.
Career plan
The career plan involves both methodological mentorship, primarily in structural equation modeling, and mentorship in identifying and seeking external funding. The methodological mentor, Dr. Gregory Hancock, is an internationally recognized expert in structural equation modeling, measurement, and quantitative methods in the social sciences. The institutional mentor, Dr. Henry May, has served as PI on some of the largest grants awarded by IES, and is also an expert in the use of test scores in research, a key motivator for this project. The mentorship of the PI by these two experts will support the PI's development of a broader research agenda.
The career development plan for this project is centered around mentorship in three areas: structural equation modeling methods, identification and pursuit of external funding, and communication of findings. To accomplish these goals, the PI will participate in regular mentorship meetings, coursework and training workshops, and a focused program of research. Dr. Hancock will provide the PI with guidance related to the PI's development of a practical framework for applying MNLFA to vertical scaling, on the implementation of the methodological approach, and to understand the landscape of funding opportunities that might be appropriate for the PI's planned research agenda. Dr. May will work with the PI to understand the landscape of funding opportunities that might be appropriate for the PI's planned research agenda and provide feedback. Drs. Hancock and May will both provide the PI with ongoing feedback on the manuscript(s) that this project will produce including traditional academic journal articles and non-traditional dissemination products, such as practitioner guides or even podcasts.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: Products for this project include a methodology for the application of MNLFA to vertical scaling. The PI will present findings at conferences and produce publications as well as develop other dissemination products (e.g., podcasts and code) that will reach the professional measurement community, students, and practitioners.
ERIC Citations: Find available citations in ERIC for this award here.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
