Project Activities
The study will fulfill two aims. The first aim is to develop fast and accurate Gaussian variational algorithms for polytomous scoring and complex multidimensional structure. This development will greatly expand the capacity of the project team's existing VEMIRT (Variational Expectation-Maximation for multidimensional IRT) package to accommodate growing polytomous items and enable analytic dimension reduction. The second aim is to propose innovative factor-augmented regularized latent regression (FARLR) for building robust population models and generating plausible values (PVs), to facilitate a whole host of secondary analyses.
Structured Abstract
Research design and methods
The project team will conduct two comprehensive simulation studies to evaluate the proposed methods in comparison to the current status-quo operational methods. The first study consists of two parts. The first part focuses on evaluating the performance of pGVEM for MGPCM and MGRM respectively, in comparison to the operational standard Metropolis-Hastings Robbins-Monro (MH-RM) algorithm implemented in flexMIRT. The second part focuses on mixed-format items and longitudinal designs to showcase the efficiency brought by analytic dimension reduction. The second study aims to evaluate FARLR methods, and it is again composed of two separate sub-studies to imitate real scenarios slightly differently. In the first sub-study, the true latent trait values will be generated from a LR model, whereas in the Methods Training Using Single-Case Designs second sub-study, the PVs from real data will be used as true latent trait values. In addition, two empirical data sets will be used to further demonstrate the performance of new methods. They are: NAEP 2017 grade 4 math assessment data and PISA 2018 reading assessment data. These two data sets differ in terms of assessment design (linear test vs. multistage tests), hence demonstrating the performance of proposed methods in a full range of application scenarios.
User Testing: The App and R package will be tested in the graduate-level course first and then more broadly at professional venues. An exit survey will be given via Qualtrics at those venues, and attendees will be asked both rating scale items and open-ended questions. Feedback will be used to further improve the usability of the App and the package.
Use in Applied Education Research: This research will equip educational researchers and practitioners with an important tool to keep up with the ever-changing landscape of LSA. The proposed methodologies are timely, and they can be applied in different applied scenarios.
First, multidimensionality in LSA either may arise from assessment frameworks or specific research questions. For instance, modeling a complex theoretical construct such as overall math proficiency that comprises several correlated subdomains (e.g., number properties and operations, algebra, etc.), or modeling reading proficiency and math competence together to explore how the different aspects literacy affects acquisition of math skills, or modeling longitudinal trajectories of a construct.
Second, FARLR methods can very well handle high dependency among covariates both cross-sectionally and serially, where the latter case is more nuanced when survey questions are administered repeatedly, such as the school monthly COVID survey. When the same survey questions are administered repeatedly, they tend to have high serial correlation, and FARLR is well designed to tackle such a challenge. In addition, FARLR with a new debiasing approach can also adjust for the effects induced by the unmeasured confounders, thereby affording valid inference on the relationships between key predictors (e.g., mode of instruction) and education outcomes.
Third, generating robust PVs is critical for secondary uses of LSA. Lastly, the tool can benefit researchers and practitioners who manage data that is relatively "local". For instance, IES has invested millions of dollars to support building the infrastructure of State Longitudinal Data Systems (SLDS). The VEMIRTP software can be used to conduct robust analysis and generate PVs from SLDS. The PVs can be used to examine long-term learner outcomes and pathways, especially for those who were in a "treatment" group. This way, their long-term outcomes can be tracked to provide evidence for supporting state education policy making.
People and institutions involved
IES program contact(s)
Products and publications
The new sets of methods will be added to the VEMIRT software currently available both as a R Shiny app and as a R package, namely "VEMIRTP". Here, the subscript "P" has three meanings: a plus version, its capacity to handle polytomous items, as well as embedded functions to construct population models to produce PVs. The new methods will greatly expand VEMIRT not only because polytomous scored items and tasks are increasingly prevalent in modern assessments, but also the robust latent regression feature offers flexibility and stability in constructing population models and produce PVs to facilitate a whole host of secondary analyses. The project team will implement four strategies intended to reach broad audiences, including psychometricians, national and state assessment practitioners, math education and disability researchers, through multiple mechanisms. First, the project team will share findings through national and international conference presentations and publications. The team will produce a user manual that documents detailed user cases and recommended guidelines for the VEMIRTP package. Second, the team will disseminate results through regional workshops and conferences such as the Washington Educational Research Association (WERA). Third, the team will engage relevant community partners such as the NAEP Survey Assessment Innovations Lab (SAIL) program, NAEP research department at NCES, by visiting them and giving seminars and talks. Lastly, the team will collaborate with (1) the International Association for the Evaluation of Educational Achievement (IEA) to promote the uses of the project's methods and software by giving workshops at their academy; (2) Dr. Paul Bailey, the lead developer of the EdSurvey R package. Given the wide user base of the EdSurvey package, the aim is to combine efforts by integrating the new FARLR method into the EdSurvey package.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigator: Xu, Gongjun
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
