|Title:||Project MIDAS: Development of a Multi-Informant Decisional Assessment System|
|Principal Investigator:||von der Embse, Nathaniel||Awardee:||University of South Florida|
|Program:||Social and Behavioral Context for Academic Learning [Program Details]|
|Award Period:||4 years (08/01/2021 – 07/31/2025)||Award Amount:||$2,000,000|
Co-Principal Investigators: Kilgus, Stephen; Eklund, Katie; Suldo, Shannon; Bonifay, Wes
Purpose: The purpose of this project is to develop and validate a multi-informant decisional assessment system (MIDAS) to integrate and use multiple sources of data for accurate and efficient identification of social-emotional and behavioral (SEB) concerns in middle school students. Universal screening for SEB concerns is promising to facilitate early identification and implementation of evidence-based interventions but schools are not able to integrate such information from different informants who contribute unique information. In addition, researchers have yet to determine how to best integrate screening data with other academic and behavioral data sourcessuch as suspensions, office discipline referrals, and academic achievement scores to facilitate effective decision making to support students.
Project Activities: The researchers will use multiple research methods including an iterative assessment design process to calibrate and refine the MIDAS model with an existing data set and cross validation of the MIDAS framework. They will also determine whether the MIDAS system is feasible for middle schools to use and how much it will cost.
Products: This project will yield a fully developed and validated MIDAS computer scoring system, which is intended for use as a multi-informant rating system. This project will also inform the development of resources to support MIDAS implementation including an implementation guide and checklist, prerequisites for MIDAS adoption, and key considerations related to decision-making such as how to interpret the data. A final resource includes an open Application Interface(API) that allow MIDAS to be integrated within existing schoolwide information systems (EduClimber, PowerSchool) to support secure entry and storage of MIDAS data and generation of data reports.
Setting: This project will take place at six middle schools across two sites, including one in the Southeast (Florida) and another in the Midwest (Wisconsin).
Sample: Primary research participants will include middle school students at-risk for emotional and behavioral problems, as verified by multiple universal screening tools, and middle school teachers, who will be responsible for completing various assessment tools.
Instruments: In this project, researchers will develop an online and automated multi-informant rating system, with related decisional processes. The assessments will include teacher and student versions of the Social, Academic, and Behavioral Risk Screener(SAEBRS).
Research Design and Methods: For this project, the first phase is to calibrate the MIDAS system using a large extant database of schools that have utilized the SAEBRS (N=6,000) and an item response theory-based scoring system across teacher and student SAEBRS forms. Researchers will calibrate the system with pre-test risk factors and existing data from school wide information systems. The second phase will cross-validate using new universal screenings from two teachers (homeroom, subject area), student-self report, extant student data (such as office discipline referrals), and demographic data (such as ethnicity and age). The research team will use a Bayesian approach to estimate individual student probabilities and evaluate the resulting MIDAS score with respect to test fairness/measurement invariance, usability, and accuracy. A third phase will test the feasibility of MIDAS and associated procedures within educational contexts. The team will give schools access to a MIDAS prototype and trained in its use to inform screening decisions. Via a mixed-methods approach, MIDAS end-users (administrators, teachers, and student support personnel such as school psychologists, counselors, and social workers) will then provide information regarding the feasibility of MIDAS implementation at the end of the school year. These data will inform how MIDAS will be used in real-world decision-making, with a focus on usability and sustainability. The research team will use findings from the final phase to inform the refinement of the MIDAS prototype, implementation guidance, and training procedures.
Control Condition: Due to the nature of the study, there is no control condition.
Key Measures: Measures include SAEBRS andvarious extant academic data such as test scores and curriculum-based measurements and behavioral data such as office discipline referrals.
Data Analytic Strategy: The research team will examine MIDAS development, refinement, and validation using modern psychometric analyses including a Bayesian data analytic framework, item response theory, and correlational analyses. Analyses will incorporate the multi-level nature of resulting data and will include hierarchical linear modeling.
Cost Analysis: The researchers will determine the cost of MIDAS using the ingredients method. Four categories of basic costs are of interest, including personnel, facilities, equipment and materials, and other costs.
Related IES Project: Development and Validation of Measures Supporting the Selection and Modification of Tier 2 Emotional and Behavioral Interventions (R305A180515).