Co-Principal Investigators: Hill, Andrew; Stoddard, Christiana

Partner Institutions: Montana Office of Public Instruction (OPI) and Montana State University

Purpose: Montana's OPI and Montana State University will examine Montana's Early Warning System (EWS) to predict student dropout risk and risk factors for students in grades 3–12.

Project Activities: The project team will carry out three research activities:

• Examine the degree to which EWS predicts students will graduate but who drop out instead and identify possible adjustments to reduce this type of incorrect prediction,
• Examine the degree of implementation of EWS by schools and districts, and
• Compare school-level outcomes for schools using EWS versus those not using EWS.

Products: The project team will produce a technical report for use within OPI; a practitioner guide for SEA, LEA, and school staff as well as for state legislators; and presentations and articles for state education associations and committees, state legislators, researchers, and practitioners in other states interested in EWS, and Montana schools (participating and not participating in EWS).  The project team will provide a specific set of presentations and materials for tribal K–12 schools, the tribal colleges that provide technical assistance to them, and the tribal councils.

Structured Abstract

Setting: This project takes place in the state of Montana.

Sample: Grades 3–12 in all Montana public schools for the school years 2008–09 through 2019–20.

Key Issue, Program, or Policy: The Montana EWS program provides schools with reports on student dropout risk using data maintained in Montana's Statewide Longitudinal Data System (SLDS) and data voluntarily provided by participating schools. These data are combined into a logistic regression model housed within the SLDS's student data warehouse to predict student dropout risk and risk factors for students in grades 3–12. The OPI also provides training to school staff on interpreting the school and student reports. Schools are responsible for developing and implementing interventions to reduce the chances of dropout for the students identified at risk. After a pilot, the EWS was made available statewide in 2015–2016 and by the 2018–2019 school year, approximately 41 percent of Montana's total eligible students (n = 34,117) attended a school participating in the EWS program.

Research Design and Methods: The research team will use different methodologies depending on the project activity. They will use predictive modeling to examine and reduce the number of students predicted to graduate who actually drop out. The team will use a combination of secondary data analysis, a survey of all participating schools, and interviews with 15 districts to examine implementation of EWS and how schools make use of the findings. In addition, they will use a quasi-experimental design (difference-in-differences) to compare outcomes for schools using EWS with those not participating in it overall and by subgroups based on student characteristics, school characteristics, and implementation level.

Control Condition:  For schools using EWS, the project team will first compare student outcomes in the same schools before and after EWS was adopted.  Second, the change in outcomes from before and after adoption in those schools will be compared to the same change in outcomes in other schools that did not adopt EWS.

Key Measures: Outcomes at the student and school levels include school attendance rates, test scores, progression or repeating a grade, high school graduation or dropout, college attendance and persistence into second year of college. Implementation outcomes include how often schools update their EWS data and evidence that schools are using the EWS findings to implement interventions to reduce drop out.

Data Analytic Strategy: The project team will use linear probability models for binary outcomes and standard linear regression models for continuous outcomes.

State Decision Making: The findings will be useful as Montana OPI considers:

• Use of vendors models
• Guidance to schools on the effective use of the model
• Determining if elementary school data is worth including and if separate grade calculations are necessary
• Targeting of resources based on findings of subgroup heterogeneity
• Continued OPI Staffing of the program
• School decisions to participate in the EWS