Co-Principal Investigators: Dossett, Dena; Shand, Robert; Pan, Yilin
Partner Agency: Jefferson County Public Schools, KY (JCPS)

Purpose: The overall goal of this partnership project was to investigate whether and how cycle-based budgeting and the concept of academic return on investment can be used to improve decision-making about education programs and strategies and ultimately lead to stronger student education outcomes. In particular, the research partnership sought to identify methods that districts could use to analyze the relative costs of programs to aid in decision-making. During the course of the project, the partners explored different methods and different programs to identify optimal approaches.

Partnership Activities:
Interviews with Jefferson County Public Schools (JCPS) decisionmakers at beginning and end of project: In fall 2018, the research partners conducted a set of interviews with 11 JCPS decisionmakers about the district's budget decision-making practices and reviewed information provided to these decisionmakers during the budget process. In May 2021, the researchers conducted end-of-project interviews with the objective of assessing the extent to which the process of making decisions changed over the course of the project.

Training JCPS school administrators: In 2018–19, the project team provided training to 68 school-based personnel across 43 JCPS schools to help school level decisionmakers improve the way in which Title I funds are allocated and their impact evaluated. In 2019–20, the state identified 23 JCPS schools as Comprehensive Support and Improvement Schools, and these schools received targeted training to provide support on the development of turnaround plans. Between March 2020 and June 2021, the JCPS partners conducted one-on-one sessions with investment item owners (e.g., program directors) to review performance metrics and recommendations for action. They also created online training modules for cost center heads on developing logic models.

Review of cycle-based budgeting process and use of evidence to inform decisions about which items to invest in. The team reviewed 55 budget request proposals totaling $5.7mm submitted by 42 unique JCPS cost center heads to investigate the use of evidence in supporting budget requests.

Quantitative analyses to refine and validate AROI metrics:
Academic return on investment (AROI): The project team iterated several times on the district's original AROI formula and calculated AROI metrics for 2018–19 and 2020–21 end-of-cycle investment items and for 27 math and reading interventions.

Value-added analysis (VAA): The project team adapted methods used to estimate teacher value-added to evaluate multiple JCPS programs implemented simultaneously. The project team executed a VAA of 68 math and reading interventions and of 47 academic year 2018–19 end-of-cycle investment items. The project team also conducted an additional analysis to compare the results of AROI and VAA using an “ideal” data set.

Cost-effectiveness analysis (CEA). The project team used quasi-experimental studies and the ingredients method to estimate effects and costs of three programs implemented in JCPS schools: Reading Recovery, the school nurse program, and restorative practices.

Comparative analyses: The project team calculated AROI and VAA metrics for the three programs evaluated using CEA and the results were compared across the three methods as a means to evaluate the validity of AROI metrics.

Dissemination: The project team presented several times to internal JCPS audiences and externally at multiple conferences and wrote three briefs on the application of AROI to decision-making about educational programs, three blogs, and several peer-reviewed manuscripts targeting both researchers and educators.

Structured Abstract

Setting: Jefferson County Public Schools (JCPS) is a large, urban school district in Louisville, KY. With 6,800 teachers serving over 96,000 students in 165 school sites and an annual budget of $1.7 billion in academic year 2018–19. JCPS is the largest district in Kentucky and one of the largest in the nation.

Population: JCPS students are 38 percent White, 36 percent Black, and 12 percent other races; 14 percent are Hispanic/Latinx. Nearly two-thirds of JCPS schools were eligible for Title I funding in 2018–19, and roughly 67 percent of students were eligible for free or reduced-price lunch. Students live in urban (59 percent), suburban (40 percent), and rural (1 percent) areas.

Data Analytic Strategy: Data analytic methods included a variety of multiple regression analyses, such as Ordinary Least Squares (OLS), difference-in-difference (DiD), multilevel, and General Linear Modeling (GLM). They also included DiD and prior cohort (PC) AROI approaches, optimal multilevel matching, cost analysis using the ingredients method, document analysis, qualitative analysis of personnel interviews with theming and coding of transcripts, and an exploratory application of value-added statistical models (effect- or dummy-coded) to student outcome data.

Key Findings: Comparison of three methods for providing local evidence to inform school and district budget decisions: The project team evaluated Reading Recovery, Restorative Practices, and the school nurse program as implemented in JCPS schools using three different methods: CEA, program VAA, and AROI. The team found similar results across all three methods. The team evaluated the methods and findings against three criteria: rigor of the methodology, difficulty of execution, and usability of the results for decision-making. They concluded that, although AROI suffers from methodological limitations and should not be considered definitive evidence on program performance, its value lies in the fact that it can be calculated for any academic program on a timely basis to inform annual budget decisions. Program VAA is useful for evaluating many programs at once in order to identify interventions that perform particularly well or poorly, thereby meriting closer scrutiny. Combining CEA's more rigorous approach to estimating costs with AROI's more feasible methods for estimating program effects may represent a useful synthesis of methods for evaluating district programs to help inform decisions about whether to continue, discontinue, or scale up a program (Hollands et al., 2022).

Value-added analysis (VAA): The researchers found that, under specific conditions, program VAA can provide evidence to help district decision-makers identify outlier interventions and to inform decisions about scaling up or disinvesting in such interventions (Shand et al., 2022).

Academic Return on Investment: The researchers' comparison of AROI, VAA and AROI results for Reading Recovery, the school nurse program, and restorative practices showed that effectiveness results were quite similar in direction and magnitude across the three methods. This was positive news in terms of validating AROI, but the actual impact of each program on student outcomes was disappointing for most measures, suggesting that district funds were being expended on programs that were not improving student academic or other outcomes. The district concluded that AROI can serve as a flexible approach to generating evidence that is local, timely, and relevant to inform budget decisions in the post-adoption improvement phase of an investment. However, due to limitations of the method, results should be considered informative rather than definitive. (Leach & Yan, 2021)

Costs and effects of restorative practices for improving behavioral outcomes and school climate: After 1 year of implementation, the researchers found no statistically significant effects of Restorative Practices on referrals, suspensions, school belonging, or site safety either for the overall sample or by racial subgroup. They found negative effects on student-reported personal safety. An additional analysis one year later showed that Black students in schools implementing restorative practices for 2 years experienced a greater reduction in suspensions than Black students in schools implementing the program for only 1 year. The researchers' reference case cost analysis results using a societal perspective, national average prices, and a 3 percent discount rate were $57,450 per school and $139 per student for the first year of restorative practices implementation. These estimates included training costs from the prior year and were incremental to the costs of positive behavioral interventions and supports, which served as the business-as-usual condition. The researchers concluded that restorative practices are relatively low cost in comparison to other behavioral interventions but may need to be implemented for several years with greater fidelity in order to produce the desired improvements in behavior events and school climate (Hollands, et al., 2022).

Analysis of the JCPS school nurse program and its costs: The researchers found that the presence of a full-time licensed practical nurse did not significantly improve elementary school student attendance or absenteeism. The incremental costs of providing a full-time licensed practical nurse above and beyond the basic health services provided in all schools was $43,000 per school or $102 per student (Leach et al., 2022).

Products

ERIC Citations: Find available citations in ERIC for this award here.

Additional Online Resources and Information:

• Hollands, Fiona & Dossett, Dena. (2021, March 9). Use of evidence to inform school and district budget decisions. Research 4 Schools.
• Hollands, F., Yan, B., Leach, S., Shand, R., Dosset/transfer.asp?sec=true&location=hang F., Pan, Y., Wang, Y. & Head, L. (2020). LEA use of evidence in budget decisions. Poster session at the Annual Meeting of the SREE, Online.
• Leach, S. M., & Yan, B. (2021). ITS 2.0 user's guide. Getting more bang for your buck: Academic Return on Investment and the Investment Tracking System. Louisville, KY: Jefferson County Public Schools, Accountability, Research, & Systems Improvement.
• Leach, S. M., & Yan, B. (2021). Academic return-on-investment (AROI) and budget decision-making: A research brief. Louisville, KY: Jefferson County Public Schools.

• Stone, E. (2022). Racial equity and school nursing. Jefferson County Public Schools Envision Equity, 115, 3–5.

Select Publications:

Inside IES Research Blog

Leach, S., Shand, R. D., Yan, B., & Hollands, F., (2022, February 15). Unexpected value from conducting value-added analysis. Inside IES Research.

Leach, S., Hollands, F., Yan, B., & Shand, R. D. (2022, February 3). Unexpected benefits of conducting cost-effectiveness analysis. Inside IES Research.

Journal articles

Hollands, F. M., Leach, S. M., Shand, R., Head, L., Wang, Y., Dossett, D., Chang, F., Yan, B., Martin, M., Pan, Y., & Hensel, S. (2022).  Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions. Journal of School Psychology 92,188–208.

Hollands, F. M., Shand, R., Yan, B., Leach, S. M., Dossett, D., Chang, F., & Pan, Y. (2022). A comparison of three methods for providing local evidence to inform school and district budget decisions. Leadership and Policy in Schools.

Leach, S. M., Hollands, F. M., Stone, E., Shand, R., Head, L., Wang, Y., Yan, B., Dossett, D., Chang, F., Chang, Y., & Pan, Y. (2022). Costs and effects of school-based licensed practical nurses on elementary student attendance and chronic absenteeism. Prevention Science.

Shand, R., Leach, S., Hollands, F., Chang, F., Pan, Y., Yan, B., Dossett, D., Nayyer-Qureshi, S., Wang, Y., & Head, L. (2022). Program value-added: A feasible method for providing evidence on the effectiveness of multiple programs implemented simultaneously in schools. American Journal of Evaluation.