Co-Principal Investigator: Goldhaber, Dan

Purpose: Weak academic preparation before college matriculation is often cited as a key reason for persistently low college graduation rates. To address this issue, about 40 states provide high school-to-college transition interventions at the state and local levels. This study examined the efficacy of one such effort—Kentucky's Targeted Interventions Program (TIP)—and explored the conditions under which TIP may help close the college readiness gap. The study focused on transition interventions that high school seniors received as a result of missing college readiness benchmarks on the ACT that they took in the Spring of their junior year. Each intervention cycle typically includes a diagnostic pretest, the development of instructional targets and their associated formative assessments, direct delivery of instruction, and a posttest. The study found that TIP achieved its policy goal of significantly reducing students' need for developmental education (DE) in college. However, gains made during the first year in college did not lead to higher likelihood of completing college or successful transfers to 4-year colleges among students who started in 2-year colleges. The study explored several possible reasons for the lack of longer-term benefit.

Project Activities: Researchers surveyed and interviewed school and district staff to collect information on TIP implementation and resource use. Researchers used this information to establish the types of support that targeted students received and conducted a cost study. Researchers used a difference-in-regression discontinuity (DiRD) method to evaluate the efficacy of TIP using longitudinally linked high school-college data for eight cohorts of 11th grade students who took the ACT between 2009 and 2016. Researchers also conducted exploratory regression studies to investigate possible reasons why short-term benefits of TIP did not lead to improved longer-term college outcomes.

Key Findings:

• TIP reduced DE enrollment in math by 8-10 percentage points in both 2- and 4-year colleges. TIP also increased the likelihood of passing college level math before the end of the first year in 4-year universities by 4 percentage points on average and by 9 percentage points among free/reduced-price lunch eligible students (Xu et al., 2022).
• TIP increased the likelihood that students took at least 15 credits during the first term, a key early momentum measure that predicts college completion. These early effects, however, did not translate into statistically significant impacts on the likelihood of transfers from a 2-year to a 4-year college, or the likelihood of earning enough credits to graduate from college (Xu et al., 2021. forthcoming in Community College Review).
• Evidence suggests several possible explanations for the lack of longer-term benefits. (a) Effects on early college outcomes may not be large enough to move the needle. (b) The increase in transition math course-taking came at the expense of regular HS math courses such as Algebra II, a critical precollegiate milestone for earning a college degree. And (c) The skills that high schools required for students to successful exit from TIP are consistent with the skills needed for placing out of DE math in college, but they are not sufficient for students to succeed in college (Xu et al., 2021. forthcoming in Community College Review).
• The estimated costs of TIP range between $400 and $900 per student. Per student costs vary depending on the number of students who needed TIP services, whether services were delivered during or after regular school time, and whether services were primarily computer-based, teacher-led or blended (Levin et al., 2020).

Because the target student population of TIP overlapped with students who were likely to concentrate in career technical education (CTE), the study also examined whether TIP crowded out CTE course taking and the relationship between CTE fields of concentration and key college outcomes. The study found that

• TIP did not significantly reduce CTE course taking, and that most students who specialized in a CTE area went on to study and earn credentials in something else in college (Xu & Backes, 2022.).

Structured Abstract

Setting:The study took place in the state of Kentucky. It included schools and students in both urban and rural settings across the entire state.

Sample:The primary analytic sample consisted of eight cohorts of 11th grade students in Kentucky public schools who took the ACT for the first time in the Spring of their junior year of high school between 2009 and 2016. The ACT has been mandatory for all high school juniors in Kentucky since Spring 2007. Each cohort consists of around 43,000 students.

Intervention:TIP uses 11th grade ACT test scores to identify students who may not be on track to be college ready when graduating high school. Students scoring below 19 in math and 18 in English are required to receive intervention services or remedial support, with the goal of getting them ready to access credit-bearing coursework without the need for DE or supplemental courses in college. TIP was implemented for high school math in 2010–11 and for high school English in 2011–12.

Each intervention cycle starts with a diagnostic pretest to identify specific deficiency areas. Algebraic thinking, math reasoning, math computation, writing mechanics, and writing content were the areas in which students mostly frequently needed help during the study period. Most students received interventions in the form of transition courses, which could be either integrated into an existing course or a stand-alone course. Most transition courses make use of online curricula (such as ALEX, Dreambox, Edgenuity, and IXL) in combination with teacher developed materials. An average intervention session was about 55 minutes long and was delivered four times a week. Around half of TIP participants in our sample were judged by schools to have exited remediation successfully.

Research Design and Methods:The efficacy study team used a DiRD design to evaluate the impact of TIP on student outcomes. DiRD is based on the regression discontinuity (RD) framework in which, except for the treatment status, factors that could affect the outcomes are continuous through the policy cutoff (i.e., the ACT benchmarks). Under this assumption, any abrupt changes in outcomes can be attributed to the treatment. In the current context, the same ACT benchmarks used to assign students to TIP were also used to assign students to DE once they reach college. In other words, any discontinuities in outcomes could reflect the combined effect of TIP and college placement policies. Because college placement policies remained unchanged before and after the implementation of TIP, the study used pre-TIP data to estimate the college placement policy effect and differenced that out from the post-TIP estimate to isolate the effect of TIP.

The implementation and cost study used administrative data and primary data collected through interviews and surveys to understand intervention types, intensity, delivery modes and setting, staffing, materials, and outcomes. A resource cost model (RCM) was used to support data collection and analyze cost analysis results. In an RCM, every program resource was assigned to a specific resource type category. Researchers used the resource categories as a guide to ensure that all resources were considered. Resource quantities were then multiplied with corresponding unit prices to obtain the total costs.

Control Condition:The control group consisted of students who scored just above the college readiness benchmarks and therefore did not receive transition interventions.

Key Measures:The researchers focused on the likelihood that a high school junior enrolled in DE in college, enrolled in and passed introductory college-level courses during the first year, took at least 15 credits during the first term, and completed a minimum number of credits required to graduate college after three years for 2-year college students and after six years for 4-year college students. For high school juniors who started college in 2-year institutions, the researchers also examined the likelihood of successful transfer to 4-year colleges three years after initial enrollment.

Data Analytic Strategy:The project used a DiRD design to estimate the impact of TIP on student college outcomes. TIP effect was estimated by subject and for different subgroups (e.g., race/ethnicity, FRL eligibility), type of intervention, type of school setting, and number of benchmarks a student failed to meet. The researchers used the RCM for the cost analysis.

Products and Publications

Researchers produced evidence of the efficacy of, and a cost analysis for, the Kentucky TIP. Research findings were published as peer reviewed journal articles, working papers, and blog posts. Access to extant state administrative data was described in journal articles, and the primary data set collected for the implementation and cost study was shared via Inter-university Consortium for Political and Social Research. Details are listed below.

ERIC Citations:Find available citations in ERIC for this award here

Select Publications:

Xu, Z., Backes, B., & Goldhaber, D. (forthcoming). Transition Intervention in High School and Pathway through College. Community College Review.

Xu, Z., Backes, B., Oliveira, A., & Goldhaber, D. (2022). Ready for College? Examining the Effectiveness of Targeted Interventions in High School. Educational Evaluation and Policy Analysis, 44(2), 183–209.

Xu, Z., & Backes, B. (2022). Linkage Between Fields of Concentration in High School Career-Technical Education (Working Paper No. 269-0722). National Center for Analysis of Longitudinal Data in Education Research (CALDER). Arlington, VA.

Xu, Z. & Backes, B. (2022, September 12). The case for specialized career and technical education. Fordham Institute.

Xu, Z., Backes, B., & Goldhaber, D. (2021, August 31). High school transition programs have mixed success in improving college readiness. Brookings Institutions.

Xu, Z. Kentucky Targeted Intervention Program Study. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2022-09-22. https://doi.org/10.3886/E180501V1.