Co-Principal Investigator: Eric Bettinger (Stanford University)

Related Network Teams: College Completion Network Lead (PI: Eric Bettinger, R305N170003); The Men of Color College Achievement (MoCCA) Project (PI: Lashawn Richburg-Hayes, R0305N160025); Affording Degree Completion: A Study of Completion Grants at Accessible Public Universities (PI: Sara Goldrick-Rab, R305N170020); A Scalable Growth Mindset Intervention to Raise Achievement and Persistence in Community College (PI: Gregory Walton, R305A150253); An Experimental Evaluation of Corequisite Developmental Education in Texas (PI: Trey Miller, R305H170085)

Description: A Research Network involves several teams of researchers who are working together to address a critical education problem or issue. The objective is to encourage information sharing, build new knowledge, and assist policymakers and practitioners to strengthen education policies and programs and improve student education outcomes. The College Completion Network is evaluating promising strategies for moving students at open- and broad-access postsecondary institutions beyond college enrollment and entry-level courses to degree completion. Each network research team is evaluating a specific intervention strategy.

Purpose: This research team will investigate whether text messages that provide personalized information to students attending open- and broad-access enrollment institutions will help them complete their college degrees. The team will complete both development and evaluation activities. The main objective during the development phase of the project will be to develop a text messaging campaign that is informed by the specific informational, behavioral, and psychological challenges that can pose obstacles to students succeeding in college. During the evaluation stage, the main objective will be to assess whether the text messaging campaign causes students to engage in effective behavioral responses to the obstacles they face, and subsequently complete their degrees.

Project Activities: During the development stage, researchers will work with a text messaging provider (Persistence Plus) to design a text messaging campaign aimed at supporting the behaviors that lead to degree completion. As a preliminary step, researchers will analyze administrative data to identify students at risk of leaving college prior to completing their degrees. Based on this analysis, researchers will meet with at-risk students in focus groups to better understand the challenges they face and how they interact with technology, and then pilot different types of text messages to observe how students respond to them. Following the focus groups, researchers will conduct a relatively small randomized controlled trial (RCT) to evaluate the efficacy of specific treatment components for leading to intermediate outcomes associated with degree completion. The development stage will result in a fully-formed text messaging campaign. During the evaluation stage, researchers will conduct a large, fully-powered RCT to evaluate the efficacy of the text messaging campaign for effecting higher rates of degree completion.

Products: The products from this project include a fully-formed text messaging intervention, two interim reports drawing on findings from the development stage, and a final report including the impact analysis and details of the intervention design and implementation.

Structured Abstract

Setting: This project will take place at a variety of open- and broad-access postsecondary institutions, including: a statewide community college system; an urban university system that grants both Associate's and Bachelor's degrees; a statewide higher education system including community colleges and 4-year institutions; and two large broad-access 4-year institutions.

Sample: The sample will include approximately 500 students at each of eight institutions during the development stage, for a total of 4,000 students, and approximately 1,500 students at each of 17 institutions during the efficacy stage, making a total of 25,500 students. At the time they are selected for participation in the study, all students in the sample will have earned at least half of the credits needed for a college degree.

Intervention: The text messaging campaign will send personalized text messages to students containing information, reminders to carry out specific tasks, and guidance about how to carry out complex tasks essential to earning their degrees. Text campaign content will provide a mix of information and guidance relevant to all college students, to all students at specific institutions, and to individual students. Messages for all students will encourage social-behavioral skill development including engagement with college faculty and support services, and perseverance with complex tasks. Personalized texts will include information relevant to the specific courses and tasks that individual students need to complete in order to finish their degrees, as well as planning and implementation prompts to help students execute steps on the way to degree completion. Students at some sites will also have access to an advisor, either through the Persistence Plus technology platform or through a campus-based advising service.

Research Design and Methods: In both the development and evaluation stages of the project, researchers will use randomization to assign students near college completion to receive text messages via a text messaging campaign (the treatment) or to not receive the text messages. During the development stage, researchers will analyze historical data to create a risk model for late college departure at each of the institutions, and then use these models to generate predicted probabilities of late departure for all students in the study. To improve the precision of treatment-control comparisons in impact analyses, researchers will randomly assign students to the treatment and control conditions from "blocks" of students with very similar risk profiles.

During the development stage, researchers will use the block randomization design to test the efficacy of different intervention components aimed at producing improvements in intermediate student outcomes (e.g. course completions and semester-to-semester persistence). The intervention components will vary with regard to the level of personalized information included in the text messages, the duration of the campaign, and whether or not the messages offer access to an advisor. During the evaluation stage, researchers will again use the block randomization design to test the efficacy of the developed text messaging campaign. Students assigned to the treatment will receive the campaign for four semesters beginning in January of 2018, and the research team will collect data on student responses to the campaign and academic outcomes for three years.

Control Condition: The control condition will include "business-as-usual" information and nudges (e.g. emails) provided to all students at each of the colleges. Students assigned to the control condition will not receive the text messaging campaign or any additional supports connected to it (e.g. online or face-to-face advising).

Key Measures: During the development and evaluation stages, researchers will collect the same set of intermediate outcomes including completion of important procedural tasks such as course registration and FAFSA renewal, use of campus-based services, course completions, and transfer to another college. Researchers will measure whether students report increased motivation, commitment to degree completion, identification with college, and other non-cognitive factors associated with degree completion. Researchers will collect measures directly from students through the text messaging platform, and from administrative records provided by the colleges. During the evaluation stage, researchers will follow students for three years, by obtaining degree completion data from college administrative records as well as the National Student Clearinghouse. The observation period will be sufficient to observe degree completion in 150 percent of time (3 years) for community college students, and completion in 5 years for 4-year students.

Data Analytic Strategy: For the risk analysis that will be used to assign students to blocks, researchers will use probit regression to compute the probability of dropping out after having acquired 30 or 60 credits in two- or four-year colleges, respectively. Researchers will use probit regression for the impact analysis, with covariates necessary for sub-group analysis, and will use a multi-level model to analyze whether the treatment impact differs for students with different risks of late-college departure.

Products and Publications

Working paper

Mabel, Z, Castleman, B., and Bettinger, E. (2017) Finishing the Last Lap: Experimental Evidence on Strategies to Increase College Completion for Students at Risk of Late Departure. Available at SSRN: https://ssrn.com/abstract=3054765.