|Title:||Continuous Improvement Research to Support the Implementation of a Statewide Reform to Postsecondary Developmental Education—A RAND-THECB Research Partnership|
|Principal Investigator:||Miller, Trey||Awardee:||RAND Corporation|
|Program:||Continuous Improvement Research in Education [Program Details]|
|Award Period:||4 years (7/1/2015-6/30/2019)||Award Amount:||$2,499,276|
|Goal:||Other Goal||Award Number:||R305H150069|
Name of Partners: RAND, Texas Higher Education Coordinating Board (THECB)
Purpose: This partnership will use a continuous improvement process to help six Texas community colleges develop and implement effective responses to the Texas Success Initiative (TSI), which establishes new policies addressing the placement of and supports for incoming college students deemed in need of developmental (or remedial) education. Although institutions commit resources to developmental education to prepare students for college-level coursework, many students never complete their developmental course sequence or move onto credit-bearing coursework. Low rates of success in developmental education translate into lost returns to investments by students and institutions. The partnership will work with community colleges to improve their approaches to two TSI reforms:
The ultimate goal of the partnership is to increase the success rates of Texas community college students in developmental education and subsequent coursework.
Project Activities: The partnership will coordinate the activities of THECB staff, institution-level improvement teams, and researchers at RAND. The approaches taken by institutions at the beginning of the project will be the approaches that they have already developed in response to state policy. Over the course of three school years, the partnership will implement and test approaches to better meet the needs of developmental education students who fall a short distance below the cutoff score for entering credit-bearing courses, and those who score at the ABE level.
Products: The products of this research include revised approaches to holistic advising and assisting ABE students in developmental education, policy briefs to assist other Texas colleges in implementing similar approaches, and advice for THECB regarding policies that can assist colleges with these approaches. Researchers will also produce peer-reviewed publications.
Setting: The research will take place at six community colleges, comprising 35 campuses serving a total of 150,000 students, located across Texas.
Sample: The treatment group for the holistic advising study will include approximately 56,000 first-time undergraduate students located at three community colleges where the improvement process is taking place. The treatment group for the adult basic education comparison study will include about 89,000 first-time undergraduate students located at a different set of three community colleges. These community colleges serve a diverse student population, of whom approximately 56 percent are Hispanic, 14 percent are African American, and 43 percent are receiving Pell grants. The ongoing comparison study will include all comparable first-time undergraduate students at Texas public community colleges other than the improvement colleges, from the 2011–12 to 2017–18 academic years.
Approach: The partnership will begin with the current approaches to holistic advising and ABE students taken by each of the institutions. Currently, all institutions require advisors to meet face-to-face with incoming students, but individual institutions differ in the extent to which they rely upon advisor discretion versus quantitative criteria to place students into developmental education levels. For ABE students, all institutions are moving to align ABE curricula with developmental curricula and offer multiple tracks for transitioning from ABE to developmental coursework. One institution has gone a step further, and offers students at the ABE level direct access to developmental coursework, with additional supports.
Research Design and Methods: The partnership will use a plan-do-check-act improvement strategy with cycles operating at the state and institution levels. Researchers will provide the state- and institution-level teams with descriptive statistics drawn from student survey data and background data, to highlight connections between student characteristics and success in developmental education courses and other pathways for students with developmental needs. Where possible, researchers will also provide results from more rigorous quantitative analytical models such as differences-in-differences and regression discontinuity. Teams at both levels will then use this information to refine policy guidance and practices to improve implementation of the approaches. The ongoing comparison study will use a differences-in-differences design, and compare student outcomes at improvement colleges to similar students at matched comparison colleges that are implementing different approaches to the reforms.
Control Condition: The comparison condition will be institutions that are taking different approaches to holistic advising and ABE relative to those taken by institutions in the partnership. Researchers will draw upon a statewide institution-level survey of developmental education practice to identify approaches that institutions are taking to holistic advising and ABE and systematically categorize approaches according to different factors (such as those described under Key Measures). They will also identify sets of institutions that are implementing reforms more slowly than others, to flag "business as usual" cases for the comparison study. For each identified reform approach, researchers will use a propensity score matching method to choose a set of comparison colleges from the set of all public two-year colleges in Texas that are both implementing the reform approach and are similar to the improvement colleges in terms of student composition and pre-treatment trends in outcomes. The research team will explore the sensitivity of results to different methods for choosing control colleges based upon pre-program characteristics and outcome trends. The analysis will be based upon cohorts of first-time undergraduate students entering between the 2011–12 and 2017–18 academic years.
Key Measures: Measures for categorizing holistic advising approaches across institutions will include number and types of holistic factors used in advising; assessment cut scores used to identify students for holistic advising; and subjective versus objective advising processes. Measures for categorizing approaches to ABE students across institutions will include number and types of pathways offered to ABE-level students; cut scores above which institutions accelerate ABE-level students into academic pathways; and number and types of non-course-based support options. Short-term outcomes for both studies will include grades in assigned courses, credits accumulated, semester-to-semester persistence, and time to completion of first college-level course in math or English. Longer-term outcomes will include persistence into the third semester, total credits accumulated during the study period, and transfer to a four-year institution or completion of a degree within three years.
Data Analytic Strategy: During annual improvement cycles, the research team will conduct content analysis of qualitative data collected at each institution and summarize key themes that emerge and are relevant for improvement. The team will also conduct descriptive analyses of quantitative data and calculate means of key outputs and outcomes by relevant variables. For holistic advising institutions, the team will provide summary statistics for short-term outcomes and other student success measures by holistic factors. Where possible, researchers will also provide results from more rigorous quantitative models such as differences-in-differences and regression discontinuity. For the ABE-improving institutions, the team will calculate persistence rates and other measures of short-term success by initial pathway (e.g. traditional developmental education vs. contextualized ABE courses). The ongoing comparison study will employ a propensity score matching procedure to identify control institutions that are implementing different approaches to the reforms, but which closely match the treatment institutions on key variables including student characteristics and pre-program trends in outcome variables. The models will also include a rich array of student-level control variables that are available in state administrative databases.
Related IES Projects: Designing a RCT Experiment to Test the Impact of Innovative Interventions and Policies for Postsecondary Developmental Education: A RAND—TX Higher Education Coordinating Board Research Partnership (R305H130026) and An Experimental Evaluation of Accelerated Pathways through Developmental Education—A RAND-THECB Partnership (R305H150094)