IES Grant
Title: | Continuous Improvement Research to Support the Implementation of a Statewide Reform to Postsecondary Developmental Education—A RAND-THECB Research Partnership | ||
Center: | NCER | Year: | 2015 |
Principal Investigator: | Daugherty, Lindsay | Awardee: | RAND Corporation |
Program: | Continuous Improvement Research in Education [Program Details] | ||
Award Period: | 4 years (7/1/2015-6/30/2019) | Award Amount: | $2,499,276 |
Type: | Other Goal | Award Number: | R305H150069 |
Description: | Name of Partners: RAND, Texas Higher Education Coordinating Board (THECB), American Institutes for Research, Austin Community College, El Paso Community College, Houston Community College, San Jacinto College, South Texas College, Tyler Junior College Purpose: This partnership engaged in continuous improvement strategies and research to help Texas colleges develop and implement effective approaches to complying with the state's developmental education reform policy. Historically, institutions have committed substantial resources to developmental education to prepare students for college-level coursework, yet many students never completed their developmental course sequence or moved on to credit-bearing coursework. Low rates of success in developmental education translate into lost returns to investments by students and institutions. To reform developmental education Texas established new policies through the Texas Success Initiative and in response to Texas House Bill 2223. This partnership provided continuous improvement support and research related to three approaches within the broader reform initiative:
Project Activities: Two working groups of three colleges each were established to engage in continuous improvement. In the first three years of the project the team carried out three annual Plan-Do-Study-Act cycles with each of these working group colleges, testing improvements to holistic advising and support for ABE-level students. To inform the continuous improvement work, the team compared outcomes for the study institutions (and progress over time) to similar students at other institutions across the state. To build additional evidence on context and statewide implementation, the project team assessed the implementation of reforms through the state's annual survey of institutions and interviews with college administrators. The team also conducted quasi-experimental analysis on the developmental education reforms. In the final two years of the project the focus of the continuous improvement work expanded to institutions across the state and refocused on corequisites and disseminating findings to national audiences. Key Outcomes:
Structured Abstract Setting: Continuous improvement activities were conducted with six Texas community colleges in years 1–3 (~180,000 enrollees) and staff from more than 40 two-year and four-year colleges in years 4–5. Statewide quasi-experimental and implementation research focused on all community colleges in Texas. Sample: To assess student progress and outcomes for the continuous improvement colleges, the study team examined data on students enrolled between 2014 and 2018, comparing students in the six study colleges to similar students in other community colleges in Texas. These analyses examined outcomes for all developmental education students and outcomes for the students most likely to be impacted by the policies, students near college readiness and students testing at ABE-levels. Statewide analysis of survey data on implementation included all community colleges in the state. Quasi-experimental analysis on holistic advising factors focused on all students enrolled in Texas community colleges between 2013 and 2015, though findings from regression discontinuity are most applicable for students near the college-ready cut score. Quasi-experimental analysis assessing the outcomes associated with integrated reading and writing focused on statewide data from 2011 through 2016, with a focus on students enrolled in developmental education in community colleges. Approach: Texas developed new test scores to identify "ABE-level" students (those testing below 9th grade equivalence) and recommended that institutions consider requiring additional supports for these students, including academic support appended to the developmental education course, and referrals into adult basic education or technical programs. Although study colleges primarily continued to place ABE students into developmental education, they developed additional (typically optional) targeted supports through the project, including bridge courses, additional lab time for developmental education work, and versions of developmental education courses that integrated some additional language support. The state's holistic advising policy called on institutions to use other measures (e.g., high school grades, non-cognitive factors, transportation issues) to place students into developmental education, rather than the traditional approach of focusing solely on test scores. The study institutions developed new processes and forms for collecting the student information and integrating it into placement. Corequisites, the third focus area, places students directly into college courses with aligned developmental education support rather than requiring students to complete developmental education courses prior to entering college courses. The study provided colleges with technical assistance on developing continuous improvement plans and tracking data to inform the implementation of corequisites through a series of webinars, in-person convening sessions and workshops, and coaching sessions. The Texas colleges receiving this technical assistance implemented a wide range of corequisite models, and staff at these colleges used the tools provided to monitor and shape improvements to corequisite instruction, advising, and support. Research Design and Methods: The partnership used annual Plan-Do-Study-Act cycles with working group institutions to test approaches to implementing state-recommended reforms, and these cycles included site visits each semester and qualitative data collection and analysis. The team conducted qualitative analysis of findings from six working group institutions and descriptive analysis of statewide institutional survey data on developmental education practices to assess implementation in response to the state's reforms. The research team used a difference-in-difference approach to examine the impacts of differential access to reading and writing coursework across institutions. To assess the heterogeneity in outcomes across factors reported as being used for holistic advising in Texas, the research team used a regression discontinuity approach. Control Condition: When tracking progress for continuous improvement institutions, researchers compared outcomes for study colleges to all other Texas community colleges not participating in the study; these colleges were also likely engaging in reforms but not participating in continuous improvement. In the difference-in-difference analysis of the integration of the developmental reading and writing curriculum, researchers compared outcomes students with access to more integrated sequences to students with less integrated sequences. In the regression discontinuity analysis assessing heterogeneity in outcomes across factors used for holistic advising, the control condition is students who test below the cut score and are likely to be assigned to developmental education courses. Key Measures: Measures for categorizing holistic advising approaches across institutions included the number and types of holistic factors used in advising; assessment cut scores used to identify students for holistic advising; and subjective versus objective advising processes. Measures for categorizing approaches to ABE students across institutions included number and types of pathways offered to ABE-level students; cut scores above which institutions accelerate ABE-level students into academic pathways; and number and types of non-course-based support options. Short-term outcomes for the continuous improvement research and quasi-experimental studies included grades in assigned courses, credits accumulated, semester-to-semester persistence, and time to completion of first college-level course in math or English. Longer-term outcomes included persistence into the third semester, total credits accumulated during the study period, and transfer to a four-year institution or completion of a degree within three years. Data Analytic Strategy: During annual improvement cycles, the research team conducted analysis of qualitative and quantitative data collected at each institution and summarized key themes that emerged and were relevant for improvement and the implementation studies. For both working groups, researchers tracked key metrics and benchmarked each working group institution among the working group as a whole and against statewide averages for community colleges. To compare outcomes of the continuous improvement colleges to students in other Texas community colleges, we used differences-in-differences analysis of data from 2014–2018. To examine integrated reading and writing, a broad developmental education reform impacting ABE-level students, the team conducted differences-in-differences analysis on data from 2011–2016 to compare outcomes for students enrolled in colleges with more integrated sequences to students enrolled in colleges with less integrated sequences. To build evidence on which factors might be useful for holistic advising, the research team used regression discontinuity analysis with data from 2013–2015 to assess heterogeneity in outcomes for different groups of students when accelerated into college coursework. The quasi-experimental models included a rich array of student-level control variables that were available in state administrative databases. Related IES Projects: Designing a RCT Experiment to Test the Impact of Innovative Interventions and Policies for Postsecondary Developmental Education: A RANDTX Higher Education Coordinating Board Research Partnership (R305H130026); An Experimental Evaluation of Accelerated Pathways through Developmental Education—A RAND-THECB Partnership (R305H170085) Products The products of this research included a policy brief on Texas approaches to serving students who test at ABE levels on the state exam, a quasi-experimental journal article on holistic factors, a conference proceedings working paper, a quasi-experimental working paper on integrated reading and writing, a toolkit on corequisite continuous improvement, and a wide range of technical assistance resources, sessions, and workshops for Texas practitioners and policymakers. ERIC Citations: Find available citations in ERIC for this award here. Selected Publications Carew, D.G., Daugherty, L., Karam, R., Miller, T., Mendoza-Graf, A. (2018). Practitioner Perspectives on Implementing Developmental Education Reforms: A Convening of Six Community Colleges in Texas.Santa Monica: CA, RAND. Daugherty, L., Karam, R., Basco, D., and Kaufman, J.H. (2019). Tools for Improving Corequisite Models: A Guide for College Practitioners.Santa Monica: CA, RAND. Daugherty, L., Miller, T., Carew, D.G. (2019). Supporting College Enrollees Who Test at the Lowest Levels of Readiness: Lessons from Texas Community Colleges. Santa Monica: CA, RAND. Daugherty, L., Gerber, R., Martorell, P., Miller, T., and Weisburst, E. (2021). Heterogeneity in the Effects of College Course Placement, Research in Higher Education,62(7), 1086–1111. Miller, T, Gerber, R. and Daugherty, L. (2016). New Approaches to Developmental Education Pathways Integrating Reading and Writing Remediation. AEFP Conference Working Paper. |
||
Back |