Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Continuous Improvement Research to Support the Implementation of a Statewide Reform to Postsecondary Developmental Education—A RAND-THECB Research Partnership
Center: NCER Year: 2015
Principal Investigator: Daugherty, Lindsay Awardee: RAND Corporation
Program: Continuous Improvement Research in Education      [Program Details]
Award Period: 4 years (7/1/2015-6/30/2019) Award Amount: $2,499,276
Type: Other Goal Award Number: R305H150069
Description:

Name of Partners: RAND, Texas Higher Education Coordinating Board (THECB), American Institutes for Research, Austin Community College, El Paso Community College, Houston Community College, San Jacinto College, South Texas College, Tyler Junior College

Purpose: This partnership engaged in continuous improvement strategies and research to help Texas colleges develop and implement effective approaches to complying with the state's developmental education reform policy. Historically, institutions have committed substantial resources to developmental education to prepare students for college-level coursework, yet many students never completed their developmental course sequence or moved on to credit-bearing coursework. Low rates of success in developmental education translate into lost returns to investments by students and institutions. To reform developmental education Texas established new policies through the Texas Success Initiative and in response to Texas House Bill 2223. This partnership provided continuous improvement support and research related to three approaches within the broader reform initiative:

  1. Support for students at the Adult Basic Education (ABE) level. The new state test identified students with scores at ABE-level (i.e., below 9th grade levels of readiness), and recommended that these students receive the approaches being scaled to all students (e.g., integration of reading and writing courses) as well as additional supports (e.g., additional required academic support, advising into alternative pathways). The partnership engaged in continuous improvement support to three institutions to implement new approaches. The team conducted research on the implementation of the state's recommendations and evaluated the integration of developmental reading and writing as a key reform relevant to students entering college at the ABE level.
  2. Holistic advising. New policy required institutions to consider other factors beyond standardized test scores when placing students into developmental education courses. The partnership engaged in continuous improvement support with three institutions to implement new approaches to student placement and conducted quasi-experimental research to build evidence on factors that were being used by some colleges for the purposes of placement.
  3. Corequisite remediation: The Texas legislature passed House Bill 2223 in 2017 calling on institutions to scale corequisite remediation, where students directly enter college-level coursework while concurrently receiving aligned academic support. In the final two years of the project the team provided support to statewide professional development efforts by developing resources and providing coaching to support the adoption of continuous improvement approaches to improve corequisites as they were scaled.

Project Activities: Two working groups of three colleges each were established to engage in continuous improvement. In the first three years of the project the team carried out three annual Plan-Do-Study-Act cycles with each of these working group colleges, testing improvements to holistic advising and support for ABE-level students. To inform the continuous improvement work, the team compared outcomes for the study institutions (and progress over time) to similar students at other institutions across the state. To build additional evidence on context and statewide implementation, the project team assessed the implementation of reforms through the state's annual survey of institutions and interviews with college administrators. The team also conducted quasi-experimental analysis on the developmental education reforms. In the final two years of the project the focus of the continuous improvement work expanded to institutions across the state and refocused on corequisites and disseminating findings to national audiences.

Key Outcomes:

  • Access to a more integrated sequence of reading and writing DE courses was associated with an increase in the likelihood of enrolling and passing college-level courses, as well as overall credits earned. However, access to a more integrated sequence was also associated with small negative impacts on persistence and degree completion. (Miller, Gerber, and Daugherty, 2016)
  • While more than two-thirds of colleges reported offering the state's recommended supports for students who tested at ABE levels, many offered the support as optional and continued to rely primarily on developmental education courses to support students. Barriers to implementation included small numbers of students and low participation rates, competing priorities (e.g., scaling corequisites), confusion about the policy, and a lack of evidence on effective strategies (Daugherty, Miller, and Carew, 2019).
  • Corequisite math course appear to have benefited students who declared an academic major, had bachelor's degree aspirations, tested below college readiness on multiple subjects, were designated as Limited English Proficiency (LEP). However, students enrolled part-time and students over the age of 21 were less likely to benefit from placement into college-level math (Daugherty, Gerber, Martorell, Miller, and Weisburst, 2021).

Structured Abstract

Setting: Continuous improvement activities were conducted with six Texas community colleges in years 1–3 (~180,000 enrollees) and staff from more than 40 two-year and four-year colleges in years 4–5. Statewide quasi-experimental and implementation research focused on all community colleges in Texas.

Sample: To assess student progress and outcomes for the continuous improvement colleges, the study team examined data on students enrolled between 2014 and 2018, comparing students in the six study colleges to similar students in other community colleges in Texas. These analyses examined outcomes for all developmental education students and outcomes for the students most likely to be impacted by the policies, students near college readiness and students testing at ABE-levels. Statewide analysis of survey data on implementation included all community colleges in the state. Quasi-experimental analysis on holistic advising factors focused on all students enrolled in Texas community colleges between 2013 and 2015, though findings from regression discontinuity are most applicable for students near the college-ready cut score. Quasi-experimental analysis assessing the outcomes associated with integrated reading and writing focused on statewide data from 2011 through 2016, with a focus on students enrolled in developmental education in community colleges.

Approach: Texas developed new test scores to identify "ABE-level" students (those testing below 9th grade equivalence) and recommended that institutions consider requiring additional supports for these students, including academic support appended to the developmental education course, and referrals into adult basic education or technical programs. Although study colleges primarily continued to place ABE students into developmental education, they developed additional (typically optional) targeted supports through the project, including bridge courses, additional lab time for developmental education work, and versions of developmental education courses that integrated some additional language support. The state's holistic advising policy called on institutions to use other measures (e.g., high school grades, non-cognitive factors, transportation issues) to place students into developmental education, rather than the traditional approach of focusing solely on test scores. The study institutions developed new processes and forms for collecting the student information and integrating it into placement. Corequisites, the third focus area, places students directly into college courses with aligned developmental education support rather than requiring students to complete developmental education courses prior to entering college courses. The study provided colleges with technical assistance on developing continuous improvement plans and tracking data to inform the implementation of corequisites through a series of webinars, in-person convening sessions and workshops, and coaching sessions. The Texas colleges receiving this technical assistance implemented a wide range of corequisite models, and staff at these colleges used the tools provided to monitor and shape improvements to corequisite instruction, advising, and support.

Research Design and Methods: The partnership used annual Plan-Do-Study-Act cycles with working group institutions to test approaches to implementing state-recommended reforms, and these cycles included site visits each semester and qualitative data collection and analysis. The team conducted qualitative analysis of findings from six working group institutions and descriptive analysis of statewide institutional survey data on developmental education practices to assess implementation in response to the state's reforms. The research team used a difference-in-difference approach to examine the impacts of differential access to reading and writing coursework across institutions. To assess the heterogeneity in outcomes across factors reported as being used for holistic advising in Texas, the research team used a regression discontinuity approach.

Control Condition: When tracking progress for continuous improvement institutions, researchers compared outcomes for study colleges to all other Texas community colleges not participating in the study; these colleges were also likely engaging in reforms but not participating in continuous improvement. In the difference-in-difference analysis of the integration of the developmental reading and writing curriculum, researchers compared outcomes students with access to more integrated sequences to students with less integrated sequences. In the regression discontinuity analysis assessing heterogeneity in outcomes across factors used for holistic advising, the control condition is students who test below the cut score and are likely to be assigned to developmental education courses.

Key Measures: Measures for categorizing holistic advising approaches across institutions included the number and types of holistic factors used in advising; assessment cut scores used to identify students for holistic advising; and subjective versus objective advising processes. Measures for categorizing approaches to ABE students across institutions included number and types of pathways offered to ABE-level students; cut scores above which institutions accelerate ABE-level students into academic pathways; and number and types of non-course-based support options. Short-term outcomes for the continuous improvement research and quasi-experimental studies included grades in assigned courses, credits accumulated, semester-to-semester persistence, and time to completion of first college-level course in math or English. Longer-term outcomes included persistence into the third semester, total credits accumulated during the study period, and transfer to a four-year institution or completion of a degree within three years.

Data Analytic Strategy: During annual improvement cycles, the research team conducted analysis of qualitative and quantitative data collected at each institution and summarized key themes that emerged and were relevant for improvement and the implementation studies. For both working groups, researchers tracked key metrics and benchmarked each working group institution among the working group as a whole and against statewide averages for community colleges. To compare outcomes of the continuous improvement colleges to students in other Texas community colleges, we used differences-in-differences analysis of data from 2014–2018. To examine integrated reading and writing, a broad developmental education reform impacting ABE-level students, the team conducted differences-in-differences analysis on data from 2011–2016 to compare outcomes for students enrolled in colleges with more integrated sequences to students enrolled in colleges with less integrated sequences. To build evidence on which factors might be useful for holistic advising, the research team used regression discontinuity analysis with data from 2013–2015 to assess heterogeneity in outcomes for different groups of students when accelerated into college coursework. The quasi-experimental models included a rich array of student-level control variables that were available in state administrative databases.

Related IES Projects: Designing a RCT Experiment to Test the Impact of Innovative Interventions and Policies for Postsecondary Developmental Education: A RAND—TX Higher Education Coordinating Board Research Partnership (R305H130026); An Experimental Evaluation of Accelerated Pathways through Developmental Education—A RAND-THECB Partnership (R305H170085)

Products

The products of this research included a policy brief on Texas approaches to serving students who test at ABE levels on the state exam, a quasi-experimental journal article on holistic factors, a conference proceedings working paper, a quasi-experimental working paper on integrated reading and writing, a toolkit on corequisite continuous improvement, and a wide range of technical assistance resources, sessions, and workshops for Texas practitioners and policymakers.

ERIC Citations: Find available citations in ERIC for this award here.

Selected Publications

Carew, D.G., Daugherty, L., Karam, R., Miller, T., Mendoza-Graf, A. (2018). Practitioner Perspectives on Implementing Developmental Education Reforms: A Convening of Six Community Colleges in Texas.Santa Monica: CA, RAND.

Daugherty, L., Karam, R., Basco, D., and Kaufman, J.H. (2019). Tools for Improving Corequisite Models: A Guide for College Practitioners.Santa Monica: CA, RAND.

Daugherty, L., Miller, T., Carew, D.G. (2019). Supporting College Enrollees Who Test at the Lowest Levels of Readiness: Lessons from Texas Community Colleges. Santa Monica: CA, RAND.

Daugherty, L., Gerber, R., Martorell, P., Miller, T., and Weisburst, E. (2021). Heterogeneity in the Effects of College Course Placement, Research in Higher Education,62(7), 1086–1111.

Miller, T, Gerber, R. and Daugherty, L. (2016). New Approaches to Developmental Education Pathways Integrating Reading and Writing Remediation. AEFP Conference Working Paper.


Back