Co-Principal Investigators: Schudde, Lauren; Kim, Dae

Purpose: This project will examine the effectiveness of sweeping developmental education (DE) reforms implemented across California community colleges, given increasing evidence that traditional DE harms students' chances of completing a credential, contributes to racial/ethnic gaps in completion, and represents a significant investment on the part of colleges and families. California's systemwide DE curricular reforms include revising placement criteria for introductory transfer-level math and English courses and providing cocurricular supports to maximize students' probability of success. The research will analyze the impact of these reforms on student outcomes, and the costs of implementing these reforms. Similar reforms are being implemented in states, systems, and institutions across the country, but the field still lacks evidence about which reforms are most effective and for which student subpopulations. This project will answer the need for more detailed research on developmental education reform.

Project Activities: The study will use a mixed-methods design to examine the implementation, impact, and cost-effectiveness of curricular reform across community colleges in California. The implementation study will explore the variation in scale of implementation across colleges and between math and English departments; the level of institutional capacity and buy-in among faculty and staff in support of the reform; and students' understandings of and experiences with the curricular reforms. The impact study will include both student- and institution-level analyses to evaluate whether placement of first year students into transfer-level math and English courses with cocurricular support improves success for California community college students on a variety of short- and long-term outcomes. Student-level analyses will evaluate the effects of transfer-level course placement compared to placement into traditional DE courses, and will also explore variation in effects across transfer-level courses with different models of cocurricular support and across various student subgroups. Institution-level analyses will examine how increases in access to transfer-level math and English courses over time predict aggregate student outcomes. Finally, the cost study will compare the costs associated with introductory transfer-level math and English courses with cocurricular support to the traditional sequence of DE courses leading to transfer-level courses. A cost-effectiveness analysis will account for student outcomes and answer whether introductory transfer-level courses with support are more cost-effective than traditional DE courses, and if there are differences in cost-effectiveness across corequisite support models.

Products: The project will produce internal reports both to the California Community Colleges Chancellor's Office and to institutions participating in qualitative data collection, as well as publicly available briefs, reports, and peer-reviewed publications. The research team will present its findings at conferences attended by researchers and policymakers.

Structured Abstract

Setting: The research will take place in the California Community College system across 114 community colleges, the largest postsecondary system in the country.

Sample: Analytic samples for the impact study include multiple cohorts of approximately 200,000 first-time students entering California's community colleges, including both pre-policy (fall 2014 through fall 2018) and post-policy periods (fall 2019 through fall 2023). The implementation study, which explores the variation in implementation of the reforms, will focus on a sample of 15 institutions representing variation in the scale of implementation, and will include interviews and focus groups of approximately 150 institutional administrators, faculty, and staff and 360-480 students across the sample. The cost effectiveness study will be conducted using data collected from the 15 implementation sample colleges.

Intervention: Assembly Bill 705 (AB 705) was signed into law in 2017, requiring all California community colleges to revise their placement policies to maximize the likelihood that entering students can complete transfer-level English and math courses within one year; institutions were required to comply by fall 2019. In response to AB 705, California's community colleges are modifying placement policies and practices to rely primarily on high school transcript data to place more students out of developmental education and into transfer-level courses. Colleges are concurrently implementing structural and curricular reforms to accommodate the increased placement of students in introductory transfer-level math and English courses as well as designing cocurricular supports to promote student success in transfer-level courses.

Research Design and Methods: The study will use a mixed-methods design that combines a qualitative implementation study, a quasi-experimental impact study, and a cost-effectiveness analysis. Part I of the impact study will use a regression discontinuity (RD) design to examine whether there is a jump in the observed student outcome at the cutoff score of the running variable, high school GPA, by comparing outcomes of students whose high school GPAs are below the cutoff and enrolled in a prerequisite DE course (control condition) to those who are above the cutoff and enrolled in a transfer-level introductory course (treatment condition). Part II of the impact study will use an interrupted time-series (ITS) design with matching to measure the effects of the reform by comparing the outcomes of first-time college students in post-policy cohorts (treatment condition) to the outcomes of those in pre-policy cohorts (control condition). The implementation study will use interviews, focus groups, and surveys to explore the variation in scale of implementation across colleges and between math and English departments, the level of institutional capacity and buy-in among faculty and staff, and student understanding of and experience with the curricular reforms, particularly among underrepresented student groups. The cost study will assess the cost-effectiveness of offering an introductory transfer-level math or English course with cocurricular support, compared with prerequisite DE courses, using the ingredients method.

Control Condition: In the RD analysis, students who remain in developmental courses will serve as the control group. In the ITS analysis, students in pre-policy (AB 705) courses will serve as the control group.

Key Measures: From high school transcript data, the researchers will use high school GPA as the running variable for the RD analysis. They will also leverage additional measures, such as course grade and highest course completed in math or English, to assess and validate colleges' course placement standards. College-level independent variables include those capturing course enrollment type (e.g., prerequisite developmental, transfer-level, and corequisite support courses) and type of cocurricular support provided. The short-term outcome measures include completion of introductory transfer-level math and/or English within the first year of college, transfer-level credits accrued in first year, and persistence into the subsequent term. Long-term outcome measures include certificate and associate degree attainment, time to credential attainment, overall college GPA, and transfer to baccalaureate-granting colleges and universities in California.

Data Analytic Strategy: Researchers will employ the RD model to estimate the impact of placing students in a transfer-level course vs. a prerequisite DE course, controlling for the effects of the assignment variable (i.e., high school GPA), student characteristics (e.g., gender, race/ethnicity, socioeconomic status, etc.), and college fixed effects. They will further extend the RD model to conduct a moderator analysis by introducing interaction terms to test whether specific student subgroups receive more benefit from the new placement policy, and to explore variation in effects across types of cocurricular supports. The ITS model will examine whether student outcomes have improved after the placement reform by comparing the post-reform trend in student outcomes to the baseline trend, controlling for the same set of student variables used in the RD model. Researchers will conduct a series of subgroup analyses to examine the variation in the policy effects across students in different demographic subgroups as well as students in different high school grade bands (e.g., high, middle, and low GPA groups). They will conduct cost analyses following the ingredients method framework.