Purpose: In this project, researchers examined two distinct interventions for Texas community college students who were referred to non-credit remedial math. At the time of this study, the majority of entering community college students were referred to developmental (or remedial) math, and success rates in the developmental math course sequence and college more broadly were discouraging. The first intervention the researchers examined was called ModMath, a modularized course using computer-assisted instruction. ModMath allowed students to move at their own pace through the math curriculum. The second intervention was a state-wide course-placement policy that routed students who score very low on a placement test into adult basic education (ABE) services at the community college rather than the lowest level of developmental math. Over the course of this project, researchers studied the implementation of these approaches and gathered data to provide evidence about their impact on student outcomes such as academic progress as measured by completing developmental math, credit attainment, and passing the first college-level math course.

Project Activities: Researchers used a randomized controlled trial to study the impact of ModMath at a large college in Texas. Researchers conducted descriptive analyses to understand the implementation of Texas policy on a college's use of ABE referrals for low-scoring students. In both cases, implementation data was collected, and student academic outcomes were tracked for at least two semesters.

Pre-registration site: The ModMath study's pre-analysis plan can be found at https://sreereg.icpsr.umich.edu/sreereg/subEntry/2351/pdf?section=all&action=download.

Key Outcomes:

Main findings of the ModMath study (Weiss and Headlam, 2019)

• ModMath was well-implemented.
• Researchers did not find evidence that ModMath was superior to the "traditional" non-modularized direct-instruction math course.

Main findings from the ABE study (Boatman et al., 2020)

• Regardless of remedial course referral type (traditional developmental or ABE alternative), 50 percent of enrolled students did not attempt any math remediation upon entering college.
• Compliance with course referrals was low. The full population of test-takers with math skills ranging from 4th grade to 8th grade were less likely to be enrolled at HCC and less likely to enroll in their assigned math class than students with math skills ranging from 9th grade to 12th grade, both in the first semester, but also up to two years after taking the placement test.
• Among enrolled students, few enrolled in college-level math by the end of four semesters. Only 17 percent of students assigned to developmental math enrolled in a college-level math course by the end of 4 semesters, compared to 9 percent of students assigned to ABE.
• Approximately 50 percent of students assigned to ABE enrolled in the ABE course within 4 semesters. However, only 32 percent of students eventually passed the course within that time frame. Of the students assigned to developmental math, 63 percent enrolled in the course by the end of 4 semesters, and 33 percent of these students had passed the course within 4 semesters. There was a five percent difference in the percent of ABE students compared to developmental math students passing college-level math by the end of four semesters.

Structured Abstract

Setting: This project took place in publicly funded community colleges in Texas (one community college for the ModMath study, and two community colleges for the ABE study).

Sample: Approximately 1,400 students participated in the ModMath Study, and the data from approximately 5,500 students was collected for the ABE Study.

Intervention: The first intervention, ModMath, was a computer-assisted instructional approach with the goal of accelerating students' progress through the developmental math sequence. ModMath divided the developmental math curriculum into six discrete computerized modules, allowing students to learn at their own pace and spend less time on skills they either already know or can master quickly. Students worked through modules identified by an assessment and then advanced as quickly as they are capable of learning the material. ModMath classrooms were set up like computer labs, with an instructor who circulates among the students to offer assistance when needed.

The second intervention consisted of a set of policy guidelines developed by the Texas Higher Education Coordinating Board. They required colleges to students with low scores on the statewide math placement exam into ABE offerings, which vary by college, rather than developmental education. Scores on the new placement test in Texas, the Texas Success Initiative Assessment (TSIA), first divided students into two groups: college-ready and below college-ready. Those who score below the college-ready cutoff were given a second set of items, the Adult Basic Education Diagnostic. Students who tested into the top two levels (Levels 6 and 5) were referred to developmental education courses; students who score at the middle two levels (4 and 3) were referred to ABE programs associated with the community college; and students at the bottom two levels (1 and 2) were referred to other programs, e.g., either federally funded ABE programs in the community, workforce programs on campus, or other alternatives.

Research Design and Methods: The research team conducted two separate studies of these two interventions, using different methodologies.

For the ModMath Study, the team carried out a randomized control trial to estimate the impact of ModMath on student outcomes and to determine if impacts vary by student characteristics. Researchers also gathered data on the implementation of ModMath to determine the degree to which it was implemented with fidelity and to describe how the ModMath experience differs from "business as usual" —lecture-based math instruction.

For the ABE Study, researchers attempted a regression discontinuity design (RDD) to compare students whose scores on the ABE Diagnostic place them at the cusp between developmental education and ABE. Conducting an RDD analysis in this context requires a continuous test score used to independently assign students to the treatment (ABE) or control condition (developmental math). However, due to missing data, the researchers had to calculate a "predicted test score" in instances where the actual test score was missing. Because the predicted scores failed empirically testable assumptions of the regression discontinuity design (many laid out in the What Works Clearinghouse (WWC) standards for RDD validity), the attempted RDD did not prove fruitful, and descriptive analyses of students' progression through ABE and developmental math classes were conducted.

For the ABE Study, researchers attempted a regression discontinuity design (RDD) to compare students whose scores on the ABE Diagnostic place them at the cusp between developmental education and ABE. Conducting an RDD analysis in this context requires a continuous test score used to independently assign students to the treatment (ABE) or control condition (developmental math). However, due to missing data, the researchers had to calculate a "predicted test score" in instances where the actual test score was missing. Because the predicted scores failed empirically testable assumptions of the regression discontinuity design (many laid out in the What Works Clearinghouse (WWC) standards for RDD validity), the attempted RDD did not prove fruitful, and descriptive analyses of students' progression through ABE and developmental math classes were conducted.

Control condition: For the ModMath Study, the control group students were referred to traditional, lecture-based developmental math classes. These semester-long classes primarily used direct-instruction, were not self-paced, and were not modularized. For the ABE Study, students who are referred to developmental math served as the control group while students who are referred to ABE serve as the program group.

Key Measures: Researchers used college transcript records for primary outcomes. Key measures include students' academic progress in mathematics and non-mathematics courses (such as completing the ABE or developmental math course, passing the first college-level math course, credit accumulation in mathematics and overall). The research team also collected data on implementation fidelity and the treatment contrast through interviews with faculty and students and classroom observations. For the ModMath Study, researchers also used student and faculty surveys as implementation measures.

Data Analytic Strategy: For both the ModMath Study, researchers computed intent-to-treat estimates by comparing average outcomes of the program group and control group members and those of selected subgroups (e.g., part-time versus full-time status, high versus low familiarity with technology, gender, minority status). Because the ABE study will take place in two community college systems, the researchers will analyze the data from each system separately.

Products and Publications

ERIC Citations:  Find available citations in ERIC for this award here.

Publicly available data: MDRC's The Higher Education Randomized Controlled Trials Restricted Access File (THE-RCT RAF), United States, 2003-2019 (ICPSR 37932)

Project Website: https://www.mdrc.org/work/projects/developmental-education-acceleration-project .

WWC Review: Weiss, M. J., & Headlam, C. (2019). A randomized controlled trial of a modularized, computer-assisted, self-paced approach to developmental math. Journal of Research on Educational Effectiveness, 12(3), 484-513. [WWC Review]

Selected Publications:

Journal articles

Weiss, M. J. & Headlam, C. (2019). A randomized controlled trial of a modularized, computer-assisted, self-paced approach to developmental math. Journal of Research on Educational Effectiveness, 12(3), 484-513. doi: 10.1080/19345747.2019.1631419.

Nongovernment peer-reviewed reports

Boatman, A., Cerna, O., Reiman, K., Diamond, J., Visher, M. G., & Rutschow, E. Z. (2020). Building a New" Bridge" to Math: A Study of a Transition Program Serving Students with Low Math Skills at a Community College. MDRC. Full text

Cerna, O. (2019). Building Basic Math Skills: Boot Camp at Tarrant County College. MDRC. Full text

Gardenhire, A., Diamond, J., Headlam, C., & Weiss, M. J. (2016). At Their Own Pace: Interim Findings from an Evaluation of a Computer-Assisted, Modular Approach to Developmental Math. MDRC. Full text

Visher, M., Cerna, O., Diamond, J., & Rutschow, E. Z. (2017). Raising the Floor: New Approaches to Serving the Lowest-Skilled Students at Community Colleges in Texas and Beyond. MDRC. Full Text

Weiss, M.J., Headlam, C. (2018). A Randomized Controlled Trial of a Modularized, Computer-Assisted, Self-Paced Approach to Developmental Math. MDRC, NY. Full Text