Project Activities
The research team carried out a 3,800-student, 4-college randomized controlled trial to estimate the impact of CS on student academic outcomes. Researchers examined the extent to which CS is more effective for particular subgroups of students and whether effects vary across the four participating colleges. Researchers defined confirmatory subgroups based on students' intent to enroll in the full- or part-time CS program, number of developmental needs, and college. Exploratory subgroups include race, gender, age, high school diploma/GED, cohort (Fall vs. Spring). All subgroups were defined based on pre-random assignment data. The team collected information about the implementation of CS across campuses in the CUNY system (led by project collaborator, Community College Research Center (CCRC)) and on the costs associated with CS (also led by CCRC).
Structured Abstract
Setting
The research took place at four of the six colleges implementing CUNY Start (CS) in the City University of New York (CUNY) network.
Sample
The sample for this project is students with significant remediation needs at the participating CUNY community colleges. The analytic sample includes 3,835 students. The analytic sample included more women than men and was divided almost equally between students aged 19 or younger and students aged 20 or older. Black and Hispanic students made up most of the sample, and a substantial proportion of the sample had a native language other than English. The sample was split almost evenly between students who had developmental education requirements in all three subject areas and those with fewer requirements, with more students having such requirements in math than in reading and writing.
CUNY Start (CS) is a multifaceted pre-matriculation program that provides intensive instruction in reading, writing, and math through a carefully prescribed curriculum and instructional delivery method to incoming postsecondary students with significant remedial needs. CS condenses the time students spend preparing for college-level English and math into a single semester. Full-time students take classes in reading/writing and math for 24 hours per week; part-time students take either reading/writing or math for 12 hours per week. In addition, CS delivers enhanced academic and nonacademic supports in the form of advisors, tutors, and a weekly seminar that builds college success skills. CS's instructional intensity, integrated supports for students and faculty, and position in continuing education distinguish it from the traditional developmental college experience. The underlying theory of change posits that students with developmental needs are best served through a cohort-based integrated academic and nonacademic support model designed to build academic preparedness, confidence, and college know-how prior to matriculating in college. Costing only $75 for the semester, CS allows students to preserve their financial aid for future college-credit courses.
Initial research
The research for the evaluation of CS was composed of three complementary components: (1) a randomized trial to estimate the effects of CS on student academic progress and completion, (2) implementation research focused on how the program is put into effect, fidelity, and the service contrast, and (3) cost analyses. The random assignment design enables unbiased estimation of the causal effects of CS. Over three semesters, approximately 3,800 students were assigned to either a control group or a program group during the class registration process. The target population for the study included students who failed the reading or writing and math placement assessments for the full-time program and students who failed one or more of these tests for the part-time program. The evaluation examines the extent that CS is more effective for particular subgroups of students and whether effects vary across the four study colleges. Confirmatory subgroups are defined based on students' intent to enroll in the full- or part-time CS program, number of developmental needs, and college. Exploratory subgroups include race, gender, age, high school diploma/GED, and cohort (Spring vs. Fall). All subgroups are defined based on pre-random assignment data.
Control condition
Control group members have the opportunity to matriculate in college and enroll in traditionally structured courses. Most students enrolled in one or two developmental (remedial) courses offered for 3 to 6 hours per week. They also enroll in some college-level courses (e.g., psychology).
Key measures
Using college transcript records, the researchers collected data on the confirmatory outcomes: reduction and/or elimination of remedial needs and college-level credit accumulation (a proxy for overall academic progress toward a degree). Using college records data and data from the National Student Clearinghouse the researches collected data on completion of a degree or certificate, the final confirmatory outcome, The researchers also explored the program's effects on students' knowledge and skills through college-administered standardized exams (e.g., the CUNY Assessment Test for reading and writing). In addition, they measured implementation fidelity and the service contrast through surveys (student and instructor), interviews and focus groups (instructors, advisors/counselors, academic support staff, and administrators), classroom observations, and transcript data.
Data analytic strategy
Researchers used standard statistical tests to compute intent-to-treat estimates by comparing average outcomes of the program group and control group members (controlling for college and a small set of covariates, such as baseline placement test scores). The research team also conducted subgroup analyses based on students' intent to enroll in the full or part-time program, level of remedial need, and college. Researchers estimated the program's effects for each subgroup and conducted statistical tests for evidence of differences in subgroup effects.
Key outcomes
The main findings of this study are as follows:
- Implementation: Overall, CUNY Start was implemented with fidelity to the program model, and there was a substantial contrast between the program and the colleges' standard courses and services, including their standard developmental education courses. For details see Scrivener et al., 2018 or Weiss et al. (forthcoming).
- Impact: Over three years, including one semester that students spent in the program and two and- a-half semesters after the program was complete, CUNY Start substantially increased college readiness, slightly increased credit accumulation, and modestly increased graduation rates (by increasing participation in CUNY's highly effective ASAP). These were the study's three confirmatory outcomes. For details see Weiss et al. (forthcoming).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
WWC review:
Meets WWC Standards without reservations: https://ies.ed.gov/ncee/wwc/Study/86380
Publications:
Book chapter
Cormier, M., and Bickerstaff, S. (2019). Research on Developmental Education Instruction for Adult Literacy Learners. In D. Perin (Ed.), The Wiley Handbook of Adult Literacy (pp. 541-561).
Journal article
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (accepted). An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at The City University of New York. Educational Evaluation and Policy Analysis.
Working Papers
Belfield, C. (2020). Cost and cost-effectiveness analysis of CUNY Start. New York, NY: Community College Research Center, Teachers College, Columbia University.
Bickerstaff, S., and Edgecombe, N. (2019). Teaching Matters and So Does Curriculum: How CUNY Start Reshaped Instruction for Students Referred to Developmental Mathematics. CCRC Working Paper No. 110.
Cormier, M. S. & Bickerstaff, S. (2020). How Can We Improve Teaching in Higher Education? Learning From CUNY Start. New York, NY: Community College Research Center, Teachers College, Columbia University.
Kim, J., Simon, M., Horenstein, A., & Webber, A. (2020). A Resource Guide for Reforming Developmental Education in the College Transition Space. New York: City University of New York.
Scrivener, S., Gupta, H., Weiss, M. J., Cohen, B., Cormier, M. S., & Brathwaite, J. (2018). Becoming College-Ready: Early Findings from a CUNY Start Evaluation. New York: MDRC.
Scrivener, S., and Logue, A. W. (2016). Building College Readiness before Matriculation: A Preview of a CUNY Start Evaluation. Research Brief. MDRC.
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (2020). An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at The City University of New York. New York: MDRC.
Available data:
Data may be made publicly available in early 2021 through ICPSR as part of The Higher Education Randomized Controlled Trials project.
Related projects
Supplemental information
Co-Principal Investigators: Logue, Alexandra; Wrigley, Julia
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
