|Title:||Evaluating the Impact of CUNY Start through a Researcher (MDRC) – Local Education Agency (City University of New York) Partnership|
|Principal Investigator:||Weiss, Michael||Awardee:||MDRC|
|Program:||Evaluation of State and Local Education Programs and Policies [Program Details]|
|Award Period:||5 years (7/1/2014-6/30/2019)||Award Amount:||$4,796,087|
Co-Principal Investigators: Logue, Alexandra; Wrigley, Julia
Purpose: Most community college students are referred to developmental education courses to build basic skills. Colleges have typically responded to the needs of students referred to developmental coursework by designing multi-level, non-credit course sequences in reading, English, and math. However, students often struggle in these courses and college more broadly. To help address the needs of incoming students, the City University of New York (CUNY) developed CUNY Start (CS), a multifaceted pre-matriculation program that provides intensive instruction in reading, writing, and math with a prescribed pedagogy delivered by trained teachers. CS is an ambitious initiative designed to help students with low basic skills levels succeed in college. Prior to this project there was no experimental evidence of CS's success. In this study, researchers found evidence of the estimated effect of CS on students' academic outcomes, how these effects vary by population and context, how the program is implemented, and its associated costs. These findings have helped CUNY refine its institutional strategy and contribute to the national discussion on how to increase the success rate of students who enter community college with substantial remedial needs.
Project Activities: The research team carried out a 3,800-student, 4-college randomized controlled trial to estimate the impact of CS on student academic outcomes. Researchers examined the extent to which CS is more effective for particular subgroups of students and whether effects vary across the four participating colleges. Researchers defined confirmatory subgroups based on students' intent to enroll in the full- or part-time CS program, number of developmental needs, and college. Exploratory subgroups include race, gender, age, high school diploma/GED, cohort (Fall vs. Spring). All subgroups were defined based on pre-random assignment data. The team collected information about the implementation of CS across campuses in the CUNY system (led by project collaborator, Community College Research Center (CCRC)) and on the costs associated with CS (also led by CCRC).
Pre-registration site: Not posted publicly, but available upon request.
Key Outcomes: The main findings of this study are as follows:
Setting: The research took place at four of the six colleges implementing CUNY Start (CS) in the City University of New York (CUNY) network.
Sample: The sample for this project is students with significant remediation needs at the participating CUNY community colleges. The analytic sample includes 3,835 students. The analytic sample included more women than men and was divided almost equally between students aged 19 or younger and students aged 20 or older. Black and Hispanic students made up most of the sample, and a substantial proportion of the sample had a native language other than English. The sample was split almost evenly between students who had developmental education requirements in all three subject areas and those with fewer requirements, with more students having such requirements in math than in reading and writing.
Intervention: CUNY Start (CS) is a multifaceted pre-matriculation program that provides intensive instruction in reading, writing, and math through a carefully prescribed curriculum and instructional delivery method to incoming postsecondary students with significant remedial needs. CS condenses the time students spend preparing for college-level English and math into a single semester. Full-time students take classes in reading/writing and math for 24 hours per week; part-time students take either reading/writing or math for 12 hours per week. In addition, CS delivers enhanced academic and nonacademic supports in the form of advisors, tutors, and a weekly seminar that builds college success skills. CS's instructional intensity, integrated supports for students and faculty, and position in continuing education distinguish it from the traditional developmental college experience. The underlying theory of change posits that students with developmental needs are best served through a cohort-based integrated academic and nonacademic support model designed to build academic preparedness, confidence, and college know-how prior to matriculating in college. Costing only $75 for the semester, CS allows students to preserve their financial aid for future college-credit courses.
Research Design and Methods: The research for the evaluation of CS was composed of three complementary components: (1) a randomized trial to estimate the effects of CS on student academic progress and completion, (2) implementation research focused on how the program is put into effect, fidelity, and the service contrast, and (3) cost analyses. The random assignment design enables unbiased estimation of the causal effects of CS. Over three semesters, approximately 3,800 students were assigned to either a control group or a program group during the class registration process. The target population for the study included students who failed the reading or writing and math placement assessments for the full-time program and students who failed one or more of these tests for the part-time program. The evaluation examines the extent that CS is more effective for particular subgroups of students and whether effects vary across the four study colleges. Confirmatory subgroups are defined based on students' intent to enroll in the full- or part-time CS program, number of developmental needs, and college. Exploratory subgroups include race, gender, age, high school diploma/GED, and cohort (Spring vs. Fall). All subgroups are defined based on pre-random assignment data.
Control Condition: Control group members have the opportunity to matriculate in college and enroll in traditionally structured courses. Most students enrolled in one or two developmental (remedial) courses offered for 3 to 6 hours per week. They also enroll in some college-level courses (e.g., psychology).
Key Measures: Using college transcript records, the researchers collected data on the confirmatory outcomes: reduction and/or elimination of remedial needs and college-level credit accumulation (a proxy for overall academic progress toward a degree). Using college records data and data from the National Student Clearinghouse the researches collected data on completion of a degree or certificate, the final confirmatory outcome, The researchers also explored the program's effects on students' knowledge and skills through college-administered standardized exams (e.g., the CUNY Assessment Test for reading and writing). In addition, they measured implementation fidelity and the service contrast through surveys (student and instructor), interviews and focus groups (instructors, advisors/counselors, academic support staff, and administrators), classroom observations, and transcript data.
Data Analytic Strategy: Researchers used standard statistical tests to compute intent-to-treat estimates by comparing average outcomes of the program group and control group members (controlling for college and a small set of covariates, such as baseline placement test scores). The research team also conducted subgroup analyses based on students' intent to enroll in the full or part-time program, level of remedial need, and college. Researchers estimated the program's effects for each subgroup and conducted statistical tests for evidence of differences in subgroup effects.
Related project: A Leaky Pipeline: Community College Students and Pathways to the Bachelor's Degree (R305A180139), Assessing the Long-Term Efficacy and Costs of the City University of New York's (CUNY'S) Accelerated Study in Associate Programs (ASAP) (R305A160273), The Higher Education Randomized Controlled Trial (THE-RCT) Project: Synthesizing Evidence from 15+ Years of RCTs in Postsecondary Education (R305A190161)
Publications and Products
Cormier, M., and Bickerstaff, S. (2019). Research on Developmental Education Instruction for Adult Literacy Learners. In D. Perin (Ed.), The Wiley Handbook of Adult Literacy (pp. 541–561).
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (accepted). An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at The City University of New York. Educational Evaluation and Policy Analysis.
Belfield, C. (2020). Cost and cost-effectiveness analysis of CUNY Start. New York, NY: Community College Research Center, Teachers College, Columbia University.
Bickerstaff, S., and Edgecombe, N. (2019). Teaching Matters and So Does Curriculum: How CUNY Start Reshaped Instruction for Students Referred to Developmental Mathematics. CCRC Working Paper No. 110.
Cormier, M. S. & Bickerstaff, S. (2020). How Can We Improve Teaching in Higher Education? Learning From CUNY Start. New York, NY: Community College Research Center, Teachers College, Columbia University.
Kim, J., Simon, M., Horenstein, A., & Webber, A. (2020). A Resource Guide for Reforming Developmental Education in the College Transition Space. New York: City University of New York.
Scrivener, S., Gupta, H., Weiss, M. J., Cohen, B., Cormier, M. S., & Brathwaite, J. (2018). Becoming College-Ready: Early Findings from a CUNY Start Evaluation. New York: MDRC.
Scrivener, S., and Logue, A. W. (2016). Building College Readiness before Matriculation: A Preview of a CUNY Start Evaluation. Research Brief. MDRC.
Weiss, M. J., Scrivener, S., Slaughter, A., & Cohen, B. (2020). An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at The City University of New York. New York: MDRC.
Publicly available data: Data may be made publicly available in early 2021 through ICPSR as part of The Higher Education Randomized Controlled Trials project.
WWC Review: Meets WWC Standards without reservations: https://ies.ed.gov/ncee/wwc/Study/86380