Project Activities
The research team will carry out both a descriptive analysis and an evaluation of the EAP. The descriptive analysis will identify the characteristics of students identified as college ready and the schools they attend, and will document how these schools have changed with California's switch to Smarter Balanced. The descriptive analysis will also track students in Career and Technical Education (CTE) pathways in K–12 into postsecondary schooling to examine general postsecondary outcomes, including persistence, degree attainment, and employment. Researchers will employ a regression discontinuity design to estimate the impact of being placed into the categories of college readiness and the concurrent advice provided by category on 12th grade coursetaking and academic performance, college application and enrollment, and postsecondary remedial coursetaking and for-credit coursetaking. Researchers will also examine whether any impact differs by student demographics or school characteristics.
Structured Abstract
Setting
The project will take place in the state of California.
Sample
The study will include all California 11th graders in public schools during the academic years 2009-2010 through 2019-2020.
Starting in in 2014–2015, 11th grade students take the Smarter Balanced computer adaptive assessments for English—language arts/literacy and mathematics. These tests are based on the Common Core State Standards, which California began implementing in 2010. Based on their scores, students are categorized into one of four levels of achievement, with Level 4 identified as “college ready” and Level 3 as “conditionally ready.” Level 4 students can directly enroll in college-level courses at a California college without having to take a remedial placement test. Level 3 students receive a letter advising them that they can directly enroll in college-level courses if they complete an approved course (an advanced English course or a math course that requires Algebra II as a prerequisite) in 12th grade with a grade of C or above. Level 2 students receive general advice on preparing for college and interpreting the test results, but do not receive specific advice on 12th grade coursetaking, and are expected to have to take the postsecondary remedial placement test. California began the EAP in 2004 using the California Standards Tests augmented with a set of items that students could voluntarily complete and a focus on student entry in the CSU system. Since then, the community college system joined the program, the testing has become mandatory, and the Smarter Balanced Summative Assessments have replaced the augmented California Standards Tests.
Initial research
The team will use a regression discontinuity design to compare impacts of the students just above and just below each cutoff in the Smarter Balanced Summative Assessments that is used to place students into the four college readiness categories (e.g., comparing students who scored just high enough to be identified as “college ready” with those just missing that cutoff). Researchers will estimate impacts for 12th grade coursetaking and academic performance, college application and enrollment, and postsecondary remedial coursetaking and for-credit coursetaking. Researchers will also examine how the effect of these signals on these outcomes differs by student and school characteristics (e.g., student demographics, student prior achievement, high school characteristics).
Control condition
For each college readiness category, students in the adjacent category representing a lower stage of college readiness will be the comparison group (i.e., Level 4 compared to Level 3, Level 3 to Level 2, and Level 2 to Level 1).
Key measures
In this study, measures of outcomes are 12th grade coursetaking and academic performance, college application and enrollment, and postsecondary remedial coursetaking and for-credit coursetaking. The measures of college readiness are the scores on 11th grade English language arts tests and a mathematics test. Covariates include student demographics, prior student achievement, and high school characteristics.
Data analytic strategy
For the descriptive analysis, researchers will use density distributions of student characteristics, a linear random effects model to compare schools, and correlations between the test and postsecondary success. In addition, researchers will use difference-in-difference regression to compare student characteristics and test score to postsecondary success before and after the adoption of the Smarter Balanced Summative assessments. For the evaluation study, researchers will use a regression discontinuity design using the test cutoffs used to place students into the four college readiness categories. The research team will examine scores around the cutoffs for manipulation and observable covariates for differences around the cutoffs.
People and institutions involved
IES program contact(s)
Project contributors
Partner institutions
California Department of Education
Products and publications
Products: The research team will provide evidence of the efficacy of California's EAP. Researchers will provide this evidence directly to the California Department of Education and to the three public higher education systems in California - California Community Colleges(CCC), California State University (CSU), and University of California(UC) - through ongoing joint meetings, briefings, and policy papers. Researchers will present results to the research community through presentations and peer-reviewed publications and to the practitioner and policymaker communities through a series of policy briefs, reports, seminars, and conferences supported by Policy Analysis for California Education (PACE).
Additional project information
This grant was originally funded under the FY 2015 Evaluation of State Education Programs and Policies competition.
Related projects
Supplemental information
Co-Principal Investigators: Isler, Jonathan; Carrell, Scott E.; Martorell, Francisco (Paco)
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
