The Evaluation of Career and College Promise
Co-Principal Investigators: Unlu, Fatih; Gold, Kimberly; Shah-Coltrane, Sneha
Partner Institutions: RAND Corporation, North Carolina Community College System, North Carolina Department of Public Instruction
Supplemental Award: 2 years (FY 2021 – FY 2023) $442,645
Purpose: In this project, the partnership will examine the implementation, impact, and cost of all components of Career and College Promise (CCP), a set of three structured dual enrollment pathways in North Carolina (College Transfer, Career and Technical Education, and Cooperative Innovative High Schools, which includes early colleges). The partnership will build the capacity of the participating state agencies to support a cross-sector research agenda; allow for an in-depth and comprehensive response to legislative mandates; and enable a thorough investigation of long-term outcomes of CPP, including postsecondary degree attainment, employment, and earnings. Studying the CCP program will also contribute to the broader research base on the implementation and impact of dual enrollment programs and pathways.
Partnership Activities: The state agencies will participate in all aspects of the research, including design, developing instrumentation, data collection and analyses. Each state agency will support a research specialist who will have capacity development as part of their role. The partnership will embed cross-sector communication about research within existing administrative structures.
Project Activities: Using state administrative datasets, the partners will examine the impact of the dual enrollment pathways using rigorous quasi-experimental (QE) designs and will extend a prior experimental study of the impacts of the early college program by examining students' long-term outcomes. The team will also examine implementation of the pathways using multiple methods, including descriptive analyses of administrative data, surveys, and site visits. Finally, they will conduct a comprehensive cost-effectiveness analysis of each pathway.
Products: The partnership will produce estimates of the impacts of the CCP pathways on student outcomes, estimates of the cost effectiveness of CCP pathways, and a final dataset or documentation about how the data can be accessed. The partnership will also produce peer-reviewed publications, presentations, reports, and policy briefs.
Setting: The project is set in state of North Carolina, in all of the high schools and colleges there.
Sample: The study will focus on different samples depending on the research question; the largest sample will include all NC high school students who participate(d) in CCP between the 2012-13 and 2022-23 school years and a matched set of comparison students.
Intervention: The interventions are the three CCP pathway options.
Research Design and Methods: The partners will examine the impact; implementation; and cost and cost effectiveness of the CCP pathways. The team will use both a randomized controlled trial (RCT) and quasi-experimental (QE) methods using propensity score weighting, depending on the pathway and research question. The researchers will examine cost using the ingredients method, supplemented by analyses of administrative data.
Control Condition: Weighted comparison groups of eligible students who did not participate in any of the pathways will be created for each pathway group. The weights will be based on propensity scores, which represent the probability of participating in a given pathway, estimated using both student- and school-level covariates. For the subset of early colleges in the experimental study, the control group will consist of students who applied to the attend the early college but were randomly selected not to attend and enrolled in different schools.
Key Measures: Student-level covariates include grade level, multiple demographic characteristics, and baseline measures of student engagement (such as attendance, suspensions), and academic readiness (e.g., test scores at middle school end of grade exams, prior GPA). School-level covariates include school size and urbanicity, averages of student demographics and academic performance, and measures of school conditions and climate. Outcomes include college credits accrued in high school and high school graduation, as well as outcomes measured post high school such as postsecondary enrollment and degree attainment and wage and earnings.
Data Analytic Strategy: Impact analyses will account for nested data structures via hierarchical linear modeling. The researchers will also conduct moderator and mediator analyses to test their theories of change for each pathway.
Cost Analysis: The researchers will estimate the total cost of the CCP initiative at the system-level considering all cost components, including a variety of resources that support the design, delivery, monitoring and sustaining of the intervention. They will apply the principles of the "ingredients method" for cost analysis, which requires collecting information on the resources required for program implementation and their prices (or economic values) based on actual expenditures and off-budget items (e.g., donated space). Similar information on resources used and their valuation will be collected for the business-as-usual condition, so that incremental cost of the intervention will be captured. Finally, the researchers will compare the cost of the CCP intervention with the outcomes achieved as measured by the impact study, using cost-effectiveness analysis (CEA) and benefit-cost analysis (BCA, also known as cost-benefit analysis).
Related IES Projects: Impact of Early College High School (ECHS) Model on Postsecondary Performance and Completion (R305A140361)
This project is part of the CTE Research Network (R305N180005).
Project Website: https://earlycollegeresearch.uncg.edu/
Supplement: This grant received a supplemental award on 06/07/2021 to lead a 2-year cross-CTE Network study on CTE advising.
Title: Exploring the Role and Effects of High School Advising on CTE Students' Transitions to Postsecondary Education and the Workforce
Co-Principal Investigators: Edmunds, Julie, Unlu, Fatih, Rosen, Rachel, Kemple, James, Sludden, John, Mulhern, Christine
Contribution: Counselors and advisors play a key role in guiding students as they transition from high school to different postsecondary opportunities; they are particularly important for students whose families or out-of-school networks do not have the knowledge to effectively guide them. Despite its importance, the topic of advising is understudied particularly for CTE students. In general, there is little information about the role that advisors and counselors can play regarding the postsecondary transition.
Purpose: This project is a collaboration of four research organizations—SERVE Center at University of North Carolina at Greensboro, RAND Corporation, the Research Alliance for New York City Schools, and MDRC—that are participating in the CTE Research Network. The project consists of three related sub-studies that will inform the development of a conceptual framework around advising for CTE students. These three sub-studies include: 1) an exploratory study using student survey data in NYC that examines whether advising influences student outcomes; 2) a more rigorous causal impact study of a career-focused advising model in North Carolina; and 3) a qualitative exploration of advising and the role that it plays in students' decisions.
Setting: These three sub-studies will build on three existing IES-funded projects, two in New York City and one in North Carolina. Sub-Study 1 will examine survey results for seniors in New York City Schools. Sub-Study 2 will look at a career coaching program being implemented across the state of North Carolina. Sub-Study 3 will include qualitative case studies in both states, including in P-Tech 14 schools in NYC.
Sample: The primary population of interest for these three studies is CTE students. This population will be identified in three different ways: 1) students enrolled in a CTE-specific school; 2) students in non-CTE schools who have taken at least one CTE course; and 3) students in non-CTE schools who are CTE concentrators (or students who have taken at 4+ CTE courses). The sample will differ slightly for each sub-study.
Intervention: This study focuses on advising or counseling about postsecondary transitions. It will examine the school-based ways in which CTE students receive information about postsecondary transitions and the individuals who provide that information. Sub-Study 2 will focus specifically on North Carolina's Career Coaching Program, which places community college Career Coaches in high schools to help students acquire greater career awareness, develop career goals, and explore high school and postsecondary CTE program and pathway options available in the local community to help students achieve their career goals.
Research Design: The research design differs by sub-study. The first study will use correlational analyses to examine the relationship between advising and different outcomes for different populations of CTE students. The second study will use two causal impact designs 1) an event study framework to examine the school-level impact of career coaches; 2) a propensity-score weighting approach to examine the student-level impact of career coaches. The third study will develop case studies of a purposive sample of schools using interview data.
Measures and Data Sources: Sub-Study 1 will use the Senior Exit Survey administered to seniors in NYC Schools. The survey includes information around school activities to support students' postsecondary transition, including advising, as well as students' postsecondary plans. This survey is linked to school-level information collected as part of the existing IES-funded study. Sub-Study 2 will use a longitudinal state-level dataset that connects K-12 to community college and University of North Carolina system data. This dataset will be linked to student-level data being collected by North Carolina's career coaches. The third study will use qualitative interview data collected by the three project teams.
Analytic Strategies: The first sub-study will use regression analyses to explore relationships between advising and other variables. The second sub-study will use an event study analysis for the school-level impacts and a propensity-score weighting approach for the student-level analyses. The site visit data for the third sub-study will be coded both inductively and deductively. After the three sub-studies are finished, the research team members will meet to synthesize results and develop a final recommended conceptual framework.