Co-Principal Investigators: Castleman, Benjamin; Tessler, Betsy; Finnegan, Catherine

Purpose: This project will explore malleable factors within postsecondary noncredit career technical education (CTE) programs that may influence students' academic and workforce outcomes. Despite the increasing role of noncredit CTE programs in workforce development, few research studies have addressed these programs, and very little is known about how to effectively support students enrolled in short-term CTE programs. Building on a long-standing partnership with the Virginia Community College System (VCCS) and its 23 institutions, this project aims to systematically explore noncredit CTE programs at community colleges. Researchers will leverage both statewide student-level and program-level administrative data, as well as detailed data on program features collected through surveys of administrators in a sample of the top 10 most highly enrolled noncredit CTE programs across VCCS institutions. This study will generate evidence about the effects of noncredit CTE programs for all students and key subgroups and will document the varying approaches used by programs to design and deliver training. This evidence will inform future policies and interventions to improve success in noncredit CTE programs in broad-access institutions. This project will also support VCCS to build new data about noncredit education activities into its current statewide data collection efforts.

Project Activities: Researchers will begin by collecting VCCS administrative data on student enrollment and credential attainment in noncredit CTE programs, their progression into other noncredit and for-credit programs within VCCS, and programmatic characteristics of noncredit programs. They will merge the college administrative data with college attainment data from the National Student Clearinghouse (NSC) and individual quarterly employment and earnings data from the unemployment insurance (UI) database maintained by the Virginia Employment Commission. Researchers will also collect more detailed program data that are not readily available in the administrative data through a survey of administrators in the top 10 most highly enrolled programs at VCCS. Drawing on these different sources of data, the project will start with three preliminary inquiries that intend to delineate "basic facts" about noncredit CTE programs: 1) description of enrollment, participation, and academic outcomes for all students and subgroups in noncredit CTE programs at VCCS; 2) description of key program design and implementation factors for noncredit CTE programs, and their variation across programs and colleges, along with stakeholders' perceptions of these program features; and 3) a quasi-experimental analysis to estimate the labor market returns to noncredit CTE programs. Drawing on the "basic facts" garnered through the three preliminary inquiries, the research team will link these multiple sources of data together to estimate the relationship between malleable noncredit CTE program factors and student academic and labor market outcomes. In addition to the quantitative analyses, the team will also conduct interviews with program stakeholders to provide insight into the policy and institutional conditions that underlie variation in availability of services, instructional models, and employer engagement with specific programs. Student focus groups will also provide insight into how students experience delivery of program features.

Products: The team will disseminate findings regularly to administrators at VCCS and to national audiences. Working through leadership meetings at VCCS, the CTE Center housed at MDRC, and national conferences, the team will share products including policy briefs, conference presentations, and practitioner toolkits targeting audiences both within VCCS and nationwide. The team will also generate peer-reviewed publications.

Structured Abstract

Setting: The project will take place within the Virginia Community College System.

Sample: Researchers will employ two analytic samples. The first one, a statewide sample, will comprise the full set of noncredit CTE programs and all students enrolled in them from academic years 2017-2018 through 2023-2024. Given an average annual enrollment of over 7,000 students, this sample will include a total of approximately 49,000 students. To provide a more nuanced understanding of program design features, the second "deep dive" sample will include the 10 largest noncredit CTE programs at VCCS during the 2023-24 academic year and the students enrolled in them. Researchers expect to administer surveys to approximately 150 administrators of the top 10 programs distributed across colleges within the VCCS system and follow a sample of approximately 5,000 students enrolled in these programs.

Factors: Building on the existing literature on supporting community college students, researchers will focus on five domains of program factors: instructional characteristics; academic advising and non-academic support services; financial aid; career services; and employer relationships and work-based learning opportunities.

Research Design and Methods: To delineate "basic facts" about noncredit CTE programs, the research team will start with a series of descriptive analyses to describe the characteristics of students enrolled in noncredit CTE programs and their typical outcomes, as well as the variation in the five domains of program features across programs. The team will also conduct quasi-experimental analyses to investigate the impact that noncredit programs have on students' academic and labor market outcomes, for all students and key subgroups. Drawing on the "basic facts" about noncredit CTE programs, the main analyses will use multilevel modeling that combines program fixed effects with student random effects to identify program factors that are predictive of student outcomes and are malleable by programs or institutions. Researchers will also use interview data to delineate the varying approaches used by programs to design and deliver noncredit CTE programs.

Control Condition: The descriptive analyses do not include a control condition. The quasi-experimental analysis will draw on variations in program factors across different training institutions that offer the same program. Researchers will compare student success outcomes in training institutions with high program factors to student outcomes in institutions with low program factors. Accordingly, programs with weak engagement of program factors will serve as the comparison group.

Key Measures: Researchers will explore three sets of outcomes: 1) noncredit CTE program completion and credential attainment using the VCCS administrative data; 2) subsequent academic progression metrics in for-credit programs using a combination of the VCCS and NSC data, and 3) employment and earnings in the labor market using the UI data. To measure program factors, researchers will construct dichotomous and continuous measures of instructional characteristics and financial aid from the VCCS administrative data, as well as measures of advising, non-academic, and career support services from administrator surveys. From administrator interviews, researchers will code measures of the operating channels (mechanisms) through which specific program factors may affect student outcomes—such as student program engagement and access to services. Such measures may include, for example, presence or absence of cohort structure, frequency of contact between students and coaches/case managers and career services staff, and number and quality of service referral partners. The team will also explore student characteristics that may moderate impacts on student outcomes.

Data Analytic Strategy: Researchers will use descriptive analyses of key student outcomes and program features, including the extent of variation in program outcomes by student characteristics, occupational fields, and institutional features; a comparative individual fixed effect model to estimate the labor market returns to noncredit CTE; and a model that combines program fixed effects with student random effects to explore the relationship between program factors and student outcomes.

In addition, researchers will use deductive coding to code and analyze interview data collected from program administrators, student service staff and providers, and employers. Researchers will start with an established codebook and reorganize it as the coding proceeds. To achieve a deep understanding regarding program implementation, researchers will look for themes across the coded data that address such questions as how the offer of program services, access to the program, and experiences in the program vary by student characteristics, with a focus on historically marginalized students.