Choice and Information: The Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance
Co-Principal Investigator: Garcia, Ivonne
Purpose: The purpose of the study is to rigorously test two popular and widely used technology-based career advising tools for secondary school students. These tools provide attractive options for schools, which often lack capacity to increase depth or breath of current career and technical education (CTE) and career counseling services, even as the current Perkins V policy environment seeks to strengthen and increase CTE choices for students. However, despite the popularity and widespread use of these kinds of tools across many schools and districts, there is limited evidence of their efficacy for helping students identify careers of interest, make CTE choices, or pursue CTE opportunities in high school. This project aims to rigorously test two examples of these tools (Naviance and YouScience) to understand whether and how they influence student thinking about career options, choice of relevant CTE coursework and work-based learning options, and decisions about CTE concentration in available pathways and programs of study.
Project Activities: The researchers will use a three-armed randomized clinical trial to evaluate the effects of two technology-based career advising tools on high school students' awareness of career choices, their career course taking patterns, and their academic performance.
Products: The researchers will produce a set of evidence describing the impact of technology-based career advising tools as used in secondary schools on student outcomes. They will also produce a cost effectiveness analysis, peer-reviewed publications and three final, de-identified, restricted-use, linkable datasets.
Setting: The study will take place in partnership with Communities in Schools (CIS). CIS is a national non-profit organization that provides enhanced student support services to over 1.5 million students, in 2,400 schools (about 25 percent of which are high schools) across more than 300 school districts in 25 states. CIS schools typically serves lower income populations and communities
Sample: A total of 70 to 90 high schools from the existing CIS network of schools will participate. The student sample for the study includes ninth-grade students who are enrolled in the study schools in Year 2 (SY 2021-22) and are identified as needing additional support by CIS case managers (on average about 80 students per school).
Interventions: YouScience is a tool that combines a set of psychometrically validated assessments (The Ball Aptitude Battery) designed to identify student innate aptitudes with an assessment of student interests in order to help identify a set of career options that capitalize on both student interest and their abilities that will likely make students successful in given fields. The Naviance college and career ready tool also includes a validated assessment of student talent and strengths. Specifically, Naviance uses the Clifton Youth Strengths Finder assessment, which identifies 10 talent "themes" that are related to specific skill sets that can be used to identify matching careers that capitalize on those talents. Though the assessments used by these two tools differ, the common underlying idea is that helping students to identify career "fit" through increasing self-awareness will lead students to better align their career choices with their own individual talents. The assessment tools are then linked to information on career knowledge and planning, which allows students to assess which career they may be best suited for.
Research Design and Methods: The study will use a three-arm school-level random assignment research design (RCT) to assess the effects of the CTE advising tools and accompanying support packages on students' self-expressed attitude and interest in career pathways, their CTE course taking patterns, and their engagement with and progress towards graduation. The researchers will randomly assign 24 schools to each of the three conditions: receiving Naviance, receiving YouScience, or the business as usual condition.
Control Condition: The business as usual schools in the study will not have access to the two advising tools for their ninth graders during the first 2 years of the evaluation. Therefore, the key cohort of students in the evaluation sample (9th graders in SY 2021-22) will not have exposure to either tools during the evaluation period. Rather, they will receive the "business as usual" services provided to them by their schools and their counselors.
Key Measures: The study will use three types of student outcomes. The confirmatory student outcome of the study will be a dichotomous indicator of whether students choose CTE courses within a given career pathway in both 10th and 11th grades. This measure will be constructed from students' transcript data. The study will also use multiple scales constructed from student surveys to capture the program impacts on proximal student outcomes such as their self-awareness and knowledge about career options and pathways. Finally, students' engagement with school will be captured by their attendance rates, and whether they are "on-track" to graduate will be measured by their grade progression and credit accumulation towards graduation.
Data Analytic Strategy: The impacts of the interventions will be assessed by a two-level hierarchical model that will account for the clustered nature of the study design, with students (Level 1) nested in schools (Level 2). To improve the precision of the impact estimate, the Level 1 model will also control for a set of student baseline characteristics including students' eighth-grade reading and math state test scores, their self-reported opinions about career options and CTE pathways from the baseline survey, and their demographic characteristics. At Level 2, indicators for the random assignment blocks will be included to improve precision and to account for the structure of the study design. Similar models will be used to assess impacts for student and school subgroups. The study also plans to use two-stage least squares analysis to examine how student exposure to the tools mediate the program effects, and use a modified version of a random effects meta-analysis model to examine the relationship between implementation features and program impacts.
Cost Analysis: The study will use the ingredients approach to estimate costs. This method first identifies all the "ingredients" necessary to implement the interventions, and then monetizes them using a variety of both quantitative and qualitative data sources. The ingredients are summed and averaged across the student sample in order to obtain the average per pupil costs of the intervention.