|Title:||An Experimental Evaluation of the Efficacy of Virtual Enterprises|
|Principal Investigator:||Unlu, Fatih||Awardee:||RAND Corporation|
|Program:||Career and Technical Education [Program Details]|
|Award Period:||5 years (07/01/2021 - 06/30/2026)||Award Amount:||$3,799,857|
|Type:||Initial Efficacy||Award Number:||R305A210198|
Co-Principal Investigators: Baird, Matthew; Daugherty, Lindsay; Hughes, Katherine
Purpose: For this project, the research team will provide the first causal evidence on the efficacy and cost-effectiveness of Virtual Enterprises (VE), a virtual SBE program. Many high schools are struggling to ensure that all graduates are prepared for college and the workforce, and employers consistently report difficulty in finding workers with sufficient employability skills, such as teamwork and communication. A growing number of education practitioners and researchers have argued that incorporating career and technical education (CTE), including work-based learning (WBL), into high school curricula can improve students' engagement and their readiness for postsecondary education and the workplace. School-based enterprises (SBE) are one form of WBL, in which students run a business that produces and sells goods or services, and may offer unique benefits relative to other types of WBL by providing opportunities for more students to participate, reducing the need for transportation, and allowing students more room to make, and learn from, their mistakes. However, there is very little causal evidence showing the impacts of CTE or WBL on student academic or employment outcomes, and none showing the impacts of SBE. VE has been adopted by over 430 schools across 18 states, and prior implementation studies indicate that it has the potential to increase student motivation and employability skills.
Project Activities: The research team will conduct a randomized, controlled trial (RCT) including districts in three states and approximately 40 high schools that already have an oversubscribed VE program . Eligible students from two cohorts (school years 2022-23 and 2023-24) will be randomized into the VE program. The researchers will use a combination of administrative data, assessments, and surveys during and after the intervention to compare academic and employment outcomes for eligible students who were assigned to the VE program and those who were not.
Products: The research team will produce a report documenting findings and at least three published journal articles. The research team will disseminate shorter products for education practitioners, policymakers, and CTE and workforce development organizations. These products will be shared via conferences and social media. The research team will also present results to participating schools.
Setting: The study will take place in high schools in New York, Florida, and California. The participating schools come from four urban and suburban counties.
Sample: Approximately 6,800 students in the 11th and 12th grades will be part of the RCT. The student demographics in the targeted schools range from 1 to 63 percent African American, from 4 to 76 percent Hispanic, and from 1 to 74 percent white. Between 7 and 90 percent of students are eligible for free or reduced-price lunch.
Intervention: VE is a year-long class in which students run a virtual firm. Individual students assume specific roles, such as CEO, marketing manager, or human resources employee. Together, the students develop and implement a strategy for the firm and participate in a virtual economy with other VE firms. The curriculum is provided by Virtual Enterprises International (VEI), a non-profit organization that also provides training for the VE teachers (known as facilitators), provides virtual start-up capital to the firms, and runs the online marketplace in which firms sell their goods or services. Treatment students will be assigned to the VE class, which has already been adopted in all participating schools. Treatment students will also have the option to take advantage of several other opportunities provided by VEI, including participating in business plan competitions at which they pitch their ideas to business professionals, and attending workshops on leadership and business skills.
Research Design and Methods: In this project, the research team will use a randomized control trial (RCT) to randomize the intervention within schools at the student level. All participating schools currently have VE programs that are oversubscribed; students who are deemed eligible for the VE program at each school during the course selection period will be randomized into the treatment group (being placed in the VE class) or the control group (being placed in another class). Researchers will measure the main outcomes using administrative district and National Student Clearinghouse data. The research team will also conduct student baseline (at the beginning of the intervention school year) and endline (at the end of the intervention school year) surveys, assessments of students' leadership and professionalism skills (and the end of the intervention school year), and follow-up surveys (the following year). They will rely on site visits, teacher surveys, and student focus groups to assess fidelity of implementation.
Control Condition: Control students will not take the VE class, but the exact nature of the business-as-usual counterfactual may vary depending on the alternative courses available at the school. The researchers will identify the control condition in each school based on discussions with school administrators, and will use student surveys to identify whether control students engage in any activities (e.g. business plan competitions) similar to those offered to VE students.
Key Measures: Confirmatory outcome measures will be senior year GPA, attendance, college credits earned in high school, and postsecondary enrollment. The researchers will also explore several exploratory outcomes, including completion of CTE courses after high school, personal financial management, and employment and earnings outcomes. Mediators include college and career readiness, academic motivation, financial literary, leadership, and professionalism. Moderators will include pre-program college and career readiness and academic motivation, position held in the VE firm, fidelity of implementation, and student demographics.
Data Analytic Strategy: To analyze the experimental data, the research team will use linear regression methods that account for time-invariant and time-varying student characteristics. The research team will also estimate a model that examines the effect of treatment on the treated. In addition, they will use a Cross Design Synthesis estimator that combines experimental data from the students in the RCT with observational data from all schools in the participating districts that have VE programs, to provide a bias-corrected population-wide estimate of the treatment effect. The team will assess mediators through structural equation modeling and will examine moderation by interacting treatment status with each moderator. To identify implementation fidelity and challenges to implementation, they will systematically code and analyze interview and focus group data using a coding structure based on the adherence constructs and variables.
Cost Analysis: The research team will use the "ingredients method," summing up the costs of all "ingredients" required for implementation of the VE program. They will group costs into four categories: program payments to VEI, personnel costs, materials, and facilities. They will calculate the cost-effectiveness ratio (CER) by dividing the per-student cost by the estimated effect sizes.