Project Activities
The project will take place in two phases. During Phase 1 (development and pilot), the researchers will design vSchool and collect qualitative (e.g., semi-structured focus groups) and quantitative (e.g., game log) data for iterative improvement. During Phase 2, the researchers will field test vSchool in up to 18 schools and investigate its psychometric properties.
Structured Abstract
Setting
This study will take place in Missouri, with a focus on Title 1, rural schools.
Sample
This study will include over 1,000 4th to 6th grade students. The Phase 1 small pilot and the Phase 2 field test will involve 4th to 6th grade classrooms in 18 schools that are part of an ongoing SEL intervention.
vSchool will be a single-player game-based assessment. Students will build their own avatar to move through a virtual school-like setting. There will be about six scenes with other characters to interact with using menu-driven choices for prosocial and non-prosocial behaviors that are common in schools. Scores will be based on the player's choices to behave prosocially. Game log data will generate scoring algorithms. vSchool will be created using the Unity Game Engine and will be available to play online through a web browser. It will be designed to be delivered without teacher support.
Research design and methods
The project will take place in two phases using a mixed methods approach. During Phase 1 (development and pilot), the researchers will design vSchool and collect qualitative (e.g., semi-structured focus groups) and quantitative (e.g., game log) data for iterative improvement. In Year 1, the researchers will use Evidence Centered Game Design to create vSchool. In Years 2-3, the researchers will iteratively improve vSchool using data that includes user experience (UX) testing, feedback from experts, student focus groups, and an external usability study. A small pilot study in two schools will be conducted to ascertain feasibility in schools. The pilot will include a brief survey asking students about their perceptions of vSchool. Phase 2 focuses on examining the reliability, validity, and amenability to intervention of vSchool. In Year 4, the researchers will conduct a field test of vSchool in 18 schools that are part of an on-going ED-funded SEL intervention.
Control condition
As part of the field test, schools will administer vSchool to adjacent grades (4th and 6th) who are not receiving the SEL intervention. This will allow the researchers to explore whether vSchool scores are amenable to intervention.
Key measures
Validity will be explored with teacher report of prosocial behavior (Child Behavior Scale) as well as three constructs that research and theory suggest prosocial behavior predicts: (1) student report of classroom engagement (Classroom Engagement Inventory) and (2) teacher-student relationship quality (TSRQ), and (3) students' academic achievement as indicated by proficiency test scores.
Data analytic strategy
In Phase 1 (development and pilot), qualitative data from cognitive interviews and focus groups will be analyzed using the constant comparative method to identify patterns. Each month, the team will review data in order to identify needed revisions. In Phase 2 (field test), once the field test data meet the assumptions of IRT, the researchers will calibrate the difficulty and discrimination parameters of each item. Differential item functioning analysis will be used to confirm that the test is fair across groups. The research team will establish reliability via coefficient omega and IRT information functions. They will use field test data to assess the validity of vSchool. The team will use auxiliary variables (e.g., classroom, school, and district) in each of the analyses in order to account for multilevel data. They will account for selection bias by estimating a linear regression analysis with covariates.
Cost analysis strategy
The researchers will use the "resource cost method" which entails itemizing resources needed to deploy vSchool, calculating (or estimating) their costs, and then aggregating the costs to estimate overall program costs per student (for small, medium, and large districts).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The primary product is a gamified measure of prosocial behavior that schools can use to evaluate prosocial outcomes of SEL interventions in 4th to 6th grade students. The research team will explore and publish evidence of vSchool's reliability and validity.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
