Co-Principal Investigator: Bonifay, Wes

Purpose: The purpose of this study is to develop and validate a school-friendly, game-based assessment of prosocial behavior for 4th to 6th grade students that can be used to evaluate behavioral and social-emotional learning (SEL) interventions. Prosocial behavior is voluntary behavior intended to benefit others (e.g., helping, encouraging, compassion, sharing) and is foundational to a positive school climate. Increased prosocial behavior predicts increased student achievement and well-being. These effects are strongest for at-risk, low-income students.

SEL interventions often aim to increase prosocial behavior, yet few SEL interventions actually measure increased prosocial behavior as an outcome. This is partly due to a dearth of measures of prosocial behavior that are validated, school-friendly, strengths-based (rather than deficit-based), and amenable to intervention. Thus, determining the effectiveness of many SEL programs is hampered by schools' inability to measure prosocial outcomes. The researchers will address this need by developing a game-based assessment of prosocial behavior called vSchool.

Project Activities: The project will take place in two phases. During Phase 1 (development and pilot), the researchers will design vSchool and collect qualitative (e.g., semi-structured focus groups) and quantitative (e.g., game log) data for iterative improvement. During Phase 2, the researchers will field test vSchool in up to 18 schools and investigate its psychometric properties.

Products: The primary product is a gamified measure of prosocial behavior that schools can  use to evaluate prosocial outcomes of SEL interventions in 4th to 6th grade students.  The research team will explore and publish evidence of vSchool's reliability and validity.

Structured Abstract

Setting: This study will take place in Missouri, with a focus on Title 1, rural schools.

Sample: This study will include over 1,000 4th to 6th grade students. The Phase 1 small pilot and the Phase 2 field test will involve 4th to 6th grade classrooms in 18 schools that are part of an ongoing SEL intervention.

Assessment: vSchool will be a single-player game-based assessment. Students will build their own avatar to move through a virtual school-like setting. There will be about six scenes with other characters to interact with using menu-driven choices for prosocial and non-prosocial behaviors that are common in schools. Scores will be based on the player's choices to behave prosocially. Game log data will generate scoring algorithms. vSchool will be created using the Unity Game Engine and will be available to play online through a web browser. It will be designed to be delivered without teacher support.

Research Design and Methods: The project will take place in two phases using a mixed methods approach. During Phase 1 (development and pilot), the researchers will design vSchool and collect qualitative (e.g., semi-structured focus groups) and quantitative (e.g., game log) data for iterative improvement. In Year 1, the researchers will use Evidence Centered Game Design to create vSchool. In Years 2-3, the researchers will iteratively improve vSchool using data that includes user experience (UX) testing, feedback from experts, student focus groups, and an external usability study. A small pilot study in two schools will be conducted to ascertain feasibility in schools. The pilot will include a brief survey asking students about their perceptions of vSchool. Phase 2 focuses on examining the reliability, validity, and amenability to intervention of vSchool. In Year 4, the researchers will conduct a field test of vSchool in 18 schools that are part of an on-going ED-funded SEL intervention.

Control Condition: As part of the field test, schools will administer vSchool to adjacent grades (4th and 6th) who are not receiving the SEL intervention. This will allow the researchers to explore whether vSchool scores are amenable to intervention.

Key Measures: Validity will be explored with teacher report of prosocial behavior (Child Behavior Scale) as well as three constructs that research and theory suggest prosocial behavior predicts: (1) student report of classroom engagement (Classroom Engagement Inventory) and (2) teacher-student relationship quality (TSRQ), and (3) students' academic achievement as indicated by proficiency test scores.

Data Analytic Strategy: In Phase 1 (development and pilot), qualitative data from cognitive interviews and focus groups will be analyzed using the constant comparative method to identify patterns. Each month, the team will review data in order to identify needed revisions. In Phase 2 (field test), once the field test data meet the assumptions of IRT, the researchers will calibrate the difficulty and discrimination parameters of each item. Differential item functioning analysis will be used to confirm that the test is fair across groups. The research team will establish reliability via coefficient omega and IRT information functions. They will use field test data to assess the validity of vSchool. The team will use auxiliary variables (e.g., classroom, school, and district) in each of the analyses in order to account for multilevel data. They will account for selection bias by estimating a linear regression analysis with covariates.

Cost Analysis: The researchers will use the "resource cost method" which entails itemizing resources needed to deploy vSchool, calculating (or estimating) their costs, and then aggregating the costs to estimate overall program costs per student (for small, medium, and large districts).

Related IES Projects: An Interactive Social Tutoring System to Improve and Measure Social Goals for Students Related to Academic and Other School-Related Outcomes — Phase I (EDIES10P0114) and An Interactive Social Tutoring System to Improve and Measure Social Goals for Students Related to Academic and Other School-Related Outcomes — Phase II (EDIES11C0039). Testing the Efficacy of Zoo U for Improving Children's Social and Emotional Outcomes (R305A180050); Developing a 3D-based Virtual Learning Environment for Use in Schools to Enhance the Social Competence of Youth with Autism Spectrum Disorder (R324A090197)