Co-Principal Investigators: Flynn, Kylie; DeRosier, Melissa

Purpose: The purpose of this study was to evaluate the Adventures Aboard the S.S. GRIN (Adventures) program, an online game with embedded social skills training to improve social, emotional, and academic outcomes for children experiencing social difficulties. Prior research indicated that direct social skills instruction can benefit these students, yet most schools are not equipped to provide this type of targeted support. The Adventures program was created through an IES development grant (R305A110583) to address this gap.

Project Activities: The research team randomly assigned two cohorts of third-grade teachers to provide students with the opportunity to play the Adventures game (treatment) or to continue with their typical practice (control) over an 11-week period during the school year. The researchers analyzed outcomes for all students and for those identified by their teachers as having social skills difficulties. They piloted measures and study procedures in year 1 to prepare for the full randomized control trial in years 2 and 3.

Pre-registration site: https://sreereg.icpsr.umich.edu/sreereg/subEntry/21021/pdf?action=view

Key Outcomes: The main findings of this study are as follows:

• All of the students who played the Adventures game showed positive and statistically significant improvements in several social and emotional skills compared to students who did not play the game (Li. et al., 2021).
• Teachers reported high levels of student engagement with Adventures and positive user experiences (Li., et al., 2021).

Structured Abstract

Setting: This study took place in four primarily urban and suburban school districts in California.

Sample: The sample for this study included 88 third-grade teachers and their students (n = 1,645) across 37 schools.

Intervention: Adventures Aboard the S.S. GRIN is an online game developed with IES funding that incorporates the content and instructional strategies of a widely used, efficacious, small-group program (Social Skills Group Intervention - S.S. GRIN). The Adventures program provides schools with a more feasible way to support students experiencing social difficulties like peer rejection, bullying, and social anxiety. The game consists of 9 episodes (about 25 minutes each) that use a single-player format and intelligent social tutoring software (ISTS) to provide individualized feedback and a safe environment for practicing newly learned skills to regulate emotions, control impulses, build friendships, and respect self and others. The ISTS software captures in-game behaviors to assess the student's progress and adapt responses by the ISTS to generate a next step individualized for that student.

Research Design and Methods: The researchers used a teacher-level, delayed-treatment, cluster randomized controlled experimental design, where teachers were randomly assigned to treatment or control groups. Teachers randomly assigned to the treatment group provided opportunities for all of the students in their classroom to play the Adventures game over an 11-week period during the school year (approximately 1 episode per week). The researchers piloted measures and study procedures in 5 schools in year 1 to prepare for the full randomized control trial in years 2 and 3.

Control Condition: Students in the control condition engaged in business-as-usual activities in their classrooms.

Key Measures: Teachers completed the Devereux Student Strengths Assessment (DESSA)-Mini to identify third-grade students with social skills deficits. Student outcomes were assessed using teacher ratings on the Behavioral and Emotional Rating Scale Second Edition (BERS-2) and through direct assessment using ZooU (EDIES11C0039) and SELweb (R305A110143). The researchers measured implementation fidelity through weekly electronic implementation logs, the Adventures' analytics, site visits, and interviews. The researchers had planned to use the Smarter Balanced Summative Assessments to measure student academic performance but this was not possible as the state did not administer these assessments in the 2019–2020 (cohort 1) and 2020-2021 (cohort 2) school years when school were closed due to the COVID-19 pandemic.

Data Analytic Strategy: The research team analyzed differential gains between treatment and control groups on measures of student outcomes using two-level hierarchical linear models. They used structural equation modeling to analyze the impact of potential mediator (students' perceived self-efficacy for social situations) and moderator (implementation) variables.

Cost and Cost-Effectiveness Analyses: Researchers used the "ingredients method" to estimate program costs for schools to implement the intervention and used the average cost per student to calculate a cost-effectiveness ratio for the Adventures program. For 42 treatment teachers with a total of 744 students, the estimated total cost for implementing Adventures is $15,748.08, and the average cost per student is $21.17. The cost-effectiveness ratio is $34.74.

Related IES Project: Interactive Social Tutoring System for Social Skills Training with Elementary Students (R305A110583)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Publicly Available Data: Interested individuals can contact the research team.

Select Publications:

Flynn, K.S, Li, L., Huang, CW, Patel, R., Luttgen, K., Yang, S., & Chow, E. (under review). Leveraging Technology to Address Social-Emotional Learning during the Pandemic: Findings from an Efficacy Trial. Social and Emotional Learning: Research, Practice, and Policy.

Li, L., Flynn, K.S., DeRosier, M.E., Weiser, G., & Austin-King, K. (2021). Social-Emotional Learning Amidst COVID-19 School Closures: Positive Findings from an Efficacy Study. Frontiers in Education, (6), 683142. DOI:10.3389/feduc.2021.683142.