Project Activities
The research team randomly assigned two cohorts of third-grade teachers to provide students with the opportunity to play the Adventures game (treatment) or to continue with their typical practice (control) over an 11-week period during the school year. The researchers analyzed outcomes for all students and for those identified by their teachers as having social skills difficulties. They piloted measures and study procedures in year 1 to prepare for the full randomized control trial in years 2 and 3.
Structured Abstract
Setting
This study took place in four primarily urban and suburban school districts in California.
Sample
The sample for this study included 88 third-grade teachers and their students (n = 1,645) across 37 schools.
Intervention
Adventures Aboard the S.S. GRIN is an online game
developed with IES funding that incorporates the content and instructional strategies of a widely used, efficacious, small-group program (
Social Skills Group Intervention - S.S. GRIN). The
Adventures program provides schools with a more feasible way to support students experiencing social difficulties like peer rejection, bullying, and social anxiety. The game consists of 9 episodes (about 25 minutes each) that use a single-player format and intelligent social tutoring software (ISTS) to provide individualized feedback and a safe environment for practicing newly learned skills to regulate emotions, control impulses, build friendships, and respect self and others. The ISTS software captures in-game behaviors to assess the student's progress and adapt responses by the ISTS to generate a next step individualized for that student.
Research design and methods
The researchers used a teacher-level, delayed-treatment, cluster randomized controlled experimental design, where teachers were randomly assigned to treatment or control groups. Teachers randomly assigned to the treatment group provided opportunities for all of the students in their classroom to play the Adventures game over an 11-week period during the school year (approximately 1 episode per week). The researchers piloted measures and study procedures in 5 schools in year 1 to prepare for the full randomized control trial in years 2 and 3.
Control condition
Students in the control condition engaged in business-as-usual activities in their classrooms.
Key measures
Teachers completed the Devereux Student Strengths Assessment (DESSA)-Mini to identify third-grade students with social skills deficits. Student outcomes were assessed using teacher ratings on the Behavioral and Emotional Rating Scale Second Edition (BERS-2) and through direct assessment using ZooU (
EDIES11C0039) and SELweb (
R305A110143). The researchers measured implementation fidelity through weekly electronic implementation logs, the Adventures' analytics, site visits, and interviews. The researchers had planned to use the Smarter Balanced Summative Assessments to measure student academic performance but this was not possible as the state did not administer these assessments in the 2019–2020 (cohort 1) and 2020-2021 (cohort 2) school years when school were closed due to the COVID-19 pandemic.
Data analytic strategy
The research team analyzed differential gains between treatment and control groups on measures of student outcomes using two-level hierarchical linear models. They used structural equation modeling to analyze the impact of potential mediator (students' perceived self-efficacy for social situations) and moderator (implementation) variables.
Key outcomes
The main findings of this study are as follows:
People and institutions involved
IES program contact(s)
Products and publications
ERIC Citations: Find available citations in ERIC for this award here.
Publicly Available Data: Interested individuals can contact the research team.
Select Publications:
Flynn, K.S, Li, L., Huang, CW, Patel, R., Luttgen, K., Yang, S., & Chow, E. (under review). Leveraging Technology to Address Social-Emotional Learning during the Pandemic: Findings from an Efficacy Trial. Social and Emotional Learning: Research, Practice, and Policy.
Li, L., Flynn, K.S., DeRosier, M.E., Weiser, G., & Austin-King, K. (2021). Social-Emotional Learning Amidst COVID-19 School Closures: Positive Findings from an Efficacy Study. Frontiers in Education, (6), 683142. DOI:10.3389/feduc.2021.683142.
Related projects
Supplemental information
Co-Principal Investigators: Flynn, Kylie; DeRosier, Melissa
Pre-registration site: https://sreereg.icpsr.umich.edu/sreereg/subEntry/21021/pdf?action=view
- All of the students who played the Adventures game showed positive and statistically significant improvements in several social and emotional skills compared to students who did not play the game (Li. et al., 2021).
- Teachers reported high levels of student engagement with Adventures and positive user experiences (Li., et al., 2021).
Cost and Cost-Effectiveness Analyses: Researchers used the "ingredients method" to estimate program costs for schools to implement the intervention and used the average cost per student to calculate a cost-effectiveness ratio for the Adventures program. For 42 treatment teachers with a total of 744 students, the estimated total cost for implementing Adventures is $15,748.08, and the average cost per student is $21.17. The cost-effectiveness ratio is $34.74.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.