Co-Principal Investigators: D'Mello, Sidney; Baker, Ryan S.

Purpose: The purpose of this project is to research how to design educational games that integrate assessment and learning to promote both STEM competency development and interest. Preparing students to succeed in the 21st century requires fresh thinking on how to cultivate STEM-related interest and competencies. A potential path to meeting these goals involves using STEM learning games to engage students and enhance learning, while seamlessly assessing STEM competencies. Identifying the features make games effective at increasing STEM interest and competencies, and understanding the reasons for the effectiveness, can inform the development of theory and design of highly effective learning games.

Project Activities: The researchers will study theoretically-guided learning supports and elements of design as malleable factors that can improve both the learning experience and learning outcomes in STEM learning games. The research goals are to enhance understanding of the types of cognitive and affective supports that promote formal STEM learning, enhance science interest, and improve the learning experience. The researchers also plan to test whether there are advantages to combining supports; whether supports are more effective if controlled by the student or the game; whether there is value added to tailoring supports based on sophisticated modeling of student affect; and the factors that mediate the influence of supports on the play experience and learning outcomes.

Products: The team will produce published research findings that will inform the research and development of learning games and assessments while contributing to theory on effective learning game design.

Structured Abstract

Setting: The research will take place in one medium sized public school in Florida and in a New York City charter school.

Sample: The research sample includes 675 grade 7 to 9 students, who are racially, ethnically, and economically diverse.

Intervention: The researchers will investigate the learning supports and elements of design within Physics Playground, a learning game that supports informal understanding of Newtonian physics and which prior research shows to be highly engaging. The team will leverage the game, its stealth assessments of learning, and its automated affect detectors to explore the effects of cognitive and affective learning supports that offer praise and just-in-time instruction when students succeed and provide encouragement and instructional scaffolding when students struggle.

Control Condition: In Experiment 1, the researchers will compare the cognitive and affective supports to a no-support control condition. In each of the three subsequent experiments, the control condition will be the best condition from the previous year's experiment.

Research Design and Methods: The project includes four between-subject experiments (with random assignment at the student level), each involving two hours of gameplay across four 50-minute sessions. In Experiment 1, the researchers will investigate the effects of theoretically-grounded cognitive and affective supports (two treatment conditions) compared with a no-support control condition. In Experiment 2, the researchers will examine the benefits of combining supports (cognitive and affective) compared to the most effective individual support from Experiment 1. In Experiment 3, the researchers will investigate the relative effectiveness of learner-controlled compared to game-controlled supports, using the most effective supports from the previous years. Finally, in Experiment 4, the researchers will examine the added value of tailoring supports based on automated assessments of confusion and frustration.

Key Measures: Outcome measures, used throughout the four years of the project, include learning of physics principles (from external assessments), interest in science, and subjective perceptions of the play experience.

Data Analytic Strategy: The researchers will use multiple linear regression to predict outcome variables after controlling for covariates (e.g., prior knowledge, school affiliation, grade, gender, ethnicity, and prior gaming experience). The research team will test mediation using the Preacher and Hayes bootstrap procedure.

Products and Publications

Conference Proceeding

Karumbaiah, S., Baker, R.S., and Shute, V. (2018). Predicting Quitting in Students Playing a Learning Game. International Educational Data Mining Society.

Journal article, monograph, or newsletter

Brom, C., Stárková, T., and D'Mello, S.K. (2018). How Effective is Emotional Design? A Meta-Analysis on Facial Anthropomorphisms and Pleasant Colors During Multimedia Learning. Educational Research Review, 25, 100–119.

Spann, C.A., Shute, V.J., Rahimi, S., and D'Mello, S.K. (2019). The Productive Role of Cognitive Reappraisal in Regulating Affect During Game-Based Learning. Computers in Human Behavior.