Co-Principal Investigator: Tanner Jackson

Purpose: In this project, the researchers will systematically explore how individual differences among students (e.g., motivation, engagement, persistence, prior knowledge) interact with educational game designs and integration of those games into classrooms. Researchers will use components of an existing game-based intelligent tutoring system, Interactive Strategy Training for Active Reading and Thinking-Motivationally Enhanced (iSTART-ME). The goal of the research is to improve the design of learning environments that maintain the attention of struggling students as they learn and master comprehension skills over extended periods of time.

Project Activities: The team will carry out a series of experiments and explore and manipulate several malleable factors of system design to better understand when games are appropriate, which game features provide optimal benefits (and for whom), and how these features should be aligned with pedagogy to improve motivation, engagement, persistence, and learning within classrooms. The proposed project includes five laboratory experiments and one classroom-based study. In Years 1 and 2, researchers will explore when, which, and how malleable factors related to game and ITS design may interact with individual differences to affect motivation and learning. In Year 3, researchers will conduct a classroom-based study to observe the use of iSTART-ME in schools. This study will gather information about the behaviors of and outcomes for teachers and students during long-term usage and integration of the full system within an educational setting.

Products: The products of this project will be information about which components and design features of the iSTART-ME system are best suited to support the adaptive use of this ITS in classroom contexts. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: This project will take place in the Phoenix, AZ, metropolitan and surrounding area.

Sample: Participants will include 9th-12th- grade high school students.

Intervention: This project will use a current ITS called Interactive Strategy Training for Active Reading and Thinking-Motivationally Enhanced (iSTART-ME) to explore a range of potentially malleable factors that could be used to improve student outcomes as they interact with the system. iSTART-ME is a modular system and is comprised of a suite of educational games and game elements that are designed to improve students' comprehension skills and motivation to practice.

Research and Design Methods: Researchers will conduct a series of five targeted experiments in Years 1 and 2 that utilize all or parts of the iSTART-ME system. The targeted experiments will systematically explore the effects of different types of game elements, and how such elements can be combined with pedagogical practices to yield effective and motivating learning environments. Follow-up studies will be conducted to isolate particular effects and address alternative explanations. The targeted experiments will lead to a classroom-based study in Year 3 in which researchers will observe the use of iSTART-ME in schools. During the ecological study, researchers will collect qualitative and observational data to help understand the implementation of iSTART-ME within the real-world constraints and needs of the classroom. Researchers will use a variety of methodologies, including teacher focus groups, student focus groups, and classroom observations.

Control Condition: Studies 1-4 have no control condition, as comparisons will be made among different treatment conditions. Study 5 examines the possible added-value of game features, includes a no-game control condition, where users will "drag and drop" their responses (mimicking game-play mechanics) and receive accuracy feedback; however, game-like features e.g., points, levels) will not be included.

Key Measures: Assessments will examine the use of comprehension strategies, text comprehension, and perceptions of the system. Importantly, individual differences (e.g., prior reading ability, prior science knowledge) will also be evaluated and factored into analyses (e.g., as moderating variables). Questionnaires will be administered regarding technology use and attitudes, entertainment media preferences, and gameplay experience, as well as observations conducted in classrooms.

Data Analytic Strategy: Data from the five proposed studies will be analyzed using a variety of statistical analyses, depending on the experimental design. Tests include uni- and multivariate models, correlations, regressions, structural equation modeling, and hierarchical linear modeling.

Products and Publications

Book chapter

Johnson, A.M., Jacovina, M.E., Russell, D.G., and Soto, C.M. (2017). Challenges and Solutions When Using Technologies in the Classroom. Adaptive Educational Technologies for Literacy Instruction. Taylor & Francis eBooks.

Crossley, S.A. and McNamara, D.S. (2017). Educational Technologies and Literacy Development. Adaptive Educational Technologies for Literacy Instruction. Taylor & Francis eBooks.

Jacovina, M. E., and McNamara, D. S. (in press). Intelligent Tutoring Systems for Literacy: Existing Technologies and Continuing Challenges. Intelligent Tutoring+Systems: Structure, Applications and Challenges.

McNamara, D. S., Allen, L. K., Crossley, S. A., Dascalu, M., and Perret, C. A. (2017). Natural Language Processing and Learning Analytics. Handbook of Learning Analytics and Educational Data Mining.

Snow, E.L., Jacovina, M.E., Jackson, G.T., and McNamara, D.S. (2017). iSTART-2: A Reading Comprehension and Strategy Instruction Tutor. Adaptive Educational Technologies for Literacy Instruction. Jul 2016: Taylor & Francis eBooks.

Stone, M., Kent, K., Roscoe, R., Corley, K.M., Allen, L.K., and McNamara, D.S. (2017). The Design Implementation Framework: Iterative Design from the Lab to the Classroom. End-User Considerations in Educational Technology Design.

Journal article, monograph, or newsletter

Higgs, K., Magliano, J.P., Vidal-Abarca, E., Martinez, T., and McNamara, D.S. (2015). Bridging Skill and Task-Oriented Reading. Discourse Processes, 54(1): 19–39.

Jackson, T.G., Boonthum-Denecke, C., and McNamara, D.S. (2015). Natural Language Processing and Game-Based Practice in iSTART. Journal of Interactive Learning Research, 28(2): 189–208.

McNamara, D.S. (2017). Self-Explanation and Reading Strategy Training (SERT) Improves Low-Knowledge Students' Science Course Performance. Discourse Processes.

Snow, E.L., Likens, A.D., Allen, L.K., and McNamara, D.S. (2016). Taking Control: Stealth Assessment of Deterministic Behaviors Within a Game-Based System. International Journal of Artificial Intelligence in Education, 26: 1011–1032.

Weston-Sementelli, J.L., Allen, L.K., and McNamara, D.S. (2016). Comprehension and Writing Strategy Training Improves Performance on Content-Specific Source-Based Writing Tasks. International Journal of Artificial Intelligence in Education: 1–32.

Proceeding

Allen, L.K., Jacovina, M.E., and McNamara, D.S. (2016). Cohesive Features of Deep Text Comprehension Processes. In 38th Annual Conference of the Cognitive Science Society. Philadelphia, PA: Cognitive Science Society.