Partner Institution(s): Indiana University, INFLO, Clevent AI Technology LLC

Purpose: The purpose of this project is to create Active Learning at Scale (L@S), a flexible learning environment where postsecondary students can learn course content and practice their skills while on the go and in a variety of settings. Active L@S will expand access to evidence-based generative learning strategies (e.g., note taking, summarizing, self-explaining, question answering, and retrieval practice) for diverse students while also testing the boundaries and interactive factors that influence the efficacy of active learning across multiple contexts. Large language models (LLMs) will be used to generate interactive, research-informed, engaging, and interactive learning prompts on the fly, customized to any course content, and to provide immediate adaptive feedback in response to students' responses across multiple domains and contexts.

Project Activities: First, the project team will iteratively develop capabilities within an existing innovative mobile technology (INFLO) and will integrate an LLM into the system. As part of development, the project team will conduct human-centered design research to increase accessibility and usability, improve product functionality, and inform initial testing studies. Next, the project team will use data engineering, machine learning, and natural language processing to explore how active learning interacts with individual differences in postsecondary settings. The researchers will also use rapid-cycle experimentation to explore impacts of student behaviors, technology features, and generative learning activities on learning outcomes facilitated through the use of Terracotta and the Arizona State University Learning at Scale infrastructure.

Products: Active L@S will be assimilated into the Canvas learning management system (LMS). Research findings will be shared with academic and practitioner communities through peer-reviewed publications and conference presentations.

Structured Abstract

Setting: Research will take place at two large, public universities, Arizona State University and Indiana University.

Sample:  The project team aims to include over 100,000 students in the proposed research.

Project Focus: Active L@S is a new technological capability built on top of an existing mobile technology (INFLO) that will provide adaptive and generative learning activities leveraging LLMs. Active L@S will be assimilated into Canvas LMS and can be deployed to students on their mobile devices. It will personalize the learning experience using generative learning prompts that encourage students to retrieve knowledge at spaced intervals and will provide real-time, adaptive feedback across multiple domains.

Research Design and Methods: First, the project team will build the technical capability to implement active learning, automated scoring, and feedback methods, and will integrate the product into Canvas LMS. Next, the project team will develop the active learning prompts and AI algorithms. During development, the project team will adopt a learning engineering approach engaging various stakeholders like students, instructors, and instructional designers. Using a human-centered design process, the project will combine usability studies, focus studies, and other iterative processes. Finally, the project team will conduct multiple rounds of a/b testing to test and enhance the effects of Active L@S on learning outcomes.

Control Condition: The control condition for all of the experimental studies will be a group that interacts with the prior version of the mobile technology (INFLO) that does not have the generative learning and adaptive feedback enhancements.

Key Measures: The project team will collect student demographic data, course-level data, assignment-level data such as discussion board data and written assignments, single ease questionnaire, satisfaction questionnaire, subjective usability questionnaire, system usability scale, net promoter score, INFLO satisfaction scale, and INFLO adoption scale.

Data Analytic Strategy: The project team will use machine learning and natural language processing to explore strategies for making active learning accessible and impactful among diverse student populations. Bayes Factors analysis, and other data science techniques will be applied to evaluate the effectiveness and outcomes of the learning strategies and tools used with diverse learners to account for individual and contextual factors.

Cost Analysis: The research team will conduct a thorough cost-analysis throughout the project, considering development, maintenance, scalability, and integration of Active L@S. They will also conduct a comprehensive cost-benefit analysis to determine the return on investment for enhancing student outcomes. A sensitivity analysis will also be undertaken to factor in uncertainties.

Related IES Projects: The ASU Learning at Scale (L@S) Digital Learning Network (R305N210041), The Canvas+Terracotta LMS-Based Experimental Education Research Platform (R305N210035)