Project Activities
The GENIUS Center will (1) conduct exploratory studies to examine how generative AI is currently used in classrooms to improve teaching and learning outcomes, (2) informed by the exploratory studies, the Center will develop, test, and refine the GenAgent system in science classrooms, and (3) conduct a pilot study to assess the promise of GenAgent for improving learners' education outcomes. In addition, the GENIUS Center will provide national leadership and outreach activities on the responsible use of GenAI to improve learner outcomes.
Focused program of research
The GENIUS Center will conduct five exploratory studies to investigate the necessary multimodal features that are needed for GenAgent to serve multiple key roles to facilitate STEM+C teaching and learning. Informed by the exploratory studies, GENIUS will iteratively develop and pilot GenAgent using a design-based approach. The GENIUS Center will use a large corpus of de-identified student responses to develop open-sourced multimodal large language models. The models will serve as the foundation model for GenAgent, which will support visual, auditory, kinesthetic, and digital learning preferences, offering personalized feedback and fostering collaborative learning environments.
National leadership and outreach activities
The GENIUS Center will collaborate with key stakeholders, including teachers, school districts, academic institutions, entrepreneurs, and policymakers, to shape its research agenda. National leadership activities will include the development of webinars, workshops, and policy briefs for key stakeholders.
Structured Abstract
Setting
The research will take place in urban and rural middle schools across 5 states: Georgia, Michigan, South Carolina, Tennessee, and California.
Sample
The research will include a diverse sample of approximately 175 middle school science teachers and their 15,000 students, representing a wide range of racial and ethnic backgrounds, including students with disabilities. Specifically, during Phase I, five teachers and their 250 students will be recruited for each exploratory study. In Phase II, to evaluate the acceptability and usability of GenAgent, 15 teachers and their 750 students will be recruited. Additionally, six teachers and their 300 students, randomly selected from the prior usability study, will participate in the feasibility study to assess the refined GenAgent tool. In Phase III, 135 teachers and their 13,500 students will participate in the pilot study to evaluate the promise of GenAgent on STEM+C learning.
Research design and methods
The focused program of research under the GENIUS Center will occur in three phases.
In Phase I, the GENIUS Center will conduct five exploratory studies to investigate the necessary multimodal features that are needed for GenAgent to serve multiple key roles to facilitate STEM+C learning. Utilizing a design-based research approach, the exploratory studies will employ a series of short-term, small-scale, quasi-experiments to examine the core features of the Agents to be included in the fully developed GenAgent. Data collection for each exploratory includes classroom observations, focus groups, and students' progress on science learning outcomes. Exploratory study 1 will focus on GenAI's ability to elicit cognitive engagement to prompt students' questioning and problem-defining practices. Exploratory study 2 will examine the accuracy of GenAI in providing feedback and scaffolding on computational models. Exploratory study 3 will use GenAI as a learning buddy to aid in discourse and argumentation practices. Exploratory study 4 centers around GenAI's role in scientific investigation, acting as a collaborative learning agent. Exploratory study 5 will investigate GenAI's utility in analyzing and interpreting data, examining how these tools affect students' understanding of the nature of science.
In Phase II, building on the findings from the exploratory studies, the GENIUS Center will iteratively develop and test a multi-role GenAgent to provide various forms of adaptive support to students and teachers, engaging students in science and engineering practices (SEPs) and promoting learning outcomes and discipline-based AI literacy. Based on the multimodel learning theory and the potential of multimodal large language models (LLMs) from prior work, the GENIUS Center will develop and design GenAgent, including training open-sourced multimodal LLMs such as Meta's LLaMA series, explainable AI (XAI) outcomes, using a large student data corpus. Afterwards, the researchers will conduct an acceptability and usability study and a feasibility study through surveys and interviews. After each component is developed and tested, GENIUS will compose the multiple agents (mentor, learning buddy, collaboration partner, and teaching assistant) with a central controller to form GenAgent.
In Phase III, the GENIUS Center will conduct a pilot study using a cluster randomized design in middle school STEM classrooms to examine GenAgent's promise for enhancing multimodal STEM learning. Classrooms will be randomly assigned to either the treatment group that uses the STEM+C curriculum with GenAgent or to the control group that uses same STEM+C curriculum without GenAgent. Data will be collected through surveys, pre-and post-tests, classroom observations, instructional logs, and student log file data.
Key measures
For students' STEM+C learning outcomes, key measures include researcher developed measures of students' knowledge-in-use and problem-solving skills, and their computational thinking skills when engaging in science and engineering practices. State science assessments or MAP Growth assessments will also be used to assess science learning outcomes as distal measures. Student engagement will be measured using the Math and Science Engagement Scale and the Student Attitudes Toward STEM survey. The GENIUS Center will also assess student AI literacy using a researcher developed measure.
Data analytic strategy
The GENIUS Center will use thematic analysis to analyze qualitative data from interviews and classroom observations in Phase I and II to gain in-depth insights into GenAgent's strengths and challenges. In Phase III, quantitative data analysis will employ both descriptive and inferential statistics to examine the promise of the GenAgent on student outcomes. Hierarchical linear modeling (HLM) and structural equation modeling (SEM) will be used to account for the complex nature of educational data and to identify the factors most strongly associated between GenAgent and successful learning outcomes in STEM+C. A set of moderator variables and mediating variables will be included in the SEM analysis. Qualitative thematic analysis will be employed to explore teachers' and students' perceptions, experiences, and implementation challenges of GenAgent.
Cost analysis strategy
The GENIUS Center will analyze implementation costs of GenAgent using the "ingredients method" and will include assessing direct costs such and classroom resources and teacher training, along with broader societal impacts. They will collect data from technology officers during the field tests to determine the costs of computing resources, start-up expenses, and ongoing operational costs, adjusting for inflation and geographic differences. The analysis will report on per-student costs, learning outcome costs, and compare GenAgent's cost-effectiveness to traditional classrooms over a five-year period across various districts.
People and institutions involved
IES program contact(s)
Project contributors
SubAwardee(s)
Products and publications
The project findings will result in significant contributions to research on multimodal collaborative learning environments through the integration of generative AI and computing with science and engineering practices. The GENIUS Center will develop and test the GenAgent tool to enable educators to integrate GenAI and computational thinking into the STEM+C curriculum, enhancing student engagement and learning outcomes. The GENIUS Center will produce publications and presentations and other dissemination products (for example, website, webinar) that will reach a broad range of stakeholders including educators and policymakers. The GENIUS Center will produce a suite of open-sourced multimodal disciplinary LLMs trained on contextual resources (classroom data), tools, student learning resources, teacher professional learning materials, empirical research findings on the potential efficacy of GenAgent in education, policy briefs, webinars, and best practices for its ethical and responsible use.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
