Project Activities
First, the research team will analyze existing OpenStax data from digital textbooks with interactive tool interventions. They will use these fine-grained data to detect behavioral cues of self-regulated learning (SRL). Next, the research team will examine associations among individual learner characteristics, SRL behaviors, and text comprehension and knowledge retention measures. Finally, the research team will co-design SRL interventions with postsecondary students and explore the interventions' effects on tool use, comprehension, and knowledge retention within the OpenStax/Kinetic platform.
Structured Abstract
Setting
The data will come from OpenStax, a digital learning platform used nationwide.
Sample
The existing dataset includes data from approximately 360 students who interacted with 3 OpenStax digital textbooks (120 students per digital textbook). These students will represent ethnic/racial and sex groups. For the data collected as part of the study, the team will recruit at least 31 learners from each of 3 digital textbooks (93 students per study) across 2 rounds of 5 total studies (465 students total). This sample will represent U.S. college-level learners from diverse ethnic/racial and sex groups. For the development of SRL scaffolds, the researchers will collect data from representative members of the OpenStax population (10 students and 5 teachers).
The researchers will explore relationships between self-regulated learning (SRL) behaviors and interactive tools such as digital highlighters and notetaking tools within digital textbooks.
Research design and methods
First, the research team will analyze data in OpenStax from multiple existing randomized control experiments. These prior studies used three different digital textbooks and focused on the use of interactive tools for learning. The team will use these datasets to build a model to detect behavioral cues that indicate SRL. After developing the model, the team will test for algorithmic bias by investigating whether the model performs differently for different learning subpopulations and will apply methods to improve model fairness if needed. They will then apply the model to detect SRL to the existing datasets to explore the relationship between SRL behaviors and individual learner characteristics, patterns, interactive tool use, text comprehension, and knowledge retention. The team will also examine how SRL behaviors develop over time. Finally, the research team will co-design SRL interventions with postsecondary students and explore the interventions' effects on tool use, comprehension, and knowledge retention using two waves of experimental studies (a pretest-posttest design known as an A/B design) within the OpenStax/Kinetic platform. Across both waves, students will be randomly assigned to condition. The first wave of A/B tests will examine the effectiveness of different SRL scaffolds to identify the optimal set of scaffolds for supporting SRL. The second wave will compare the best SRL scaffolds to the original, unmodified learning system.
Control condition
For the A/B testing, the first wave will have four comparison conditions but no clear control condition. The second wave will include a control condition, which will be the original, unmodified digital textbook experience within OpenStax.
Key measures
Data for each study will include log, textual, self-reported pre-test learner characteristics (achievement goal orientation, anxiety, and self-efficacy for learning), and immediate and 1-week delayed knowledge assessments.
Data analytic strategy
The researchers will leverage educational data mining methods and text replays to log and textual data to engineer the data into a form that can be used to build machine learning algorithms to detect theoretically based, fine-grained SRL behaviors. Next, they will apply methods for identifying and mitigating algorithmic bias in the SRL detectors. Then, they will use a combination of correlation analysis (with post-hoc corrections), multi-level structural equation models, and epistemic network analysis to examine associations between learner characteristics, SRL behaviors, patterns, and outcome measures to examine whether SRL behaviors mediate the relationships between interactive tool use, comprehension, and knowledge retention, using multi-level structural equation models. Finally, statistical comparisons will be made between A/B test conditions.
Cost analysis strategy
The team will analyze the cost of using OpenStax Kinetic resources using the CostOut tool.
People and institutions involved
IES program contact(s)
Products and publications
This project will result in preliminary evidence of the relationship between SRL behaviors, interactive tool use, comprehension, and knowledge retention in the context of a digital learning platform that offers access to digital textbooks. The project will also result in a final dataset to be shared, peer-reviewed publications and presentations, and additional dissemination products that reach education stakeholders such as practitioners and policymakers.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Additional project information
This project is part of the Digital Learning Platforms to Enable Efficient Education Research Network (Digital Learning Platforms Network), which aims to leverage existing, widely used digital learning platforms for rigorous education research.
Supplemental information
Co-Principal Investigators: Barany, Amanda; Hutt, Stephen
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
