Co-Principal Investigator: Vincent Aleven

Purpose: When preparing instructional materials and lesson plans, teachers and instructional designers choose from an almost overwhelming set of student activity types. A fundamental problem in education is determining what combinations and sequences of activity types are most effective in supporting student learning. This research team hypothesizes that significant gains in robust learning are possible by careful selecting among a diverse set of activity types. To address this issue, the researchers will extend their existing web-based intelligent tutoring system (ITS), the Fractions Tutor, to incorporate a broad set of activity types and to create a new method for automatically selecting individualized activities. This research will contribute new knowledge about how to effectively leverage a broad range of activity types to achieve more robust learning. In addition, it will lead to a new, automated procedure for selecting activity types customized to the individual student's progress. These policies will be integrated into a real-world, publicly-accessible, web-based ITS for fractions learning.

Project Activities: The research team will first develop new activities that will promote sense-making and fluency-building and will incorporate these into the Fractions Tutor. Next, the team will gather initial data from students' use of the Fractions Tutor along with the new activities, which will inform the development of the activity selection method. Finally, the research team will conduct two experiments. The first will examine whether robust learning is enhanced in a tutor that supports multiple activity types or supports multiple activity types sequenced adaptively to the individual. The second will ascertain whether there is value in a self-improving version of the activity selection method that leverages data gained as the tutor interacts with more students.

Products: The products of this project will be a fully developed, publicly accessible, web-based ITS for fractions learning that effectively leverages a broad range of activity types to achieve more robust learning. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: Research will be carried out at several elementary schools located in an urban area in Pennsylvania. The collaborating school districts are diverse in terms of their demographics and academic achievement.

Sample: Initial data collection will occur with approximately 700 fourth- and fifth-grade students, Experiment 1 will occur with between 75–270 fourth- and fifth-grade students, and Experiment 2 will occur with approximately 800 fourth- and fifth-grade students.

Intervention: The research will produce a new version of the researchers' existing web-based ITS (the Fractions Tutor) for fourth- and fifth-grade fractions learning that will incorporate a wider range of activity types that support a broader set of learning mechanisms and a new, adaptive selection method. The new tutor will support sense-making processes and fluency-building processes, in addition to induction and refinement processes typically supported by intelligent tutoring systems. The new version of the Fractions Tutor will be made freely available on the Mathtutor website, which was funded by a previous IES grant (Bringing Cognitive Tutors to the Internet: A Website that Helps Middle-School Students Learn Math) to the co-PI.

Research Design and Methods: The research team will carry out activities designed to answer three research questions: (1) Does an ITS support more robust learning when activities are added to support fundamental learning mechanisms more evenly? (2) Does using a sequential decision-theoretic approach to adaptively sequence activities to the individual improve robust learning outcomes compared to a fixed sequence? and (3) Does using a self-optimizing action selection system, which uses accumulating student interaction data to improve its selection mechanism, lead to higher robust learning gains of students who use the system later on compared to students that use the initial system?

To answer these questions, the researchers will carry out three lines of research. First, they will develop new activity types for the Fractions Tutor. These activity types will be developed so that they promote sense-making and fluency-building processes. Next, they will gather initial data in classrooms using the Fractions Tutor in order to inform the development of an adaptive policy that selects among multiple activity types. Lastly, the research team will carry out two experiments. In Experiment 1, they will compare the standard tutor to two new versions of the ITS to evaluate whether robust learning is enhanced in a tutor that (unlike the standard tutor) supports multiple activity types or supports multiple activity types sequenced adaptively by the individual. This is a between-subjects design, where students will be randomly assigned to one of the three conditions. In Experiment 2, the research team will test the value of a self-improving version of the activity selection policy that leverages data gained as the tutor interacts with more students. Data will be conducted serially by classroom in five “waves.” Each wave will be considered a condition and analyses will look for differences between waves.

Control Condition: In Experiment 1, the control condition is the current version of the Fractions Tutor. In Experiment 2, the waves of data collection are compared to each other and there is no control condition.

Key Measures: The research team will use researcher-designed pre-tests and immediate and delayed post-tests to assess students' procedural and conceptual knowledge. These tests will include standardized test items gathered from readily available sources (e.g., NAEP, state tests, etc.) and items that closely track the Fractions Tutor problem types. In addition, they will log data of students' interactions with the tutor to see how different tutor versions affect learning behavior.

Data Analytic Strategy: The initial data collected will be used to inform the development of the activity selection method. Experiment 1 and Experiment 2 pre-test and post-test data will be analyzed using multivariate and simple analyses of variance. Researchers will use pairwise comparisons to determine whether students in each later wave performed better than students in each prior wave. In addition, the research team will analyze the temporal course of student performance and see whether parametric models fit the observed data (e.g., if a linear trend in the data is observed).

Related IES Projects: Bringing Cognitive Tutors to the Internet: A Website that Helps Middle-School Students Learn Math (R305A080093)

Products and Publications

Journal article, monograph, or newsletter

Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Ringenberg, M., and Koedinger, K.R. (2016). Example-Tracing Tutors: Intelligent Tutor Development for Non-Programmers. International Journal of Artificial Intelligence in Education, 26(1): 224–269.

Proceeding

Doroudi, S., Holstein, K., Aleven, V., and Brunskill, E. (2016). Sequence Matters, But How Exactly? A Method for Evaluating Activity Sequences from Data. In 9th International Conference on Educational Data Mining (pp. 70–77). Raleigh, NC: International Educational Data Mining Society (IEDMS).