Project Activities
The researchers will conduct five experiments to evaluate a set of malleable factors of tutoring. The factors were chosen based on the following criteria: (1) the factors all challenge commonly held beliefs about what makes tutoring effective; (2) the researchers observed many instances of each option associated with each malleable factor; and (3) correlational analyses from the researchers' prior IES–funded work (Improving a Natural-Language Tutoring System that Engages Students in Deep Reasoning Dialogues about Physics) suggest that at least one option per factor predicts learning for high ability students, low ability students, or both types of students. The researchers will manipulate automated dialogues in a computer-based tutoring system in order to determine if some tutoring decisions are better than others for physics content, and whether students' self-efficacy and aptitude play a role in determining optimal tutoring decisions.
Structured Abstract
Setting
Six high schools from urban, suburban, and parochial school districts in Pennsylvania will participate in the study. The research will be conducted in the high schools' computer labs or at the research team's laboratory at the University of Pittsburgh.
Sample
For the first study, the participants are 70 high school students currently taking physics. For each of the remaining four studies, participants will be 100 high school students currently taking physics.
The research team will identify effective malleable factors of tutoring in order to inform the development of a future ITS intervention and more broadly to inform one-on-one tutoring and classroom instructional practices.
Research design and methods
The research team will conduct five studies to address particular malleable factors of tutoring. Each study will address a different factor. For each study, a feature of Rimac, the natural-language tutoring system for high school physics developed under a previously-funded IES grant (Improving a Natural-Language Tutoring System that Engages Students in Deep Reasoning Dialogues about Physics), will be manipulated based on the malleable factor of interest for that particular study. For all studies, there will be two conditions and the design will be between-subjects, so each student will be randomly assigned to only one condition. Participants in all studies will first complete a pre-test and self-efficacy survey, will then interact with a set of problems in Rimac and engage in automated dialogues that will vary depending on the experimental condition, and will finally complete a post-test. Study 1 will measure the gain score from pre-test to post-test for overall physics knowledge and Studies 2-5 will measure overall gain score as well as gain scores for different types of knowledge such as conditional reasoning, ability to instantiate abstractions, ability to break a whole into its parts, ability to specify quantities in terms of appropriate units, directional specifiers, and planning ability.
Control condition
Across the experiments, the control or comparison condition will vary as a function of the research question.
Key measures
Key measures include researcher-designed pre-tests and post-tests which will measure the specific knowledge components that the dialogue focused on. When possible, researchers plan to reuse questions from the Force Concept Inventory, which targets known misconceptions. The Physics Self-Efficacy Questionnaire and PSAT scores will be collected to address questions about different types of students.
Data analytic strategy
For each study, multiple regression will be used to analyze the effect of the manipulated independent variable on learning outcomes. In Studies 2–5, self-efficacy will be included as a second quantitative moderator variable and the analysis will be performed at multiple levels.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The products of this project will be preliminary evidence of promising policies to guide adaptive scaffolding in natural dialogue systems. In addition, strategies for instructional dialogue in one-on-one tutoring and classroom settings will be presented. Peer-reviewed publications will also be produced.
Publications:
Journal article, monograph, or newsletter
Jordan, P., Albacete, P., and Katz, S. (2015). When is it Helpful to Restate Student Responses Within a Tutorial Dialogue System?. Artificial Intelligence in Education, 9112: 658-661.
Lipschultz, M., Litman, D., Katz, S., Albacete, P., and Jordan, P. (2014). Predicting Semantic Changes in Abstraction in Tutor Responses to Students. International Journal of Learning Technology, 9(3): 281-303.
Proceeding
Jordan, P., Albacete, P. and Katz, S. (2015). Exploring the Effects of Redundancy within a Tutorial Dialogue System: Restating Students' Responses. In 16th Annual SIGdial Meeting on Discourse and Dialogue (pp. 51-59). Prague, CZ: Special Interest Group on Discourse and Dialogue SIGdial.
Jordan, P., Albacete, P., and Katz, S. (2016). Exploring Contingent Step Decomposition in a Tutorial Dialogue System. In 24th Conference on User Modeling, Adaptation and Personalization (UMAP). Halifax, NS Canada: ACM.
Katz, S., Albacete, P., and Jordan, P. (2016). Do Summaries Support Learning from Post Problem Reflective Dialogues?. In 13th International Conference on Intelligent Tutoring Systems (ITS 2016) (pp. 519-520). Zagreb, HR: Springer International Publishing.
Katz, S., Jordan, P., and Albacete, P. (2016). Exploring How to Adaptively Apply Tutorial Dialogue Tactics. In Proceedings of the 16th IEEE International Conference on Advanced Learning Technologies (ICALT 2016) (pp. 36-38). Austin, TX: IEEE .
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
