Project Activities
This study employs a student-level, completely randomized design. Students who participate in the after-school program will be randomly assigned to work either in a classroom with the ALEKS® system or one led by a certified math teacher. Both the experimental and control conditions will provide additional math help to students and have the same duration (three 20-min blocks per session, twice weekly). Researchers will monitor the implementation of the intervention using specific measures that target key features of the ALEKS® system.
Structured Abstract
Setting
This project will take place in a public school system in Jackson, Tennessee.
Sample
This study will take place in a Tennessee County School System and approximately 200 students will participate. Three cohorts of sixth grade students will participate in the program and students who are below average in mathematics from selected schools will be recruited. Specifically, participant criteria include sixth grade students who score test below a 40% proficiency cutoff in math on the Tennessee Comprehensive Assessment Program (TCAP) Achievement Test.
ALEKS® is a Web-based intelligent tutoring system (ITS) that instructs students on the mathematical topics that they are most ready to learn. ALEKS® is fundamentally different from previous educational software because it has an artificial intelligence engine that assesses each student individually and continuously at a fine-grain rather than a course-grain level. ALEKS® assesses students' current knowledge, instructs students on the topics they are most ready to learn, and evaluates student performance on problems related to those topics. ALEKS® uses adaptive questioning to quickly and accurately determine exactly what a student knows and does not know in a math course. To ensure that topics learned are retained in long-term memory, ALEKS® periodically reassesses the student, using the results to adjust the student's lesson plan. Students must demonstrate mastery of the instructed content through mixed question assessments.
Research design and methods
This study uses a student-level completely randomized design. At each site, 75% of eligible students (those in the lowest 40% measured by TCAP) will be randomly selected to receive an invitation to participate in the study. Each year a cohort of up to 400 sixth grade students (minimum 55 and maximum 100 per school site) will be involved in the program, thereby involving up to 1200 students from the target schools over the intervention period. Students who agree to participate in the after-school program will be randomly assigned to work with either the ALEKS® system or a time-equivalent control where students will work on additional math problems. Experimental condition students will work with ALEKS® using an Individualized Lesson Plan (ILP), which will deliver a customized curriculum based on the student's identified weaknesses. Each school site will be supervised by two monitors who will have undergone two days of training on the system. The monitors will have two distinct tasks: provide technical assistance on the system (but not on the content that the children are learning) and provide bimonthly system progress reports to parents. Implementation of the intervention will be monitored using specific measures that target key features of the program. The dependent variable will be student TCAP scores, resulting in a clean, randomized pre–post test design with one, two-level independent variable (program type) and one primary dependent variable (TCAP).
Control condition
Students in the control group will participate in a study-hall style after-school-program, completing extra mathematics practice based on the TCAP, but without individual tailoring from either ALEKS® or teacher monitors.
Key measures
The main outcome measure will be the TCAP Achievement Test in mathematics. Additional measures include attitudes towards technology and learning mathematics and observational data.
Data analytic strategy
Researchers will analyze the overall effectiveness of the treatment using an analysis of covariance linear model with pretest and possible other factors as covariates, such as prior math achievement, sex of student, student ethnicity, and school affiliation. System-generated data related to time spent on ALEKS®, number of learning objectives mastered, and rate of learning during each session will be analyzed and cross-referenced to understand how those factors impact learning. Sixth grade test data will be analyzed to determine gains between fifth and sixth grade for both experimental and control students. Analysis of teacher effect data for both experimental and control students will provide additional explanation of change in student achievement.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: The products of this project will be evidence of the efficacy of the Assessment and LEarning in Knowledge Spaces (ALEKS®) system as an intervention in after-school settings to improve the mathematical skills of struggling 6th grade students. Peer reviewed publications will also be produced.
Book chapter
Brawner, K., and Graesser, A. (2014). Natural Language, Discourse, and Conversational Dialogues Within Intelligent Tutoring Systems: A Review. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 189-204). Orlando, FL: Army Research Laboratory.
Cai, Z., Feng, S., Baer, W., and Graesser, A. (2014). Instructional Strategies in Trialogue-Based Intelligent Tutoring Systems. In R. Sottilare, A.C. Graesser, X. Hu, and B. Goldberg (Eds.), Design Recommendations for Intelligent Tutoring Systems: Instructional Management, Volume 2 (pp. 225-235). Orlando, FL: Army Research Laboratory.
Journal article, monograph, or newsletter
Graesser, A.C., Li, H., and Forsyth, C. (2014). Learning by Communicating in Natural Language With Conversational Agents. Current Directions in Psychological Science, 23(5): 374-380.
Mo, L., Yang, F., and Hu, X. (2011). An Empirical Examination of IRT Information for School Climate Surveys. Educational Research and Evaluation, 17(1): 33-45.
Mo, L., Yang, F., Hu, X., Calaway, F., and Nickey, J. (2011). ACT Test Performance by Advanced Placement Students in Memphis City Schools. The Journal Of Educational Research, 104(5): 354-359.
Nye, B.D., Graesser, A.C., and Hu, X. (2014). AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education, 24(4): 427-469.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
