Project Activities
The team carried out the study in two phases. In the initial phase, the team watched and coded videos of instructional sessions captured during a previous efficacy trial of a learning-trajectories approach. During phase two, the researchers estimated hierarchical ordered logit models to produce patterns of strategy use over time within and between instructional sessions for particular problem structures. These models informed the construction of two novel indicators of instructional efficacy.
Structured Abstract
Setting
This project used secondary data collected from IES Grant R305A150243 (Evaluating the Efficacy of Learning Trajectories in Early Mathematics), conducted in a mountain western state within urban schools districts.
Sample
The sample of the earlier study included 40 kindergarten children (19 girls) across 16 classrooms within 4 schools in 2 urban districts. No demographic information beyond child sex was collected because the school district has a policy of precluding sharing child-level data.
The researchers examined the development of mathematical competency by observing the ways children approach and make sense of problems and how they learn to develop, apply, and generalize problem-solving strategies over time from one type of problem to the next.
Research design and methods
In the original study (R305A150243) each child in the sample received 2 hours of 1-on-1 instruction during the spring semester (divided into 15-minute sessions). All instructional sessions were videotaped, and students took pre- and post-assessments. In phase I of the current study, the research team watched and coded student problem-solving behavior in videos of 522 instructional sessions (from 40 students). They developed a coding rubric to track information about the type and difficulty of the story problem structures, the pattern of strategies children used for each attempt to solve each problem, and the instructors’ feedback. In phase 2, the team used the coded data from first phase to estimate statistical models where the outcome of interest was strategy sophistication and breadth. The model output was used to (1) construct two novel indicators of problem-solving strategy sophistication and breadth and (2) examine their concurrent validity using pre- and post-assessment data.
Key measures
The pre- and post-assessments used in the prior efficacy study were a collection of arithmetic items derived from the Research-based Early Mathematics Assessment (REMA; Clements et al., 2008) and the Test of Early Mathematics Assessment – Third Edition (TEMA-3; Ginsburg & Baroody, 2003). The REMA and TEMA-3 measure pre-K through grade 3 children’s mathematical knowledge, and both have been standardized and validated using representative U.S. samples. Items were administered in individual interviews of each child, with explicit protocol, coding, and scoring procedures. No feedback on performance was provided. Assessors record correctness of responses and strategies used by children. All student assessments were video recorded.
Data analytic strategy
Phase I data analysis included establishing a coding rubric, beginning with an initial set of variables to be coded. the researchers also established interrater agreement. Phase II data analyses used statistical models to analyze the ordered strategy data derived from phase I coding. The analyses also assessed the concurrent validity of the two novel indicators that were created.
Key outcomes
The main findings of this project are as follows:
- The researchers analyzed videos of children solving arithmetic story problems. They found that kindergarteners develop more sophisticated problem-solving strategies as they were exposed to different types of story problems over the course of a semester. Importantly, the results showed that strategy sophistication is primarily driven by the location of the unknown quantity in the story problem (5+6=x vs. 5+x=11 vs. x+5=11) and not the mathematical operation (addition vs. subtraction) needed to solve the problem. These findings are consistent with research promoting using story problems to strengthen understanding of early addition and subtraction (Kutaka et al., 2023)
- Children who received Learning Trajectory-aligned instruction used more sophisticated strategies than their peers who received teach-to-target skill instruction (Kutaka, Chernyavskiy, Sarama, & Clements, 2023).
- The researchers developed a novel statistical method for dealing with missing data, a common problem in projects focused on describing and understanding complex human behavior. Unlike common solutions, this method does not require researchers to discard any data (see R code in Additional Online Resources below). The new model can be applied broadly (inside the field of education and to other fields, such as psychology, medicine, etc.) (Chernyavskiy et al., 2024).
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Select publications:
Kutaka, T. S., Chernyavskiy, P., Sarama, J., & Clements, D. H. (2023). Ordinal models to analyze strategy sophistication: Evidence from a learning trajectory efficacy study. Journal of School Psychology, 97, 77-100.
Chernyavskiy, P., Kutaka, T.S., Keeter, C., Sarama, J., & Clements, D.H. (2024). Addressing uncodable behaviors: A Bayesian ordinal mixture model applied to a mathematics learning trajectory teaching experiment. Journal of Research on Educational Effectiveness, 1-31. doi.org/10.31234/osf.io/p4qjf.
Kutaka, T.S., Chernyavskiy, P., Cong. M., McCredie, K., Sarama, J., & Clements, D.H. (2024). How Story Problems Strengthen Arithmetic Problem-Solving Strategies: Evidence from a Learning Trajectory Teaching Experiment in Kindergarten. Learning and Instruction, 93, . doi.org/10.31234/osf.io/hjbzg
Available data:
For those interested in working with the Strategies Database, please contact the research team: Traci Shizu Kutaka (traci.kutaka@virginia.edu) or Pavel Chernyavskiy (pchern@virginia.edu).
Additional project information
Additional online resources and information:
- R functions and analysis code for new Bayesian mixture model developed in Chernyavskiy et al. (2024): https://github.com/pchernya/oclhm_jree
- R analysis code used for Kutaka et al. (2023): www.github.com/pchernya/Strat_sophist_ordinal_models
- R analysis code used for Kutaka et al. (2024): https://github.com/pchernya/LI_soph_ovr_time
Related projects
Supplemental information
Co-Principal Investigators: Sarama, Julie; Kutaka, Traci S.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
