Co-Principal Investigators: Sarama, Julie; Kutaka, Traci S.

Purpose: Children's ability to answer assessment questions correctly paints an incomplete portrait of what they know and can do, yet it remains a common basis for program evaluation. Researchers should study the development of competences by observing the ways children approach and make sense of problems, and how they learn to develop, apply, and generalize problem-solving strategies over time, from one type of problem to the next. This project has two goals. First, the research team will determine how kindergarten children's problem-solving strategies vary within and across 12 arithmetic problem structures and how they evolve over the course of an intervention. Second, they will construct two novel indicators of instructional efficacy, which jointly account for patterns of strategy use over time, application of strategies to increasingly sophisticated arithmetic problem types, and correctness of response: (1) a Sophistication indicator and (2) a Breadth indicator.

Project Activities: The team will carry out the study in two phases. In the initial phase, the team will watch and code videos of instructional sessions captured during a completed efficacy trial of a learning-trajectories approach. During phase two, the researchers will estimate hierarchical ordered logit models to produce patterns of strategy use over time within and between instructional sessions for particular problem structures. These models will then inform the construction of two novel indicators of instructional efficacy.

Products: The findings from this project will be reported in peer-reviewed publications and disseminated via conference proceedings.

Structured Abstract

Setting: This secondary analysis will leverage data collected from IES Grant R305A150243, Evaluating the Efficacy of Learning Trajectories in Early Mathematics.

Sample: This sample is composed of kindergarten children (N = 292) in a mountain western state across 16 classrooms within 4 schools in 2 urban districts.

Data: Beginning in January 2018, the intervention group (n = 145) received 2 hours of one-on-one instruction (divided into 15-minute sessions) during the spring semester. Instruction was informed by the Learning Trajectories approach. All one-on-one instructional sessions were videotaped, as were the pre- and post-assessments. This project will repurpose the video and assessment data from only the intervention group. The data from the counterfactual will not be used since instruction in this condition was limited to two kinds of story problems.

Research Design and Methods: In Phase I, the team will watch and code student mathematical behavior in videos of instructional sessions. They will develop a coding rubric to track information about the type and difficulty of the story problem structure, the pattern of strategies used for each attempt to solve each problem, and teacher feedback with respect to the correctness of the response per attempt. Importantly, they will also code how students incorporate teacher feedback from the previous attempt into future strategy selection. In Phase II, the team will use the coded data from Phase I to estimate hierarchical ordered logit models in a Bayesian setting in R, where the outcome of interest is problem-solving strategy sophistication. These models will produce fitted patterns of strategy use over time within and between instructional sessions for particular problem structures. Additionally, fitted values will be used to: 1) construct two novel indicators of problem-solving strategy sophistication and breadth; and 2) examine their concurrent validity using pre- and post-assessment data.

Key Measures: The pre- and post-assessments used in the efficacy study were a collection of arithmetic items derived from the Research-based Early Mathematics Assessment (REMA) and the Test of Early Mathematics Assessment – Third Edition (TEMA-3).

Data Analytic Strategy: Phase I data analysis will include establishing a coding rubric, beginning with an initial set of variables to be coding, and permitting new codes to emerge. Researchers will also establish inter-rater reliability. Phase II data analysis will use Generalized Linear Mixed Models to analyze the ordered categorical data derived from Phase I coding. They will also assess the concurrent validity of the two novel indicators they create via the argument-based approach.

Related Projects: Evaluating the Efficacy of Learning Trajectories in Early Mathematics (R305A150243)