Project Activities
The researchers will develop and validate assessments that middle school mathematics teachers can readily incorporate into their existing curricula in order to guide instructional decisions related to three key concepts in algebra—equality, notion of a variable, and multiplicative reasoning.
Structured Abstract
Setting
The setting for this study includes middle schools in diverse regions and locations across the United States.
Sample
The study sample will consist of a nationally representative sample of approximately 6,000 students in Grades 6 to 8. The students will be diverse in academic proficiency, socioeconomic status, and ethnicity.
Two types of assessments will be developed and validated—periodic locator assessments to provide profiles of groups of students that require different instructional approaches, and incremental assessments to inform ongoing instructional decision-making. The incremental assessments will be small sets of items targeted on specific concepts that teachers could use with a group of students in the classroom as part of on-going formative assessment activities.
Middle school mathematics teachers can readily incorporate these assessments into their existing curricula in order to guide their instructional decisions related to three key concepts in algebra—equality, notion of a variable, and multiplicative reasoning. In addition to these three algebra constructs, the assessment framework includes four dimensions of competency (conceptual depth, conceptual breadth, procedural fluency/flexibility, and representational fluency). This two-dimensional assessment matrix—algebraic constructs by dimensions of competency—is further layered by a third dimension focusing on developmental models for describing how student learning develops over time.
Research design and methods
The project has three distinct phases of assessment development and validation activities. During Phase I, the researchers will develop the assessments and evaluate the items by seeking input and review from 12 middle school teachers and an advisory group of content experts. In Phase 2, the researchers will explore the validity of the items and assessments using two approaches. First, a content validity study using expert judgments will evaluate the degree to which student responses to items reflect the hierarchy of the developmental model and the dimensions of competency. Thirty experts, including middle grade mathematics teachers, teacher educators, and mathematics cognition researchers will be randomly assigned to one of three groups, with each group responsible for evaluating student responses focused on one of three algebraic constructs. Second, the researchers will carry out a field test of the items with groups of students in order to conduct statistical analyses to examine the validity of the items and sets of items. To field test the items, a sample of 4,000 to 6,000 middle school students will be assigned a set of items using a randomized-block design. The researchers expect that approximately 180-240 items will be tested. The goal will be to collect at least 1,000 student responses per item in order to have sufficient data for the exploratory and confirmatory factor analyses. Finally, in Phase 3, the researchers will examine the consequential validity evidence for the assessment. Twelve middle school teachers will use the assessments throughout the school year. Teachers will be asked to complete an online log each time they use either the periodic locator assessment or the incremental assessment. In addition, teachers will complete an assessment of pedagogical content knowledge in algebra, and periodic interviews will be conducted with the teachers.
Control condition
There is no control condition.
Key measures
The key measures for the study include students' responses on the assessments, student scores on standardized state assessments, online logs, teacher interviews, and teacher pedagogical content knowledge in algebra.
Data analytic strategy
Exploratory and confirmatory factor analysis will be used to investigate the dimensionality of the proposed matrix structure of the assessment. Differential item functioning analyses will also be performed. In addition, multi-dimensional item-response theory approaches will be used to construct score reports that provide information to teachers about groups of students with similar estimates of proficiency.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: Products include formative assessments that middle schools mathematics teachers can use to improve student learning and instruction and published reports.
Journal article, monograph, or newsletter
Graf, E.A., and Arieli-Attali, M. (2015). Designing and Developing Assessments of Complex Thinking in Mathematics for the Middle Grades. Theory Into Practice, 54 (3): 195-202.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
