|Title:||Developing Enhanced Assessment Tools for Capturing Students' Procedural Skills and Conceptual Understanding in Math|
|Principal Investigator:||Bottge, Brian||Awardee:||University of Kentucky|
|Program:||Cognition and Student Learning in Special Education [Program Details]|
|Award Period:||4 years (7/1/2015 – 6/30/2019||Award Amount:||$1,599,999|
Co-Principal Investigator: Allan Cohen (University of Georgia)
Purpose: The purpose of this measurement study is to develop, test, and refine a set of assessments for measuring the conceptual understanding and procedural math skills of middle school students with math difficulties (MD). Traditional measurement methods often do not effectively capture what students with MD know and are able to do in mathematics. Despite best efforts, many students with MD are still unsure of what test items are asking of them. In addition, current test formats do little to improve our understanding of the kinds of errors that students are likely to make. This project will develop mathematics tests that are intended to help students with MD gain a deeper understanding of the problem contexts and the questions posed, thereby seeking to minimize variability in scores due to irrelevant constructs, such as problem-decoding.
Project Activities: During the first year, the team will develop and refine a set of three new assessments: Interactive Computer-Based Test (ICBT), Formative Fractions Assessment (FFA), and Diagnostic Fractions Computation Test (DFCT). They will also develop training materials for teachers and develop and revise scoring protocols. The second and third years will involve multiple studies to determine the extent to which the new measures can assess student understanding relative to paper-and-pencil versions of the measures that existed from prior research studies. Additional activities across the years include the refinement and extension of psychometric models to aid in the interpretation of test scores and student misconceptions of math concepts including rational numbers.
Products: The products of this project will include fully developed versions of the ICBT, FFA, and DFCT. Products will also include evidence of validity and reliability for the developed measures, publications, and presentations.
Setting: The research will take place in middle schools in Kentucky.
Sample: The study sample includes students with MD who receive math instruction in self-contained special education classrooms. The total sample of MD students will be drawn from 24 middle schools with approximately 10-20 students per school.
Assessment: The new middle school math assessments to be developed include an Interactive Computer-Based Test (ICBT), Formative Fractions Assessment (FFA), and Diagnostic Fractions Computation Test (DFCT). The ICBT will assess students' problem solving skills and will include interactive item and information clusters, tracking features, and information for teachers. Both the FFA and DFCT will be computer-administered measures of students' fraction understanding with interactive features and detailed feedback provided to teachers on student understanding and student errors.
Research Design and Methods: The research design involves a number of methods to develop the new math measures. In the first year, researchers will collect data on the usability and feasibility of the measures using classroom observations, online teacher logs, and informal interviews with teachers and students. Subsequent studies will involve quasi-experimental and randomized experimental designs to assess performance of students using computerized, interactive tests versus paper-and-pencil versions. Navigation maps will be utilized to understand students' navigations through the assessment items using the software during the test-taking process. Simulation studies will be conducted to assess the quality of the new psychometric methods for interpreting test scores.
Control Condition: Due to the nature of the research design, there is no control condition.
Key Measures: The project will involve development and validation of new measures, including the ICBT, FFA, and DFCT.
Data Analytic Strategy: The project will use predominantly item response theory (IRT) modeling but will also utilize Markov chain Monte Carlo estimation. Differential item function (DIF) analyses will also be conducted to help detect item bias.
Related IES Projects: Evaluating the Efficacy of Enhanced Anchored Instruction for Middle School Students with Learning Disabilities in Math (R324A090179)