Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Decision Rule Research Project: Curriculum-Based Measurement in Reading
Center: NCSER Year: 2013
Principal Investigator: Christ, Theodore Awardee: University of Minnesota
Program: Reading, Writing, and Language      [Program Details]
Award Period: 07/01/2013–6/30/2017 Award Amount: $1,599,994
Type: Measurement Award Number: R324A130161
Description:

Purpose: Curriculum-based measurement (CBM) is frequently used to estimate the rate of individual student progress and evaluate instructional effects. CBM resources provide teachers with guidelines and decision rules for interpreting CBM scores and making instructional and placement decisions based on the data. However, these guidelines and rules have little information about their technical adequacy, indicating that potentially high-stakes decisions could be made with little support for the reliability or validity of the data interpretation and decision-making procedures. This project will evaluate CBM progress monitoring data, decision rules, and data interpretation procedures in reading. The project will answer questions related to the amount of data needed to best guide decisions, optimal standardization of assessment conditions, quality of measures and instrumentation, precision of growth estimates, accuracy of decision rules, schedule for data collection, and combinations of these factors to yield reliable and valid results to guide education decisions. It will result in evidence-based guidelines for the reliable and valid interpretation of CBM progress monitoring reading data as it is used to guide education practices and teacher decisions.

Project Activities: Four activities are planned for the development and validation of the guidelines. First, the project will use extant datasets to review technical documentation of CBM instrumentation related to reading, analyze extant datasets, and conduct simulation studies to investigate accurate methods for estimating growth and determining decision rules. Second, progress monitoring data will be collected to evaluate the rules generated in the first activity using a subset of progress monitoring measures. Third, there will be a comparison of the results and guidelines based on simulated data from the first activity with the results from the data collection and analyses from the second activity. Fourth, the project will develop a web-based program that enables practitioners to enter and interpret data. A combination of analytic strategies, including linear mixed effects regression, ordinary least squares, and analysis of variance, will be used to determine guidelines and compare their reliability and validity across activities.

Products: The primary products will consist of a fully developed and validated set of guidelines for progress monitoring data collection, interpretation, and use, as well as a web-based interface that enables practitioners to enter and interpret data.

Structured Abstract

Setting: Data collection for the second activity will occur in urban, rural, and suburban schools in Minnesota.

Sample: Students included in the extant datasets are from samples participating in CBM activities across the United States who were in kindergarten through Grade 6 and defined as either having a disability or being at risk for disabilities. Students participating in primary data collection will be in Grades 1 through 5.

Assessment: The project will use extant data from frequently used reading progress monitoring assessments to develop and validate guidelines. Sources of extant data include: AIMSweb, EasyCBM, Dynamic Indicators of Basic Early Literacy Skills, EdCheckup, and the Formative Assessment Instrumentation and Procedures. Based on the findings from the extant data analyses, a subset of these commonly used measures will be administered for primary data collection in the second activity.

Research Design and Methods: There are four activities in the development and validation of the guidelines. First, the project will review technical documentation of CBM instrumentation related to reading, analyze extant datasets, and conduct simulation studies to investigate accurate methods for estimating growth and determining decision rules. Second, the project will collect progress monitoring data to evaluate the rules generated in the first activity. Based on the findings from the extant data analyses, a subset of commonly used progress monitoring measures will be administered. Third, there will be a comparison of the results and guidelines based on simulated data from the first activity with the results from the data collection and analyses from the second activity. Fourth, the project will develop a web-based program that enables practitioners to enter and interpret data. A combination of analytic strategies, including linear mixed effects regression, ordinary least squares, and analysis of variance, will be used to determine guidelines and compare their reliability and validity across extant, simulated, and primary data collection activities.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: The measures being used are the assessment measures listed above.

Data Analytic Strategy: A combination of analytic strategies, including linear mixed effects regression, ordinary least squares, and analysis of variance, will be used to determine guidelines and compare their reliability and validity across extant, simulated, and primary data collection activities.

Publications

Journal article, monograph, or newsletter

Van Norman, E.R., and Christ, T.J. (2016). The Folly of Data Point Decision Rules: An Exercise in Basic Probability Theory. School Psychology Review. Full text


Back