IES Grant
Title: | Updating Middle School Mathematics Diagnostics Teacher Assessments in Mathematics & Science (DTAMS) | ||
Center: | NCER | Year: | 2012 |
Principal Investigator: | Peters, Susan | Awardee: | University of Louisville |
Program: | Effective Instruction [Program Details] | ||
Award Period: | 3 years (7/01/2012-6/30/2015) | Award Amount: | $1,291,941 |
Type: | Measurement | Award Number: | R305A120553 |
Description: | Co-Principal Investigator: Susan A. Peters Purpose: The purpose of this project is to update the Middle School Mathematics portion of the Diagnostics Teacher Assessments in Mathematics and Science (DTAMS) so that it will be aligned with the Common Core State Standards in Mathematics (CCSSM) and its Standards for Mathematical Practice. DTAMS is a widely used set of assessments that measure teacher mathematical content knowledge (i.e., number/computation, algebraic ideas, geometry/measurement, and probability/statistics), mathematical problem-solving and reasoning, and mathematics knowledge for teaching. The intended uses of this revised version of the Middle School Mathematics DTAMS include assessing the impact of teacher professional development and providing feedback to teachers about their knowledge of teaching mathematics. Project Activities: In the first year of the study, researchers will review and revise items from the original DTAMS assessments. New items will be developed by panels of mathematics education and assessment experts via an in-person item development workshop. In the second year of the study, researchers will pilot items on 1,200 middle school mathematics teachers, who will be randomly assigned to item sets. Results from data analyses and teacher comments for each item will inform additional revisions to the measure and development of the four parallel forms. A web-based system will also be developed for more efficient instrument delivery and scoring. In the third year of the study, researchers will conduct the final field testing and predictive validity study involving 80 middle school mathematics teachers and their students (approximately 1,600). Measurement predictive validity will be tested against teacher instructional practices and student achievement. Researchers will randomly assign teachers to item sets. Measurement validity analyses will include the revised DTAMS scores, independent observation of teacher instruction, and student mathematics achievement scores, as well as teacher historical variables, student demographics, and previous achievement. Products: The products of this project will be a fully-developed and validated measure for middle school teachers' knowledge of mathematical content, mathematical problem-solving and reasoning, mathematics knowledge for teaching, and implementation of mathematical practices. A web-based system for instrument delivery and scoring and peer reviewed publications will also be produced. Structured Abstract Setting: This project will take place at the University of Louisville and in middle schools in urban, suburban, and rural communities in Kentucky. Sample: During the pilot test of assessment items, approximately 1,200 middle school mathematics teachers will be recruited. During the final field testing and predictive validity study, 80 middle school mathematics teachers and their students (approximately 20 students per Grade 6–8 classroom, totaling 1,600 students) from varying racial, ethnic, and income backgrounds will be recruited. Assessment: The revised version of the Middle School Mathematics DTAMS will assess teacher knowledge of mathematical content, teacher implementation of mathematical practices, and mathematics knowledge for teaching. This measure will assess four content areas of middle school mathematics, including number/computation, geometry/measurement, probability/statistics, and algebraic ideas. Each of these content areas will contain 30 multiple-choice and open-ended questions and require 60 to 90 minutes to complete. Thus, the number of items for the entire set of math assessments will total 120 and teachers are expected to complete the entire assessment in 4 to 6 hours. Ultimately, four parallel forms of this instrument will be developed and validated. Answer keys for multiple-choice questions and scoring rubrics for open-ended questions will be developed. A web-based system will also be developed for more efficient instrument delivery and scoring. Research Design & Methods: In the first year of the study, researchers will review and revise items from the original DTAMS assessments. This process will include developing a revised conceptual blueprint to describe the mathematics content areas according to the CCSSM, knowledge for teaching, and cognitive complexity levels appropriate for the middle school grade levels. Items from the original DTAMS will be evaluated based on this blueprint, previous performance of items via Item-Response Theory analyses results, alignment with CCSSM source materials (e.g., newly developed curriculum review tools, state assessments), and expert ratings. New items will be developed by panels of mathematics education and assessment experts via an in-person item development workshop. In the second year of the study, researchers will pilot items on 1,200 middle school mathematics teachers, who will be randomly assigned to item sets. Each item will have responses from approximately 300 teachers. Results from data analyses and teacher comments for each item will inform additional revisions to the measure and development of the four parallel forms. In the third year of the study, researchers will conduct the final field testing and predictive validity study involving 80 middle school mathematics teachers and their students (approximately 1,600). Researchers will randomly assign teachers to item sets. Teachers will complete one version of the revised DTAMS at the beginning of the school year (pre-test) and a different version at end of the school year (post-test). Each item will have responses from approximately 40 teachers. In addition, teachers will report the number and types of mathematics courses completed, level and type of teaching certification and years of experience, previous standardized test scores (i.e., PRAXIS, ACT, or SAT), and start and finish times in completing the assessments. Students will complete pre-test and post-test using the TerraNova Third Edition mathematics assessments. Researchers will observe each teacher three times during the school year and code the cognitive complexity of instruction of the content area of the DTAMS. Control Condition: There is no control condition. Key Measures: Teachers will report the number and types of mathematics courses completed, level and type of teaching certification and years of experience, previous standardized test scores (i.e., PRAXIS, ACT, or SAT), and start and finish times in completing the assessments. Students will complete the TerraNova Third Edition mathematics assessments, a set of normed assessments measuring higher-order thinking, mathematical reasoning, problem-solving, conceptual understanding, and skill proficiency in mathematics. Observations of teacher instruction will be coded for types of mathematics content introduced by teachers; types of teaching strategies used; cognitive level of teacher tasks demonstrated; cognitive level of student tasks attempted; student mathematical errors and misconceptions addressed by the teacher; and any pictorial, physical, and/or real-world representations used. These observational codes will be summarized into a depth-of-knowledge score between 1 and 3 for each teacher. Data Analytic Strategy: Researchers will use a combination of classical test theory and Rasch item response theory modeling to analyze and interpret data. For the pilot test of the items, researchers will use person-item mapping and Rasch modeling to evaluate fit of items and persons to the measurement model. In addition, researchers will review descriptive data (i.e., central tendency, score variability, skewness, and kurtosis), internal consistency reliability, and decision consistency. |
||
Back |