Co-Principal Investigators: Jonathan Brendefur, Jennifer Snow, Richard Osguthorpe

Purpose: In order for learning to be effective, teachers must accurately monitor students' learning. Monitoring is important to making informed decisions about whether educational objectives have been reached by all students, and to identify which students may need additional help to reach their objectives. This research team, using a metacognitive perspective of teaching, believe that monitoring accuracy plays an important role in effective teaching—and that by improving the accuracy with which teachers are monitoring their students learning, teachers' effectiveness and student achievement can be improved. The purpose of this study is to develop and test a professional development program called Monitoring Professional Development (Monitoring PD), a program intended to improve the accuracy of elementary teachers' monitoring of student learning, specifically in mathematics. In addition, researchers will gather evidence about the relation between teachers' monitoring accuracy and student achievement.

Project Activities: In this project, the research team will develop Monitoring PD. Training will be provided during a summer institute and professional development activities delivered during the school year. The professional development program (PD) begins with a summer institute presenting information about formative assessments and orienting teachers to the activities of the upcoming year. The program then continues throughout the school year utilizing grade-level or team work groups, individual coaching, and observations with debriefing. Using an iterative process, Monitoring PD will be developed as a stand-alone PD and as an add-on to be integrated into an existing mathematics instruction PD entitled Developing Mathematical Thinking (DMT). Over the project period, the team will implement and test the different modes of delivery in order to ascertain which delivery mode to use in the pilot study. In the final year of the study, a pilot test of the Monitoring PD will be conducted using an experimental design in order to provide preliminary evidence for the potential benefits of Monitoring PD for improving student achievement.

Products: This project will create a fully developed professional development program entitled Monitoring PD. This program is intended to improve the accuracy of teachers' assessment of their students' math progress and serve as a preliminary test of the connection between teachers' participation in this program and student math achievement. Peer reviewed publications will also be produced.

Structured Abstract

Setting: This project will take place in elementary schools in Boise, Idaho.

Sample: Participants include approximately 288 first- through sixth-grade teachers across 24 elementary schools (12 teachers per school). Each year, 96 teachers across 8 schools will be recruited.

Intervention: The intervention is intended to improve the accuracy of teachers' estimate of their students' learning by supporting teacher use of formative assessments. The intervention will consist of both a summer institute and ongoing professional development activities during the school year. During the summer institute, teachers will learn about the purpose of the training, particularly in regard to the importance of accurate assessment of student learning. Additionally, teachers will participate in discussions about the types of information sources used for estimating student learning (with an emphasis on using valid formative assessments) and receive training on developing and applying formative assessments to their teaching practices. Ongoing professional development during the school year includes working in within-grade-level groups, receiving individual coaching, and learning from classroom observations focused on teachers' creation and use of formative assessments. Two additional components will be considered for inclusion into the professional development program if they enhance program effects. One component involves having teachers writing weekly about their use of formative assessments in estimating their students' acquisition of math skills (e.g., chronicling). The purpose of this component is to prime teachers to information gained through these formative assessments. The other component involves teachers being given practice trials predicting their students' math achievement and researcher feedback during the school year.

Research Design and Methods: In the initial year of the project, the team will develop the materials for the Monitoring PD program, and then test combinations of the new PD with existing mathematics PD and teacher chronicles to identify which combinations of PD supports show the most promise. In order to carry out the comparisons, schools will be randomly assigned to six conditions: a) standard classroom practice; b) pre-existing PD program intended to improve mathematics instruction (Developing Mathematical Thinking [DMT]); c) Monitoring PD only; d) Monitoring PD plus chronicling; e) Monitoring PD plus DMT; and f) Monitoring PD plus DMT plus chronicling. The Monitoring PD program will be revised based on feedback from teachers and analysis of observational data and other measures (i.e., teacher self-report measures of attitudes and practice and teacher-predicted and actual student scores on a math achievement test). In particular, researchers will compare monitoring accuracy and use of effective formative assessments across conditions. If monitoring accuracy is greater for the chronicling condition compared to non-chronicling condition, then researchers will include chronicling in the Monitoring PD for Year 2.

In Year 2, a new cohort of schools will be randomly assigned to six conditions: a) standard classroom practices; b) DMT only; c) Monitoring PD only; d) Monitoring PD plus practice; e) Monitoring PD plus DMT; and f) Monitoring PD plus DMT plus practice. The Monitoring PD will be revised based on feedback from teachers and analysis of observational data and other measures. In particular, researchers will compare monitoring accuracy and use of effective formative assessments across conditions. If monitoring accuracy is greater for the practice condition compared to non-practice conditions, then researchers will include practice in Monitoring PD for Year 3.

In Year 3, a new cohort of schools will be randomly assigned to three conditions: a) standard classroom practices; b) Monitoring PD only (e.g., stand-alone PD); and c) Monitoring PD plus DMT. In particular, researchers will compare monitoring accuracy and use of effective formative assessments across conditions. If monitoring accuracy is greater for the stand-alone Monitoring PD condition, then researchers will offer intervention as a stand-alone PD. If monitoring accuracy is greater for the Monitoring add-on to DMT condition, then researchers will incorporate the Monitoring program into DMT. Finally, the team will integrate the findings from the three years of work into a manual for facilitators of Monitoring PD.

Control Condition: In Years 1 and 2, two control conditions will be included in the study—standard classroom practices and DMT-only. In Year 3, only one control condition will be included—standard classroom practice.

Key Measures: To gauge the effectiveness of program components and activities and to inform program revisions, researchers will gather data from two teacher self-report measures of attitudes and practice, observational coding of teacher practice, teacher-predicted and actual student scores on a math achievement test, and teacher feedback about the program. The Classroom Assessment Literacy Inventory (CALI) will be used to measure teachers' perception of their own ability to create and choose assessments that inform instructional strategies. The Conceptions of Assessment Inventory (CAI) measures teachers' self-reported attitudes toward formative assessments and practice. Self-report measures will be collected during the pre-summer-program, the post-summer-program, and at the end of the school year. Researchers will collect participant evaluations and will use observational coding of teacher formative assessment practices. At the end of each semester, teachers will predict their students' scores on a researcher-developed math test and researchers will collect and score students' actual performance on this test. Student scores on a state standardized test aggregated by teacher will also be used in the study.

Data Analytic Strategy: The accuracy of teachers' gauge of student math learning will be measured using the correlation of teacher-predicted scores and actual student scores on math achievement tests developed by the researchers. In years 1 and 2, planned comparisons will be conducted to compare monitoring accuracy of teachers. Researchers will use analysis of variance to examine the effect of the PDs on monitoring accuracy. Researchers will conduct data analyses comparing CALI and CAI scores before and after the summer institute training, scale scores on the observation measure across time, and monitor the accuracy of the chronicling conditions versus the non-chronicling condition. In year 3, multi-level modeling analyses will be used to test the effects of participation in Monitoring PD on student outcomes. Data for these analyses will be gathered at two levels: the student level and the teacher level.

Products and Publications

Journal article, monograph, or newsletter

Thiede, K. W., Brendefur, J. L., Osguthorpe, R. D., Carney, M. B., Bremner, A., Strother, S., ... and Jesse, D. (2015). Can Teachers Accurately Predict Student Performance?. Teaching and Teacher Education, 49, 36–44.