Co-Principal Investigator: Montano, Matt; Mariano, Louis; Schwartz, Heather Schwartz; Schweig, Jonathan; Goldschmidt, Pete

Partner Institution: New Mexico Public Education Department

Purpose: Twenty-six States and the District of Columbia use high-stakes teacher evaluation systems to assign teachers who are rated "ineffective" and "minimally effective" to mandatory professional growth plans (PGPs). However, evidence on the impact of these policies on student academic and socio-behavioral outcomes is limited. Researchers will examine the impact of teachers being assigned PGPs on the outcomes of K–12 students and their teachers throughout the State of New Mexico. Researchers will also examine the impact of the "minimally effective" label on teacher and student outcomes.

Project Activities: Researchers will use teaching performance and student achievement data from administrative data systems as well as student surveys to estimate the impact of teachers being assigned PGPs on the outcomes of students and teachers in K–12 public and charter schools in New Mexico. The analysis will be performed using a "fuzzy" Regression Discontinuity (RD) design. Assuming that teachers (and determinants of student outcomes) immediately above and below the cutoff score for mandatory PGP assignment are similar to one another, researchers will compare the outcomes of teachers rated immediately above the cutoff to those of teachers rated immediately below the cutoff. Researchers will conduct complementary analyses to examine the impact of being rated "minimally effective" (separate from the PGPs) and PGP dosage (e.g., proportion of PGP completed; number of PGP elements; and supports received by frequency, location, and mode of delivery). Researchers will also conduct psychometric analysis of the NMTEACH measures, an implementation analysis and a cost analysis of the PGP policy.

Products: Researchers will produce evidence of the efficacy of mandatory professional growth plan assignments for teachers and students in K–12 public and charter schools in New Mexico (NM). This project will also result in peer-reviewed publications.

Structured Abstract

Setting: This project will take place in K–12 public and charter schools in throughout the State of New Mexico.

Sample: The sample will include 21,000 educators who taught students in grades K–12 in the 2015–2016 to 2019–2020 school years. These teachers taught approximately 335,000 students per year, the majority of whom were Hispanic and from low-income households.

Intervention: The proposed study is based on New Mexico's teacher evaluation system called NMTEACH. The three primary components of this system are:

• tests of student achievement improvement such as value-added scores on state and end-of-course exams (50% of the overall evaluation);
• state-developed teacher observation rubric (25% of the evaluation);
• locally selected multiple measures from a menu of options approved by the state such as teacher attendance or student surveys (25% of the evaluation).

The components are weighted and combined to produce a summative evaluation score. Using this score, teachers are placed into five effectiveness categories—"ineffective," "minimally effective," "effective," "highly effective," and "exemplary". Teachers labeled "ineffective" or "minimally effective" are required to have professional growth plans (PGPs).

The mandatory PGP policy requires the teacher and formal evaluator to complete a document listing the areas of unsatisfactory performance, the goals for improvement, specific and measurable outcomes that will be monitored over the course of the review period, the timeline and frequency for tracking outcomes, and the resources that the teacher is suggested to consult to improve their practice. Outcomes are tracked for 90 days, and the teacher is expected to show improvement.

Research Design and Methods: Using a Regression Discontinuity (RD) design, researchers will analyze teaching performance and student achievement data from administrative data systems as well as student surveys to estimate the impact of teachers being assigned PGPs on the outcomes of teachers and students. A "sharp" RD design would not be appropriate in this context, because some teachers with ratings of "effective" or higher may be placed on a PGP due to district-level policy for individual ratings on a classroom observation score. However, being labeled "ineffective" or "minimally effective" substantially increases the chances of a teacher being assigned a PGP. Therefore, researchers will use a "fuzzy" design. Assuming that teachers (and determinants of student outcomes) immediately above and below the cutoff score for mandatory PGP assignment are similar to one another, researchers will compare the outcomes of teachers rated immediately above the cutoff to those of teachers rated immediately below the cutoff. Researchers will also examine implementation of the policy throughout the state using district personnel interviews and teacher surveys, conduct psychometric analyses of the NMTEACH measures, conduct supplementary analyses related to the PGP policy, and conduct a cost analysis.

Control Condition: In the comparison condition, teachers are not labeled "ineffective" or "minimally effective" are not mandated to have professional growth plans.

Key Measures: Student education outcome measures include the Partnership for Assessment of Readiness for College and Careers (PARCC) and end-of-course exams in math, ELA, science and social studies. Other measures include academic behaviors, student engagement and mindset as measured by validated survey instruments. Teacher measures include summative ratings on the NMTEACH Educator Effectiveness system. Researchers will measure the implementation of the intervention using district personnel interviews and teacher surveys.

Data Analytic Strategy: The primary strategy is based on a "fuzzy" RD design. Researchers will conduct additional regression discontinuity and GAMM analyses to provide context on the influence of labeling and dosage on PGP effects. Researchers will also examine the role of moderating variables, such as the role of opportunities for professional development in PGPs impact on student outcomes.