Skip Navigation
Middle School Mathematics Professional Development Impact Study Findings After the Second Year of Implementation

NCEE 2011-4024
May 2011

Exploratory Analyses

We conducted several additional analyses, extending the exploratory analyses conducted for the Interim Report, and using the added power of a "pooled" sample of teachers. This pooled sample comprises three mutually exclusive and collectively exhaustive groups of teachers: teachers who were in the first-year impact analysis sample only (from all 12 districts); teachers who were in the second-year impact analysis sample only (from the 6 two-year districts); and teachers who were in both impact analysis samples. Teachers who were in both impact analysis samples (also from the 6 two-year districts) are included in the pooled sample twice, once using their first-year outcomes, and once using their second-year outcomes, controlling for their knowledge scores at the end of the first year/beginning of the second year. We also constructed a pooled sample of students that includes students who were in the first-year impact analysis sample (from all 12 districts) and students who were in the second-year impact sample (from 6 two-year districts).12

  • One-year effects of PD on teacher knowledge. The estimated effects of one year of PD on teacher knowledge total score and CK for the pooled sample were not statistically significant. However, the estimated average effect of one year of the PD program on SK using the pooled sample was statistically significant (effect size = 0.28, p = 0.02). 13
  • Average effect of PD on student achievement. Different groups of students experienced the effect of the PD in each year of the study. The estimated average effect of the PD on student achievement using the pooled sample was not found to be statistically significant.
  • Results by provider. We also used the pooled sample to examine the impact of the PD program separately for the two PD providers, America's Choice and Pearson Achievement Solutions. These analyses did not indicate significant effects of the PD program on teacher knowledge or student achievement for either provider.
  • Baseline teacher knowledge. Similarly, we drew on the pooled analysis sample to examine whether the PD program may have been more or less effective for teachers who began the study with different levels of baseline knowledge. We hypothesized that teachers with high levels of baseline knowledge may have found the PD too easy; teachers with low levels of baseline knowledge may have found the PD too hard. The analyses did not show a statistically significant association between teachers' initial knowledge levels and treatment-control differences in teacher knowledge or student achievement outcomes.
  • Baseline student achievement. We also drew on the pooled sample to examine whether the PD may have been more or less effective for students who began the year with different levels of baseline achievement. Students with different initial achievement levels may have had different needs. The analyses indicated that the PD program did not appear to be more or less effective for students with low or high initial achievement.
  • Teacher knowledge and student achievement. Finally, we drew on the pooled analysis sample to examine whether the study's measure of teacher knowledge was associated with student achievement as was hypothesized in the study's theory of action. Correlational analyses show a statistically significant positive association between the teacher knowledge total score and the student achievement total score of 0.05 (p-value = 0.02) and between the teacher knowledge total score and the student Fractions and Decimals Score of 0.05 (p-value < 0.01). 14

Top

12 The "pooled" sample of teachers used in the per-year effect analyses includes 138 teachers who were in the first-year impact sample only, 38 teachers who were in the second-year impact sample only, and 51 teachers who were in both the first- and second-year impact samples. Since the students in each year of the study represented the teachers' current seventh-grade students, there was no overlap between the first- and second-year student samples in the pooled sample.
13 The effect of one year of PD was calculated as the average of the one-year effect of the first year of PD and the additional one-year effect of the second year of PD.
14 To examine the relationship between teacher knowledge and student achievement, we incorporated the teacher knowledge total score in the impact model in place of the treatment status indicator. Separate analyses were also conducted using the CK and SK subscores rather than the total score. We then examined the estimated coefficients for each of the teacher knowledge scores and calculated the statistical significance of the coefficients using a two-tailed t-test.