The study was conducted in Chicago Public Schools starting in the 2007–08 school year and continuing through the 2010–11 school year.
Cluster Randomized Controlled Trial
A total of 34 public elementary (grades K–8) schools in Chicago participated in the cluster randomized controlled trial part of the study. More than 90% of the students in these schools were African American, and more than 95% were eligible for free or reduced-price lunch. In spring 2007, 16 elementary schools were randomly assigned to begin Chicago TAP either in fall 2007 (eight schools in the Chicago TAP group [Cohort 1]) or in fall 2008 (eight schools in the comparison group [Cohort 2]). In spring 2009, 18 additional elementary schools were randomly assigned to begin Chicago TAP either in fall 2009 (nine schools in the Chicago TAP group [Cohort 3]) or in fall 2010 (nine schools in the comparison group [Cohort 4]). Thus, Cohort 2 served as a non-Chicago TAP comparison group for Cohort 1 during Cohort 1’s first year of implementation, and Cohort 4 served as a non-Chicago TAP comparison group for Cohort 3 during Cohort 3’s first year of implementation. Estimates of program impact after 1 year of implementation were then made by measuring the changes in student achievement for the Chicago TAP group of schools (Cohorts 1 and 3 pooled together) compared to the changes for both delayed implementation cohorts (Cohorts 2 and 4 pooled together). Students in grades 4–8 were included in the analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation: 1,717 students in the science achievement sample (808 Chicago TAP students and 909 comparison students), which is smaller than the others because standardized test data in science were available only for students in grades 4 and 7 and only for Cohorts 3 and 4; 7,661 students in the English language arts achievement sample (3,717 Chicago TAP students and 3,944 comparison students); and 7,656 students in the mathematics achievement sample (3,714 Chicago TAP students and 3,942 comparison students).
The quasi-experimental portion of the study included six purposively selected Chicago TAP schools in addition to the 34 randomly assigned Chicago TAP schools. For the quasi-experiment, Chicago TAP schools from all four cohorts were matched to other schools in the district that were not participating in Chicago TAP on measures such as school size, teacher retention, student race/ethnicity, student achievement, student poverty, student special education status, student language proficiency, and charter school status. The authors used a propensity score matching procedure where Chicago TAP schools were matched to their nearest five neighbors, with replacement. The resulting sample consisted of students in about 40 Chicago TAP schools and about 100 non-Chicago TAP comparison schools. The analytic samples for the quasi-experimental analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation were: 12,998 grade 4 and 7 students (2,464 Chicago TAP and 10,534 comparison) for science achievement; and 41,580 grade 4–8 students (8,097 Chicago TAP and 33,483 comparison) for both English language arts achievement and mathematics achievement.
Under TAP™, teachers can earn extra pay and responsibilities by being promoted to mentor or master teachers and can earn annual performance bonuses based on a combination of their value added to student achievement and observations of their classroom teaching. Unlike with the national TAP™ model, the program as implemented in Chicago Public Schools (called “Chicago TAP”) did not measure value-added performance at the individual teacher (or classroom) level; rather, value added was measured at the school level in 2007–08 and 2008–09 and at both the school- and school-grade team levels in 2009–10 and 2010–11. Additionally, the bonuses provided to teachers were typically smaller than the amounts prescribed by the national TAP™ model of $2,000 to $3,000 per teacher. In the first year of implementation, teachers in Cohorts 1, 2, and 3 (i.e., those implementing Chicago TAP in 2007–08 through 2009–10) received an average bonus of $1,100; teachers in Cohort 4 received an average bonus of $1,400 in 2010–11. Average bonuses increased to approximately $2,500 in the second and third years of implementation, and were $1,900 in the fourth year of implementation. Chicago TAP included weekly meetings of teachers and mentors, regular classroom observations by a school leadership team, and pay for principals who meet implementation benchmarks. Mentors received an additional $7,000 per year. Master teachers (called “lead teachers” in Chicago) received $15,000.
For the cluster randomized controlled trial portion of the study, comparison schools were in a “business-as-usual” condition for a year and subsequently participated in Chicago TAP. For the quasi-experimental portion of the study, comparison schools were in a “business-asusual” condition and did not receive Chicago TAP.
Student standardized test data on science (grades 4 and 7), English language arts (grades
4–8), and mathematics (grades 4–8) were obtained from Chicago Public Schools. For a more detailed description of these outcome measures, see Appendix B. The study also analyzed teacher retention at the school, teacher retention in the school district, and teacher attitudes. However, the teacher retention outcomes are rated does not meet WWC group design standards because equivalence of the analytic intervention and comparison groups is necessary and not demonstrated. Teacher attitudes were not included in this review because the outcomes fall outside of the domains of interest listed in the Teacher Training, Evaluation, and Compensation review protocol (version 3.1).
Support for implementation
The Chicago TAP model provides for observations of teachers by the principal, mentor teachers, and master teachers, all of whom undergo training and certification in using SKR rubric. SKR scores are based on observed classroom performance in four domains: designing and planning instruction, learning environment, instruction, and responsibilities.