WWC review of this study

An evaluation of the Chicago Teacher Advancement Program (Chicago TAP) after four years.

Glazerman, S., & Seifullah, A. (2012). Final report. Washington, DC: Mathematica Policy Research, Inc. http://files.eric.ed.gov/fulltext/ ED530098.pdf. Retrieved from: https://eric.ed.gov/?id=ED530098

  • Randomized Controlled Trial
     examining 
    34
     Schools
    , grades
    4-8

Reviewed: April 2018

No statistically significant positive
findings
Meets WWC standards with reservations
English language arts achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Reading Assessment

TAP: The System for Teacher and Student Advancement vs. Business as usual

1 Year

Grade: 4, 5, 6, 7, 8;
7,661 students

221.30

221.00

No

--
More Outcomes
Show Supplemental Findings

Illinois Standards Achievement Test (ISAT): Reading Assessment

TAP: The System for Teacher and Student Advancement vs. Business as usual

1 Year

Grade: 4, 5, 6, 7, 8;
41,580 students

222.00

222.40

No

--
General Mathematics Achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Mathematics Assessment

TAP: The System for Teacher and Student Advancement vs. Business as usual

1 Year

Grade: 4, 5, 6, 7, 8;
7,656 students

233.40

234.30

No

--
More Outcomes
Show Supplemental Findings

Illinois Standards Achievement Test (ISAT): Mathematics Assessment

TAP: The System for Teacher and Student Advancement vs. Business as usual

1 Year

Grade: 4, 5, 6, 7, 8;
41,580 students

235.90

235.50

No

--
Science achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Science Assessment

TAP: The System for Teacher and Student Advancement vs. Business as usual

1 Year

Grade: 4, 7;
1,717 students

204.30

200.60

No

--

Characteristics of study sample as reported by study author.


  • 2% English language learners

  • 95% Free or reduced price lunch
  • Race
    Black
    90%
    Not specified
    10%
    White
    1%
  • Ethnicity
    Hispanic
    6%
    Not Hispanic
    94%

  • Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    Illinois

Setting

The study was conducted in Chicago Public Schools starting in the 2007–08 school year and continuing through the 2010–11 school year.

Study sample

Cluster Randomized Controlled Trial A total of 34 public elementary (grades K–8) schools in Chicago participated in the cluster randomized controlled trial part of the study. More than 90% of the students in these schools were African American, and more than 95% were eligible for free or reduced-price lunch. In spring 2007, 16 elementary schools were randomly assigned to begin Chicago TAP either in fall 2007 (eight schools in the Chicago TAP group [Cohort 1]) or in fall 2008 (eight schools in the comparison group [Cohort 2]). In spring 2009, 18 additional elementary schools were randomly assigned to begin Chicago TAP either in fall 2009 (nine schools in the Chicago TAP group [Cohort 3]) or in fall 2010 (nine schools in the comparison group [Cohort 4]). Thus, Cohort 2 served as a non-Chicago TAP comparison group for Cohort 1 during Cohort 1’s first year of implementation, and Cohort 4 served as a non-Chicago TAP comparison group for Cohort 3 during Cohort 3’s first year of implementation. Estimates of program impact after 1 year of implementation were then made by measuring the changes in student achievement for the Chicago TAP group of schools (Cohorts 1 and 3 pooled together) compared to the changes for both delayed implementation cohorts (Cohorts 2 and 4 pooled together). Students in grades 4–8 were included in the analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation: 1,717 students in the science achievement sample (808 Chicago TAP students and 909 comparison students), which is smaller than the others because standardized test data in science were available only for students in grades 4 and 7 and only for Cohorts 3 and 4; 7,661 students in the English language arts achievement sample (3,717 Chicago TAP students and 3,944 comparison students); and 7,656 students in the mathematics achievement sample (3,714 Chicago TAP students and 3,942 comparison students). Quasi-experiment The quasi-experimental portion of the study included six purposively selected Chicago TAP schools in addition to the 34 randomly assigned Chicago TAP schools. For the quasi-experiment, Chicago TAP schools from all four cohorts were matched to other schools in the district that were not participating in Chicago TAP on measures such as school size, teacher retention, student race/ethnicity, student achievement, student poverty, student special education status, student language proficiency, and charter school status. The authors used a propensity score matching procedure where Chicago TAP schools were matched to their nearest five neighbors, with replacement. The resulting sample consisted of students in about 40 Chicago TAP schools and about 100 non-Chicago TAP comparison schools. The analytic samples for the quasi-experimental analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation were: 12,998 grade 4 and 7 students (2,464 Chicago TAP and 10,534 comparison) for science achievement; and 41,580 grade 4–8 students (8,097 Chicago TAP and 33,483 comparison) for both English language arts achievement and mathematics achievement.

Intervention Group

Under TAP™, teachers can earn extra pay and responsibilities by being promoted to mentor or master teachers and can earn annual performance bonuses based on a combination of their value added to student achievement and observations of their classroom teaching. Unlike with the national TAP™ model, the program as implemented in Chicago Public Schools (called “Chicago TAP”) did not measure value-added performance at the individual teacher (or classroom) level; rather, value added was measured at the school level in 2007–08 and 2008–09 and at both the school- and school-grade team levels in 2009–10 and 2010–11. Additionally, the bonuses provided to teachers were typically smaller than the amounts prescribed by the national TAP™ model of $2,000 to $3,000 per teacher. In the first year of implementation, teachers in Cohorts 1, 2, and 3 (i.e., those implementing Chicago TAP in 2007–08 through 2009–10) received an average bonus of $1,100; teachers in Cohort 4 received an average bonus of $1,400 in 2010–11. Average bonuses increased to approximately $2,500 in the second and third years of implementation, and were $1,900 in the fourth year of implementation. Chicago TAP included weekly meetings of teachers and mentors, regular classroom observations by a school leadership team, and pay for principals who meet implementation benchmarks. Mentors received an additional $7,000 per year. Master teachers (called “lead teachers” in Chicago) received $15,000.

Comparison Group

For the cluster randomized controlled trial portion of the study, comparison schools were in a “business-as-usual” condition for a year and subsequently participated in Chicago TAP. For the quasi-experimental portion of the study, comparison schools were in a “business-asusual” condition and did not receive Chicago TAP.

Outcome descriptions

Student standardized test data on science (grades 4 and 7), English language arts (grades 4–8), and mathematics (grades 4–8) were obtained from Chicago Public Schools. For a more detailed description of these outcome measures, see Appendix B. The study also analyzed teacher retention at the school, teacher retention in the school district, and teacher attitudes. However, the teacher retention outcomes are rated does not meet WWC group design standards because equivalence of the analytic intervention and comparison groups is necessary and not demonstrated. Teacher attitudes were not included in this review because the outcomes fall outside of the domains of interest listed in the Teacher Training, Evaluation, and Compensation review protocol (version 3.1).

Support for implementation

The Chicago TAP model provides for observations of teachers by the principal, mentor teachers, and master teachers, all of whom undergo training and certification in using SKR rubric. SKR scores are based on observed classroom performance in four domains: designing and planning instruction, learning environment, instruction, and responsibilities.

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

  • Glazerman, S., & Seifullah, A. (2010). An evaluation of the Teacher Advancement Program (TAP) in Chicago: Year two impact report. Washington, DC: Mathematica Policy Research, Inc.

  • Glazerman, S., McKie, A., & Carey, N. (2009). An evaluation of the Teacher Advancement Program (TAP) in Chicago: Year one impact report. Washington, DC: Mathematica Policy Research, Inc.

Reviewed: February 2013

At least one finding shows moderate evidence of effectiveness
At least one statistically significant positive finding
Meets WWC standards with reservations
English language arts achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Reading Assessment

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

1 Year

Grade: 4, 5, 6, 7, 8;
7,661 students

221.30

221.00

No

--
General Mathematics Achievement outcomes—Indeterminate effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Mathematics Assessment

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

one year posttest

Grade 4-8 students;
7,656 students

233.40

234.30

No

--
Science achievement outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Illinois Standards Achievement Test (ISAT): Science Assessment

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

one year posttest

Grade 4 and 7 students;
1,717 students

204.30

200.60

No

--
Teacher retention outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Teacher retention rate

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

three years (fall 2007-fall 2010)

Overall;
781 students

0.67

0.56

Yes

 
 
12
 
More Outcomes

Teacher retention rate

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

One year

Overall;
2,694 students

0.81

0.81

No

--

Teacher retention rate

Chicago Public Schools’ Teacher Advancement Program (Chicago TAP) vs. Business as usual

two years (fall 2008-fall 2010)

Overall;
1,879 students

0.71

0.68

No

--

Characteristics of study sample as reported by study author.


  • 2% English language learners

  • 95% Free or reduced price lunch
  • Race
    Black
    93%
    White
    1%
  • Ethnicity
    Hispanic
    6%
    Not Hispanic
    94%

  • Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    Illinois
 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

loading
back to top