WWC review of this study

An Evaluation of the Chicago Teacher Advancement Program (Chicago TAP) after Four Years. Final Report

Glazerman, Steven; Seifullah, Allison (2012). Mathematica Policy Research, Inc. Retrieved from: https://eric.ed.gov/?id=ED530098

examining

34

Schools

, grades

4-8

TAP: The System for Teacher and Student Advancement Intervention Report - Teacher Training, Evaluation, and Compensation

Review Details

Reviewed: April 2018

No statistically significant positive
findings

Meets WWC standards with reservations

For:

TAP: The System for Teacher and Student Advancement Intervention Report - Teacher Training, Evaluation, and Compensation

Using:

Research Design:

Randomized Controlled Trial

Rating:

Meets WWC standards with reservations because it is a randomized control trial with cluster level inferences and joiners, but it demonstrates baseline equivalence.

This review may not reflect the full body of research evidence for this intervention.

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Please see the WWC summary of evidence for TAP: The System for Teacher and Student Advancement.

Findings

English language arts achievement outcomes—Indeterminate effect found for the domain
Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement index
Illinois Standards Achievement Test (ISAT): Reading Assessment	TAP: The System for Teacher and Student Advancement vs. Business as usual	1 Year	Grade: 4, 5, 6, 7, 8; 7,661 students	221.30	221.00	No	--
Show Supplemental Findings
Illinois Standards Achievement Test (ISAT): Reading Assessment	TAP: The System for Teacher and Student Advancement vs. Business as usual	1 Year	Grade: 4, 5, 6, 7, 8; 41,580 students	222.00	222.40	No	--

General Mathematics Achievement outcomes—Indeterminate effect found for the domain
Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement index
Illinois Standards Achievement Test (ISAT): Mathematics Assessment	TAP: The System for Teacher and Student Advancement vs. Business as usual	1 Year	Grade: 4, 5, 6, 7, 8; 7,656 students	233.40	234.30	No	--
Show Supplemental Findings
Illinois Standards Achievement Test (ISAT): Mathematics Assessment	TAP: The System for Teacher and Student Advancement vs. Business as usual	1 Year	Grade: 4, 5, 6, 7, 8; 41,580 students	235.90	235.50	No	--

Science Achievement outcomes—Indeterminate effect found for the domain
Outcome measure	Comparison	Period	Sample	Intervention mean	Comparison mean	Significant?	Improvement index	Evidence tier
Illinois Standards Achievement Test (ISAT): Science Assessment	TAP: The System for Teacher and Student Advancement vs. Business as usual	1 Year	Grade: 4, 7; 1,717 students	204.30	200.60	No	--

Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Sample Characteristics

Characteristics of study sample as reported by study author.

2% English language learners
Urban
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Illinois

Race
Black		90%
Other or unknown		10%
White		1%

Ethnicity
Hispanic		6%
Not Hispanic or Latino		94%

Study Details

Setting

The study was conducted in Chicago Public Schools starting in the 2007–08 school year and continuing through the 2010–11 school year.

Study sample

Cluster Randomized Controlled Trial A total of 34 public elementary (grades K–8) schools in Chicago participated in the cluster randomized controlled trial part of the study. More than 90% of the students in these schools were African American, and more than 95% were eligible for free or reduced-price lunch. In spring 2007, 16 elementary schools were randomly assigned to begin Chicago TAP either in fall 2007 (eight schools in the Chicago TAP group [Cohort 1]) or in fall 2008 (eight schools in the comparison group [Cohort 2]). In spring 2009, 18 additional elementary schools were randomly assigned to begin Chicago TAP either in fall 2009 (nine schools in the Chicago TAP group [Cohort 3]) or in fall 2010 (nine schools in the comparison group [Cohort 4]). Thus, Cohort 2 served as a non-Chicago TAP comparison group for Cohort 1 during Cohort 1’s first year of implementation, and Cohort 4 served as a non-Chicago TAP comparison group for Cohort 3 during Cohort 3’s first year of implementation. Estimates of program impact after 1 year of implementation were then made by measuring the changes in student achievement for the Chicago TAP group of schools (Cohorts 1 and 3 pooled together) compared to the changes for both delayed implementation cohorts (Cohorts 2 and 4 pooled together). Students in grades 4–8 were included in the analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation: 1,717 students in the science achievement sample (808 Chicago TAP students and 909 comparison students), which is smaller than the others because standardized test data in science were available only for students in grades 4 and 7 and only for Cohorts 3 and 4; 7,661 students in the English language arts achievement sample (3,717 Chicago TAP students and 3,944 comparison students); and 7,656 students in the mathematics achievement sample (3,714 Chicago TAP students and 3,942 comparison students). Quasi-experiment The quasi-experimental portion of the study included six purposively selected Chicago TAP schools in addition to the 34 randomly assigned Chicago TAP schools. For the quasi-experiment, Chicago TAP schools from all four cohorts were matched to other schools in the district that were not participating in Chicago TAP on measures such as school size, teacher retention, student race/ethnicity, student achievement, student poverty, student special education status, student language proficiency, and charter school status. The authors used a propensity score matching procedure where Chicago TAP schools were matched to their nearest five neighbors, with replacement. The resulting sample consisted of students in about 40 Chicago TAP schools and about 100 non-Chicago TAP comparison schools. The analytic samples for the quasi-experimental analysis of the impact of Chicago TAP teachers on student achievement for the first year of Chicago TAP implementation were: 12,998 grade 4 and 7 students (2,464 Chicago TAP and 10,534 comparison) for science achievement; and 41,580 grade 4–8 students (8,097 Chicago TAP and 33,483 comparison) for both English language arts achievement and mathematics achievement.

Intervention Group

Under TAP™, teachers can earn extra pay and responsibilities by being promoted to mentor or master teachers and can earn annual performance bonuses based on a combination of their value added to student achievement and observations of their classroom teaching. Unlike with the national TAP™ model, the program as implemented in Chicago Public Schools (called “Chicago TAP”) did not measure value-added performance at the individual teacher (or classroom) level; rather, value added was measured at the school level in 2007–08 and 2008–09 and at both the school- and school-grade team levels in 2009–10 and 2010–11. Additionally, the bonuses provided to teachers were typically smaller than the amounts prescribed by the national TAP™ model of $2,000 to $3,000 per teacher. In the first year of implementation, teachers in Cohorts 1, 2, and 3 (i.e., those implementing Chicago TAP in 2007–08 through 2009–10) received an average bonus of $1,100; teachers in Cohort 4 received an average bonus of $1,400 in 2010–11. Average bonuses increased to approximately $2,500 in the second and third years of implementation, and were $1,900 in the fourth year of implementation. Chicago TAP included weekly meetings of teachers and mentors, regular classroom observations by a school leadership team, and pay for principals who meet implementation benchmarks. Mentors received an additional $7,000 per year. Master teachers (called “lead teachers” in Chicago) received $15,000.

Comparison Group

For the cluster randomized controlled trial portion of the study, comparison schools were in a “business-as-usual” condition for a year and subsequently participated in Chicago TAP. For the quasi-experimental portion of the study, comparison schools were in a “business-asusual” condition and did not receive Chicago TAP.

Outcome descriptions

Student standardized test data on science (grades 4 and 7), English language arts (grades 4–8), and mathematics (grades 4–8) were obtained from Chicago Public Schools. For a more detailed description of these outcome measures, see Appendix B. The study also analyzed teacher retention at the school, teacher retention in the school district, and teacher attitudes. However, the teacher retention outcomes are rated does not meet WWC group design standards because equivalence of the analytic intervention and comparison groups is necessary and not demonstrated. Teacher attitudes were not included in this review because the outcomes fall outside of the domains of interest listed in the Teacher Training, Evaluation, and Compensation review protocol (version 3.1).

Support for implementation

The Chicago TAP model provides for observations of teachers by the principal, mentor teachers, and master teachers, all of whom undergo training and certification in using SKR rubric. SKR scores are based on observed classroom performance in four domains: designing and planning instruction, learning environment, instruction, and responsibilities.

Additional Sources

In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.

Glazerman, Steven; Seifullah, Allison. (2010). An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year Two Impact Report. Mathematica Policy Research, Inc.
Glazerman, S., McKie, A., & Carey, N. (2009). An evaluation of the Teacher Advancement Program (TAP) in Chicago: Year one impact report. Washington, DC: Mathematica Policy Research, Inc.