Co-Principal Investigators: Carrie Conaway (Massachusetts Department of Elementary and Secondary Education; ESE), James Cowan (AIR), Heather Peske (ESE)

Partner Institutions: Massachusetts Department of Elementary and Secondary Education (ESE)

Purpose: As part of its efforts to develop its teaching workforce, the Massachusetts Department of Elementary and Secondary Education (ESE) has teacher evaluation systems that span the entire teacher pipeline. These include surveys of supervising practitioners and teacher candidates during the preservice clinical internship; a new practice-based preparation program completion assessment (the Candidate Assessment of Practice, or CAP); surveys of first-year teachers and their principals; and the state's summative performance assessment of in-service teachers. Partners worked together to explore the predictive validity of these pre-service teaching measures and later in-service teaching outcomes and student outcomes. Partners examined the relations between program review scores across 5 organizational domains and other measures of pre-service training program "quality" as measured by stakeholder surveys and regression-adjusted in-service teacher evaluation ratings of pre-service graduates.

Partnership Activities: Personnel from ESE and the Center for the Analysis of Longitudinal Data in Education Research (CALDER), housed within the American Institutes for Research (AIR) jointly developed the research agenda, participated in monthly phone calls to monitor the progress of the project and create a shared understanding of how research findings inform ESE's policy priorities, and interpreted and disseminated research findings during annual advisory board meetings that bring key personnel from ESE and CALDER together with representatives from two different advisory groups that will support the work of the partnership.

Key Outcomes: The partnership generated evidence of the predictive validity of the State's pre-service teaching measures and later in-service teaching and student outcomes. The partnership work also led to preliminary evidence to shape the State's evaluation approach to pre-service approval, particularly around measures of pre-service training program quality. CALDER and ESE worked together to identify future research opportunities.

Setting: The project took place in Massachusetts.

Sample: Participants included all teacher candidates in Massachusetts teacher preparation programs and all students (950,000 enrolled per year) and teachers (70,000 on staff per year) in public schools in Massachusetts starting in school year 2014–2015 through the present.

Data Analytic Strategy: Partners estimated a simple duration model measuring the number of years before a teacher finds a teaching position, and then estimated OLS regressions in which the outcome is one of the in-service outcomes (i.e., probability of entry into the State's public teaching workforce; and formal teacher evaluations and value-added scores) and the independent variables of interest are the preservice measures (i.e., surveys of supervising practitioners and teacher candidates during the preservice clinical internship and a new practice-based preparation program completion assessment (the Candidate Assessment of Practice, or CAP). Partners also modeled each of the pre-service and in-service measures described above as a function of indicators for whether a teacher entered the teaching profession with a preliminary teaching license, and then supplemented these models with information on the institutions that teacher candidates attend.

Findings:

Partners found that teacher candidate performance on the Candidate Assessment of Performance (CAP) significantly predicted in-service summative performance evaluations the following year, over and above state's traditional licensure test scores. Partners found a positive and statistically significant correlation between teacher candidate test scores on the Massachusetts Tests for Educator Licensure (MTEL) and later teacher in-service performance ratings and value-added modeling of student test scores. (Chen, Cowan, Goldhaber, & Theobald, 2019).

Pre-service program review scores were positively correlated with pre-service completer perceptions of program quality (i.e., quality of preparation received) and in-service teacher evaluation ratings of pre-service graduates. However, program review scores were not statistically significantly correlated with perceptions of candidates, supervising practitioners, or hiring principals. (Comb, Cowan, Goldhaber, & Theobald, 2021).

Teacher candidates of color had lower first-time pass rates and were also less likely to retake licensure tests after failure compared to white teacher candidates. Partners did not find differential relations between MTEL scores and VAM of student test scores by teacher race. However, relations between MTEL scores and later teacher performance ratings were stronger for teachers of color than for White teachers.  (Cowan, Goldhaber, Jin, & Theobald 2020)

Related IES Projects: Shaping Teacher Quality and Student of Color Experience in Massachusetts: Alignment of Preparation and Licensure Systems with Teacher Effects on Student non-Test Outcomes (R305S210012); Assessing the Potential of Outcomes-Based Licensure Test Standards (R305A220058)

Publications and Products

Chen, B., Cowan, J., Goldhaber, D., & Theobald, R. (2019). From the clinical experience to the classroom: Assessing the predictive validity of the Massachusetts Candidate Assessment of Performance. CALDER Working Paper No. 221-0819.

Comb, M., Cowan, J., Goldhaber, D., Jin, Z., & Theobald, R. (2021). State Ratings of Educator Preparation Programs: Connecting Program Review to Teacher Effectiveness. CALDER Working Paper No. 249-0321

Cowan, J., Goldhaber, D., Jin,?Z.,& Theobald, R (2020). Teacher Licensure Tests: Barrier or Predictive Tool? CALDER Working Paper No. 245-1020