Co-Principal Investigators: Cowan, James; Theobald, Roddy; Abbott, Claire; Losee, Elizabeth; Smithney, Claire; Webb, Aubree

Purpose: Researchers propose a framework for informing revisions to state licensure testing requirements, which researchers refer to as outcomes-based licensure test standards. Researchers plan to link domain-level test information for individual teacher candidates from the Massachusetts Tests for Educator Licensure (MTEL) to various measures of teaching effectiveness in Massachusetts. This will allow researchers to test how different domains of content knowledge predict teacher and student outcomes for different kinds of teachers. States may be able to improve the correlation between test scores and teaching effectiveness by focusing their testing requirements on the content knowledge domains most associated with student learning. At the same time, if some domains are not predictive of student learning, states can revise their policies and eliminate these requirements. Finally, information about the predictive validity of different domains can inform future test development, in Massachusetts and nationally. In short, this approach allows policymakers to emphasize the aspects of teacher content knowledge that are most predictive of quality teaching and balance these against any downstream consequences on the composition of candidates who pass the test.

Project Activities: Researchers seek to answer four research questions (RQs) about licensure tests:

• RQ1: What are existing relationships between teacher content knowledge in specific domains—as measured by MTEL subarea and objective scores—and student achievement, and how do these relationships vary by subject and grade level?
• RQ2: What are existing relationships between teacher content knowledge in specific domains—as measured by MTEL subarea and objective scores—and teacher summative ratings, and how do these relationships vary by subject and grade level?
• RQ3: To what extent can the relationships between MTEL scale scores and student and teacher outcomes (described by RQs 1 and 2) be strengthened by differential weights on particular MTEL test objectives, subareas, or sections?
• RQ4: To what extent does changing the weights on different subareas and objectives affect the licensure test performance of different teacher subgroups and the predicted passing rates of these teacher subgroups?

Answers to RQs 1 and 2 will provide empirical evidence on aspects of teacher content knowledge that are predictive of teacher effectiveness (value-added to test scores in RQ1 and teacher evaluations in RQ2). The exploratory evidence for RQs 1 and 2 provides a foundation for the most novel component of the proposal: assessing the extent to which reweighting specific domains of teacher content knowledge might strengthen the connection between licensure test performance and instructional effectiveness (RQ3). We additionally plan to provide evidence based on historical data about how different licensure test policies might impact the composition of candidates who meet the state's licensure requirements (RQ4).

Products: Research activities will result in policy memos and presentations to the Massachusetts Department of Elementary and Secondary Education, policy presentations to researchers, and research publications.

Structured Abstract

Setting: The setting of the proposed research is the state of Massachusetts, which has the 17th largest public school system in the country and currently has about 1 million students and 70,000 teachers in its public schools. Massachusetts requires applicants for a prekindergarten (PreK)–12 educator license to pass MTEL tests in Communication and Literacy Skills (Reading and Writing) and in at least one additional PreK–12 academic subject area when available.

Sample: The primary sample will consist of all 73,558 teacher candidates who took at least one MTEL test between 2009 and 2016, as well as their future students in K–12 public schools.

Factors: This project will focus on applicant content knowledge and weights on subgroup of the Massachusetts licensure test items. Each test consists of one to three sections (multiple choice, short answer, and/or open response) that include different objectives (broad subject matter knowledge domains) arranged within different subareas (particularized tasks).

Research Design and Methods: Researchers will use ordinary least squares (OLS), logit, constrained linear regression, regularized regression to determine weights and predictive validity of the Massachusetts license test sub-scale.

Control Condition: Due to the nature of the study, there is no control condition.

Key Measures: The key measure of interest is MTEL licensure test performance on the various items, objectives, subareas, and sections of each test. Outcomes include student performance on the state's standardized tests and the summative performance rating of teachers under the state's evaluation framework.

Data Analytic Strategy: For RQ1 and RQ2, researchers estimate associations between licensure test domain and subarea scores and teacher outcomes using regression models common in the literature that control for student, classroom, and school context. For RQ3, researchers plan to first estimate individual teacher performance measures using the summative ratings and student achievement data, and then estimate optimal predictions of these performance measures using the MTEL testing data. Using the standardized teacher fixed effects and the MTEL objective scores, researchers will estimate optimal prediction weights via constrained linear regression. Given concerns about the precision of these estimates, researchers plan to supplement these analyses with regularization methods that penalize large weights on any one subsection. Finally, RQ4 will examine changes in the distribution of scale scores across candidate subgroups (e.g., by race and ethnicity) in the historical data according to these different reweighting schemes.

Related IES Projects: Shaping Teacher Quality and Student of Color Experience in Massachusetts: Alignment of Preparation and Licensure Systems with Teacher Effects on Student non-Test Outcomes (R305S210012); The Teacher Pipeline in Massachusetts: Connecting Pre-service Performance Measures to In-service Teacher Outcomes (R305H170025)