Grant Open

Assessing the Potential of Outcomes-Based Licensure Test Standards

NCER

Program: Education Research Grants

Program topic(s): Teaching, Teachers, and the Education Workforce

Award amount: $797,814

Principal investigator: Dan Goldhaber

Awardee:

American Institutes for Research (AIR)

Year: 2022

Project type:

Exploration

Award number: R305A220058

Purpose

Researchers propose a framework for informing revisions to state licensure testing requirements, which researchers refer to as outcomes-based licensure test standards. Researchers plan to link domain-level test information for individual teacher candidates from the Massachusetts Tests for Educator Licensure (MTEL) to various measures of teaching effectiveness in Massachusetts. This will allow researchers to test how different domains of content knowledge predict teacher and student outcomes for different kinds of teachers. States may be able to improve the correlation between test scores and teaching effectiveness by focusing their testing requirements on the content knowledge domains most associated with student learning. At the same time, if some domains are not predictive of student learning, states can revise their policies and eliminate these requirements. Finally, information about the predictive validity of different domains can inform future test development, in Massachusetts and nationally. In short, this approach allows policymakers to emphasize the aspects of teacher content knowledge that are most predictive of quality teaching and balance these against any downstream consequences on the composition of candidates who pass the test.

Project Activities

Researchers seek to answer four research questions (RQs) about licensure tests:

Structured Abstract

Setting

The setting of the proposed research is the state of Massachusetts, which has the 17th largest public school system in the country and currently has about 1 million students and 70,000 teachers in its public schools. Massachusetts requires applicants for a prekindergarten (PreK)–12 educator license to pass MTEL tests in Communication and Literacy Skills (Reading and Writing) and in at least one additional PreK–12 academic subject area when available.

Sample

The primary sample will consist of all 73,558 teacher candidates who took at least one MTEL test between 2009 and 2016, as well as their future students in K-12 public schools.

Factors

This project will focus on applicant content knowledge and weights on subgroup of the Massachusetts licensure test items. Each test consists of one to three sections (multiple choice, short answer, and/or open response) that include different objectives (broad subject matter knowledge domains) arranged within different subareas (particularized tasks).

Research design and methods

Researchers will use ordinary least squares (OLS), logit, constrained linear regression, regularized regression to determine weights and predictive validity of the Massachusetts license test sub-scale.

Control condition

Due to the nature of the study, there is no control condition.

Key measures

The key measure of interest is MTEL licensure test performance on the various items, objectives, subareas, and sections of each test. Outcomes include student performance on the state's standardized tests and the summative performance rating of teachers under the state's evaluation framework.

Data analytic strategy

For RQ1 and RQ2, researchers estimate associations between licensure test domain and subarea scores and teacher outcomes using regression models common in the literature that control for student, classroom, and school context. For RQ3, researchers plan to first estimate individual teacher performance measures using the summative ratings and student achievement data, and then estimate optimal predictions of these performance measures using the MTEL testing data. Using the standardized teacher fixed effects and the MTEL objective scores, researchers will estimate optimal prediction weights via constrained linear regression. Given concerns about the precision of these estimates, researchers plan to supplement these analyses with regularization methods that penalize large weights on any one subsection. Finally, RQ4 will examine changes in the distribution of scale scores across candidate subgroups (e.g., by race and ethnicity) in the historical data according to these different reweighting schemes.

People and institutions involved

IES program contact(s)

Wai-Ying Chow

Education Research Analyst

NCER

Products and publications

Products: Research activities will result in policy memos and presentations to the Massachusetts Department of Elementary and Secondary Education, policy presentations to researchers, and research publications.

Supplemental information

Co-Principal Investigators: Cowan, James; Theobald, Roddy; Abbott, Claire; Losee, Elizabeth; Smithney, Claire; Webb, Aubree

RQ1: What are existing relationships between teacher content knowledge in specific domains—as measured by MTEL subarea and objective scores—and student achievement, and how do these relationships vary by subject and grade level?
RQ2: What are existing relationships between teacher content knowledge in specific domains—as measured by MTEL subarea and objective scores—and teacher summative ratings, and how do these relationships vary by subject and grade level?
RQ3: To what extent can the relationships between MTEL scale scores and student and teacher outcomes (described by RQs 1 and 2) be strengthened by differential weights on particular MTEL test objectives, subareas, or sections?
RQ4: To what extent does changing the weights on different subareas and objectives affect the licensure test performance of different teacher subgroups and the predicted passing rates of these teacher subgroups?

Answers to RQs 1 and 2 will provide empirical evidence on aspects of teacher content knowledge that are predictive of teacher effectiveness (value-added to test scores in RQ1 and teacher evaluations in RQ2). The exploratory evidence for RQs 1 and 2 provides a foundation for the most novel component of the proposal: assessing the extent to which reweighting specific domains of teacher content knowledge might strengthen the connection between licensure test performance and instructional effectiveness (RQ3). We additionally plan to provide evidence based on historical data about how different licensure test policies might impact the composition of candidates who meet the state's licensure requirements (RQ4).

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

Assessing the Potential of Outcomes-Based Licensure Test Standards

Purpose

Project Activities

Structured Abstract

Setting

Sample

Research design and methods

Control condition

Key measures

Data analytic strategy

People and institutions involved

IES program contact(s)

Wai-Ying Chow

Products and publications

Supplemental information

Questions about this project?

Questions about this project?

You may also like

What Works Clearinghouse Supporting and Analyzing ...

Reduced calendar years: How are they implemented, ...

Consistent Implementation of a Multi-Tiered System...