Purpose: The collection of letters of recommendation from professional references (PRs) is a standard procedure during the teacher selection process in most school districts and enhancements of this process might help a school district to make better hiring decisions. In this study, PRs will be asked to rate teaching applicants relative to their peers on a series of criteria demonstrated to be predictive of positive teacher outcomes and student achievement. Researchers will explore how these ratings relate to later teaching outcomes and student achievement. The research team will also explore how the relation between ratings and outcomes may vary according to applicant type (e.g., novice vs. experienced), job type (e.g., elementary vs. middle school), and PR type (e.g., principal vs. university professor). Findings from this project could lay the foundation for understanding the insights about teaching applicants that could be gained through PRs and could lead to the development of a tool that could be reliably and efficiently used by any school district.

Project Activities: Researchers will use administrative data from local and state records to explore how professional reference ratings of the performance of teaching applicants correlate with later teaching and student outcomes. Then, researchers will randomly assign hiring personnel to receive or not receive the professional reference ratings to examine extent to which the provision of PR ratings data influences hiring decisions.

Products: Researchers will produce preliminary evidence of potentially promising teaching candidate screening procedures and peer-reviewed publications.

Structured Abstract

Setting: This project will take place in schools located in a large district consisting of urban, suburban, and rural communities in eastern Washington State.

Sample: The project will use administrative data from local and state records on more than 3000 teaching applicants for over 600 teaching positions (from SY 2015–16 through 2019–20) and (if hired within the State) their students.

Intervention: Malleable factors are professional reference ratings of the performance of teaching applicants, applicant type (e.g., novice vs. experienced), job type (e.g., elementary vs. middle school), and PR type (e.g., principal vs. university professor). These factors could lead to the development of a tool that could be reliably and efficiently used by any school district.

Research Design and Methods: This study will employ a correlational research design to examine the predictive validity of PRs' assessments of applicants and potential moderation by applicant type (e.g., novice vs. experienced), job type (e.g., elementary vs. middle school), and PR type (e.g., principal vs. university professor). A randomized controlled research design will be used to examine extent to which the provision of PR ratings data influences hiring decisions.

Control Condition: In the control condition, the applicant online profile will not contain PR ratings and instead contain a "censored report" with the following text: "For research purposes Professional Reference ratings have been blinded for this applicant. Which applicants have blinded results is determined at random and is not indicative of the quality of the applicant."

Key Measures: Primary predictor measures include applicant background information (e.g., degree, GPA, professional experience), PR ratings, and applicant scores on a 60-point rubric used by district hiring officials to screen applicants. Outcome measures include value-added modeling scores based on student achievement scores in mathematics and reading, observational evaluation scores on the Marzano-based rubric, and teacher retention at the school, district, and state levels.

Data Analytic Strategy: Researchers will conduct descriptive analyses of the correlations and distributions of PR ratings, applicant characteristics, PR characteristics, job characteristics, and teacher outcomes. Researchers will also conduct exploratory factor analysis of the PR ratings data to understand the latent construct of the data and the relationship between observed ratings and latent factors. The relation between the PR ratings data and each teacher outcome will be examined using multivariate regression analysis. To examine the relation between PR ratings and applicant scores researchers will use regression analysis in the context of random provision of applicant data to hiring officials.

Related IES Projects: Applicants at the Doorstep: Improving Hiring Practices through a Better Understanding of the Link Between Applicant Information and Teacher Quality (R305H130030)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications

Journal articles

Goldhaber, D., Grout, C., Wolff, M., & Martinková, P. (2021). Evidence on the dimensionality and reliability of professional references' ratings of teacher applicants. Economics of Education Review, 83, 102130.