Co-Principal Investigator: Jones, Angela

Partner: Spokane Public Schools

Education Issue: District hiring could be an important policy lever with the potential to significantly influence the effectiveness of the teaching workforce in improving student academic outcomes. However, the relation between applicant data and future teacher performance is not well understood. Specifically, empirical findings regarding the relation between teacher credentials (e.g., licensure, certification, graduating institution, degree/major) and later student outcomes are mixed and at times contradictory. Previous analyses, however, have focused on data integrated into the state longitudinal databases and have not included information about teacher applicants collected and recorded by some local school systems. Moreover, previous analyses have not included the observations of local hiring officials and these observations of prospective teacher characteristics might be important predictors of teacher effectiveness in the future.

Partnership Goal: In this project, the partnership of the University of Washington Center for Education Data and Research (CEDR) and Spokane Public Schools (Spokane) studied how well the information collected during the teacher hiring process predicts future teacher performance, and whether alternative means of assessing teacher applicants to the district might lead to better hiring decisions.

The primary aims of the partnership were to better understand the relation between applicant data and teacher performance; to determine where hiring processes succeed and fail to identify the best teacher applicants; and finally, to create a foundation from which to develop a future research grant proposal to develop and analyze the effectiveness of new applicant assessment tools.

Partners and Partnership Activities: CEDR is housed at the University of Washington and is directed by Dan Goldhaber (PI). Its mission is to conduct independent education research that provides policymakers and practitioners the scientific evidence necessary to make decisions that improve education outcomes. CEDR's research work generally addresses the following areas of education policy with the intent to facilitate evidence-based policy and practice: school and teacher effectiveness, education accountability and governance, and teacher labor markets. Empirical work at CEDR focuses on the analysis of longitudinal data that associates teachers with individual students. CEDR has currently partnered with 11 Washington State school districts through SDPI, including Spokane Public Schools.

SPS is the largest school district in eastern Washington, and the second largest in the state. The district consists of 34 elementary, 6 middle, and 5 high schools. In the 2012-13 school year, the district employed 1,758 teachers who instruct 28,768 students. During the previous 5 years, Spokane has hired 988 new teachers from a pool of 5,840 applicants. The project involved the Employment Services Department of Spokane Human Resources. Employment Services, directed by Angela Jones (Co-PI), oversees the school district's recruitment, screening, and hiring processes, including the assessment of protocols used during the hiring process, and the training of administrators responsible for hiring. The Department is also responsible for the maintenance of the administrative datasets that will be used to conduct the proposed analyses.

This study built on an existing researcher-practitioner partnership, formed under CEDR's Strategic District Partnership Initiative. The partnership was strengthened by increasing Spokane's capacity to approach educational interventions through data analysis, and by the framework that was formed to guide the project. The partnership assembled an Advisory Board of researcher and practitioner experts, and at several points convened meetings of Spokane stakeholders to discuss data, methods of analysis and interpretation of results, as well as to obtain feedback. The partnership studied the relation between information collected from teacher applicants and later teacher performance.

Setting: This study was conducted onsite in Spokane and at the Center for Education Data and Research in Seattle.

Sample: The project used data from local and state records on teacher applicants to Spokane (from SY 2007–08 through 2012–13) and their students.

Data Analytic Strategy: The analysis used data about teacher applicants combined with information drawn from statewide longitudinal datasets from Washington State, which included data on student and teacher performance. Combining these datasets allowed the inclusion of the performance of applicants who were not hired by Spokane but instead were hired by other school districts in the State.

They collected data generated by Spokane Public Schools online applicant management system, applicant scores on screening instruments used by the district, and state-wide administrative data including standardized test-based measures of teacher performance, attrition, absences, and licensure exam scores.

These research aims were addressed in four steps: (1) assessment of variation in applicant data collected; (2) assessment of variation in all variables among applicant subgroups defined by progress through the hiring process; (3) assessment of variation in independent variables among teachers of varying performance level; and (4) multivariate regression analysis of the link between applicant data teacher quality.

Additionally, they performed an analysis of inter-rater reliability of the screening instrument used to determine which applicants would receive

Key Outcomes: This partnership produced an article in a peer-reviewed journal, a presentation, and a IES funded exploration project (R305A170060). The findings of this project, which informed decision-making in Spokane Public Schools, include the following:

• Two of the screening instruments used by SPS were predictive of value-added measures of teacher performance and attrition but not absences
• Soliciting letters of recommendation directly from references promoted greater candor in the recommendation
• Applicant screening forms used by SPS tallied by hand had an 18.8% error rate
• There is low inter-rater reliability of the 60-point screening instrument used by the district to determine which applicants progress to interviews

Related Projects: Learning From the Source: Can We Elicit Better Applicant Information Directly From Professional References? (R305A170060); What is the Value of Apprenticeship for Teachers? Linking Preservice Mentor Quality to Inservice Teacher and Student Outcomes (R305A180023)

Products and Publications

ERIC Citations:Find available citations in ERIC for this award here.

Journal articles

Goldhaber, D., Grout, C., & Huntington-Klein, N. (2017). Screen twice, cut once: Assessing the predictive validity of teacher selection tools.Education Finance and Policy, 12(2): 197–223. Full text

Martinková, P., Goldhaber, D., & Erosheva, E. (2018). Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability. PloS ONE, 13(10), e0203002. doi.org/10.1371/journal.pone.0203002 Full text