Contract Open

Providing State Education Agencies with Needed Teacher Shortage Reporting Through Web Scraping

NCER

Program: Small Business Innovation Research

Award amount: $1,000,000

Project director: Lauren Dachille

Awardee:

Nimble Hiring, PBC

Year: 2024

Award period: 1 year 11 months (07/01/2024 - 06/30/2026)

Project type:

Phase II Development

Contract number: 91990024C0018

Purpose

Teacher shortages are widespread on a national level, with an estimated 200,000 positions either vacant or staffed by underqualified individuals. Schools with higher vacancy rates have worse working conditions, smaller salaries, fewer early career teachers, and are more likely to have lower levels of student achievement. In prior university-based studies, researchers developed a method to gather real-time reports of teacher shortages via web scraping of district job postings. While research demonstrated that the method was effective for providing information to State Education Agency (SEA) administrators to inform hiring needs and decisions, the method was not designed for use outside of the context of research. The new product will create a data dashboard for SEA administrators in different states to access real-time data about teacher shortages in their states and with information to inform hiring decisions.

Project Activities

During the Direct to Phase II project, the team will develop: the back-end infrastructure to automatically scrape data from publicly available sources on teacher shortages while prioritizing districts serving the largest proportion of students; a cloud-based data storage and processing system to organize and host the data; define a taxonomy and test models to categorize job postings by key areas; and a data dashboard for SEA administrators to present results and inform hiring decisions. After development is complete, the researchers will conduct a pilot study to inform hiring decisions over 4-months with 6 SEA administrators who are involved in teacher recruitment. Researchers will examine whether the product accurately presents information on district hires and vacancies using scraped state administrative data, whether users understand and find information in the dashboard easy to navigate, whether users interpret the information correctly, and whether the SEA administrators believe the use of the product results in time savings and improved decision making compared to prior tools.

People and institutions involved

IES program contact(s)

Edward Metz

IES Research Scientist

NCER

Products and publications

In prior university-based projects, researchers created a method to gather real-time reports of teacher shortages via web scraping of school system job postings. In research studies this method has been shown to be a valid indicator of on-to-ground hiring needs for different student populations. This project will develop a new education technology-based data dashboard for use by states which will provide SEA administrators real-time data about K-12 labor markets to inform hiring decisions to address teacher shortages and to quickly measure the impact of such interventions.