At REL Northeast & Islands, we are charged by the Institute of Education Sciences to work with educators, school leaders, policymakers, and others to use data and evidence to improve student outcomes. In doing this, we are encouraged—in fact, required—to work in close partnership with regional stakeholders to ensure the research we conduct and the training and technical support we offer are of use and relevant to practitioners in our communities.

I was fortunate to work on a project that serves as an example of how REL researchers collaborate with regional partners. I was part of a REL Northeast & Islands team that developed the “Home Language Survey Data Quality Self-Assessment,” an IES tool that can help state agencies of education (SEAs) improve the quality of data on English learners (ELs) that districts collect in their home language survey.

To construct the 44-item, 15-minute self-assessment, REL colleagues Susan Henry, Maria-Paz Avery, Caroline Parker, Erin Stafford, and I drew on extensive research on data quality and invited state- and district-level stakeholders to identify issues that may detract from the quality of data they receive from their home language survey. Because we engaged stakeholders throughout the development of the tool—including individual consultations, advisory groups, interviews, and a pilot implementation—we were able to create a more robust and relevant product for use by SEAs and districts.

Why Home Language Survey Data Quality?

Federal law requires states to identify and assess students who may need extra services to learn English. As a result, many SEAs across the U.S. recommend or require that districts use a home language survey as the first step in a multi-step process of identifying students who qualify for EL services. Parents typically complete the survey as a part of the student registration process. Districts use this information to determine if a student speaks or is spoken to in another language at home, an indicator that the student might need EL services. However, existing home language surveys may not yield accurate information about students’ exposure to or use of English, which can contribute to the misidentification of EL students.

Educators, advocates for EL students, and the U.S. Department of Education’s Office of Civil Rights have all raised concerns about the quality of data produced by home language surveys. District- and school-level factors that can affect the data quality include the following: (1) purposes, policies, and guidelines governing home language survey administration; (2) data collection practices; (3) personnel support; and (4) data management. For example, if districts do not clearly document or communicate the purpose of the home language survey, parents or guardians completing the survey may not understand the intended use of the data collected, which is to help their child. Or, if district or school personnel responsible for new student registrations are not properly trained in how to explain the home language survey to parents, they may not correctly guide parents in how to fill out the form, which can affect data accuracy.

We created the self-assessment tool to help states and districts identify which factors are affecting their own home language survey data quality. Such information can guide them in selecting and designing appropriate technical assistance to improve data collection and data quality.

Stakeholder Engagement Process

Our research team developed the Home Language Survey Data Quality Self-Assessment iteratively, responding to stakeholder needs and feedback along the way. In 2014, REL Northeast & Islands’ English Language Learners Alliance (ELLA) had formed a small working group of SEA leaders and district EL program coordinators in Connecticut and Rhode Island who were interested in addressing and exploring EL data quality. The working group identified home language survey data quality as a common challenge.

Drawing on recommendations form the National Forum for Education Statistics publication, “Forum Curriculum for Improving Education Data: A Resource for Local Education Agencies,” and initial input from this working group, our research team developed a data-quality framework and self-assessment items to help states improve their home language survey data quality. This initial self-assessment tool was piloted through the Connecticut Administrators of Programs for English Language Learners (CAPELL) and the Rhode Island English Learner Network. The results from 23 districts were analyzed with REL Northeast & Islands support and presented to the working group and to CAPELL at the request of the Connecticut members.

Based on the pilot results and evidence of the tool’s usefulness in identifying areas of improvement for home language survey data quality, our REL Northeast & Islands team, in collaboration with the working group, decided to revise the tool for publication. The alliance formed an advisory committee of state and district leaders from New York, New Hampshire, and Massachusetts to guide the refinement of the self-assessment tool and ensure its applicability to other state contexts. These committee members recommended that district EL program coordinators participate in cognitive interviews with the development team, which—alongside feedback from content and methodological expert consultations—informed revisions to the self-assessment. One unexpected but interesting outcome of this collaborative work was that the advisory committee members found the discussions of the tool useful to their own thinking and began to consider changes to their own practices related to their home language survey.

Benefits of Stakeholder Engagement

Our project team, and our resulting product, benefited in multiple ways from engaging stakeholders throughout this work:

1. The team focused on improving the relevance of the self-assessment tool by engaging an initial small working group of state and district leaders to illuminate the challenges that both groups experienced with home language surveys and the implications these challenges had for data quality.
2. The team increased the clarity of the self-assessment’s items by engaging with methodological specialists and practitioners to address factors contributing to data quality across the data-quality framework.
3. The team learned from cognitive interviews that district EL program coordinators found value in the tool – so we adapted the tool to also serve as an opportunity for reflection on local district practices, increasing the size of the potential user-audience.
4. Finally, based on the pilot administration of the self-assessment tool across districts, we developed and then improved a self-assessment data-use protocol to help states identify priority areas for technical assistance that would impact data quality. Though the stakeholder engagement process took time and resources, the result is a much more relevant and clear self-assessment tool that is of interest to a larger user-audience.

Browse REL Program work related to English learners.

Browse tools developed by the REL Program.