IES Blog

In part 1 of this blog series, we highlight the benefits and advantages of using federal education data for policymaking at the federal, state, and district levels. In part 2, we will explore the challenges of and opportunities afforded by using these data.

States, districts, and schools are inundated with requests for data. To manage the volume of requests and avoid overwhelming educators, many districts have established processes to vet and limit the number of surveys allowed in their schools and administrative offices. District clearance processes are also understandably meant to make sure everyone is in compliance with data privacy laws. NCES data collections are sometimes not cleared by district offices, which then means NCES is not allowed to contact schools or educators to learn from their perspectives or experiences.

Since many state or district policymakers prioritize local survey collections, federal surveys are occasionally rejected by district offices that are striving to keep from overburdening their educators with too many survey requests. Without district permission, NCES surveys won’t include the educators in those schools, meaning that those districts’ voices will be missing from the table when decisions are made. This is problematic for the entire education system for a few reasons.

1. Local data rarely reach federal policymakers. We know state and district decisionmakers derive a lot of value from state and/or district survey collections—since they’re designed to provide detailed local data—and do not always see value in federal data collection and reporting efforts. More localized data are critical to their day-to-day decisions. However, the presence of numerous state-run surveys—in addition to the myriad individualized district surveys that can exist within a single state—has begun to create a data silo where information remains frozen within a state or district system. Since NCES (and therefore the Department of Education and Congress) rarely receives data from state or local collections, these data sources cannot readily be used to generate national policies, which greatly limits opportunities for state and district systems to learn from each other. These data silos can, for example, impact the focus or breadth of federal grants or funding available for schools.
   
   Federal, state, and district education agencies serve different roles in the education sector but have mutually beneficial responsibilities that should complement and support one another. The solution isn’t to supplant federal data collections with local ones, or vice versa, but instead to supplement local collections with federal collections like the National Teacher and Principal Survey (NTPS) so education decisionmakers at all levels have access to necessary information to make good decisions for our schools.
    
2. Benchmarking and comparability are limited. Without federal data collections, it can be difficult or impossible for states, districts, and local policymakers to compare their schools and educators with those in other areas because of the lack of common focus and definitions across data collections. Even if the topics being collected are similar, individualized district or state surveys can differ widely in both content and wording.
   
   National data collections—like the NTPS—are excellent tools local policymakers should use when setting priorities on behalf of the students and staff in their state or district. Since the data from the NTPS are collected from educators in the same way across the entire country, they can be used to establish benchmarks against which local collections can measure themselves.
    
3. Lack of participation decreases the representativeness of storytelling. If districts do not approve NCES’s survey research applications, we are unable to reach educators in certain schools, which can limit the kinds of perspectives that are included in the data. To paint a true picture of the education landscape, our survey teams select districts, schools, and/or educators that are as representative of the education field as possible.
   
   Teachers and principals who participate in NCES studies are grouped in different ways—such as by age, race/ethnicity, or the type of school at which they work—and their information is studied to identify patterns of experiences that people in these different groups may have had. This is what makes our datasets representative, or similar enough to the demographics of the population to able to accurately reflect the characteristics of everyone (even those who aren’t sampled to participate).
   
   For example, the NTPS is designed to support analysis of a variety of subgroups, such as those by

• school level (i.e., elementary, middle, high, and combined);
   
• school community type (i.e., urban, suburban, town, and rural);
   
• teachers’ and principals’ years of experience in the profession; and
   
• race/ethnicity of teachers and principals (figure 1).

These diverse subgroups are critically important for both federal and local policymakers who want to make decisions using information that truly represents everyone in the field.

Figure 1. Percentage of K–12 public and private school teachers who reported that they have any control over various areas of planning and teaching in their classrooms, by school type and selected school characteristic: 2020–21 

NOTE: Data are weighted estimates of the population. Response options included “no control,” “minor control,” “moderate control,” and “a great deal of control.” Teachers who reported “minor control,” “moderate control,” or “a great deal of control” were considered to have reported having “any control.”
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), “Public and Private School Teacher Data File,” 2020–21.

Larger datasets allow for more nuanced comparisons by school, principal, or teacher characteristics that aren’t possible in smaller datasets, allowing for more equity in national and local estimates and more distinct answers to key policy questions. But we need district-level support to provide these nuanced data.

Although the NTPS has a fairly large sample to support state representation, it still only includes a small percentage of all schools and educators in the country. Sampling is used to avoid collecting data from all systems, staff, and students, thus helping to limit overall respondent burden on our education system. For this reason, it’s important that all sampled schools and educators participate if selected. Since some districts also have formal review processes—through which a survey must be approved before any schools or educators can be contacted—it is also important that districts with sampled schools grant NCES surveys permission to collect data from their schools.

Both levels of participation will help us collect data that accurately describe a state or population. Otherwise, the story we are telling in the data is only augmenting some voices—and these are the experiences that will be reflected in federal policy and funding.

As the education sector strives to understand the needs of students and staff on the tails of the coronavirus pandemic, trustworthy data are only becoming more critical to the decisionmaking process. NCES datasets like the NTPS are critical resources that federal, state, and district policymakers can and have used for benchmarking strategic goals or conducting analyses on how a topic has (or hasn’t) changed over time.

The catch being, of course, that all data on the NTPS—and many other NCES surveys—come directly from schools, principals, and teachers themselves. These analyses and reports are not possible without district and educator participation. While it may seem counterintuitive that any one person could make such a large difference in federal education policy, the concept doesn’t differ from civic duties such as voting in federal or local elections.

By Maura Spiegelman and Julia Merlin, NCES

NCSER is excited to share the work of our three new Early Career Development and Mentoring Grants Program principal investigators (PI). The aim of this grant program is to support early career scholars in their academic career trajectories as they pursue research in special education. Through a series of interview blogs, each PI will share their research interests, advice for other early career scholars, and desired impact within the field of special education.

The first scholar we are spotlighting is Kelly Williams, assistant professor in communication sciences and special education at the University of Georgia (formerly at Indiana University). Dr. Williams received a grant to develop an intervention to support reading and spelling outcomes for adolescents with word-level reading disabilities (WLRD).

How did you become interested in this area of research?         

I originally became interested in research on WLRD through my experience as a high school special education teacher in rural Georgia where I taught English literature and composition to students with mild to moderate disabilities. Most of my students had difficulty reading and spelling words accurately and automatically, which significantly impacted their performance both in and out of school. In school, my students struggled to complete grade-level coursework, which, in turn, affected their ability to graduate with a regular high school diploma. Outside of school, my students had difficulty with tasks such as completing job applications that required extensive amounts of reading. Although I was well prepared to provide classroom accommodations and modifications for my students, I found that I lacked the knowledge and skills to provide intensive interventions that would help improve basic reading and spelling skills. These experiences ultimately led me to pursue my doctorate in special education with an emphasis on learning disabilities.

What advice do you have for other early career researchers?

I think it is important for early career researchers to collaborate with various stakeholders throughout the entire research process. Although many of my ideas stem from my own experiences as a teacher, I have found that listening to various perspectives has helped me identify problems, brainstorm potential solutions, and design practical interventions that will improve outcomes for students with disabilities. Sustaining effective interventions requires us to think about how we can involve students, teachers, administrators, parents/caregivers, schools, and other community members in research.

What broader impact are you hoping to achieve with your research?

We know low reading achievement is associated with numerous negative outcomes across domains (social, emotional, behavioral, academic, economic). My hope is that this project will provide secondary teachers with a feasible and practical intervention to improve reading outcomes for older students with WLRD, which, in turn, may help prevent or ameliorate the effects of these negative consequences. Ultimately, I envision that this intervention could be used independently or as part of a multi-component reading intervention for secondary students with WLRD.

How will this intervention be distinct from other reading and spelling interventions?

There are two ways that this intervention is distinct from other word reading and spelling interventions. First, this intervention will embed spelling instruction within word reading, which is not currently happening in research or practice for secondary students with WLRD. Many existing programs teach spelling in isolation or through rote memorization, despite a large body of research demonstrating a connection between spelling and word reading. Second, the proposed intervention will emphasize a flexible approach to multisyllabic word reading instead of teaching formal syllable division rules. The goal of this approach is to reduce cognitive load, thereby improving the ability to accurately and automatically read and spell words.

Thank you, Kelly Williams, for your thoughtful insights and commitment to improving reading and spelling among students with word-level reading disabilities. NCSER looks forward to following your work as you progress in developing this intervention.

This blog was produced by Emilia Wenzel, NCSER intern and graduate student at University of Chicago. Katie Taylor (Katherine.Taylor@ed.gov) is the program officer for NCSER’s Early Career Development and Mentoring program.

While federal, state, and district policymakers have used education data as the backbone of their policy and funding decisions for years, there’s nothing quite like a global pandemic to highlight the criticality of reliable data and illuminate the gaps in our collective knowledge. 

But where can policymakers find education datasets that are large enough for comparative or trend analyses while still specific enough to measure timely issues in local contexts? How can policymakers extract and interpret information from these education datasets? The National Center for Education Statistics (NCES) is a perfect resource for these data and information needs.  

Torrence Williams, director of teacher advancement at the Associated Professional Educators of Louisiana, leads a professional learning module training for Louisiana’s New Teacher Experience program.

The Regional Educational Labs (REL) program, operated by the U.S. Department of Education’s Institute of Education Sciences (IES), supports state education agencies, schools and school districts, and institutions of higher education nationwide in using data and evidence-based practice to improve opportunities and outcome for learners. Operating in all 50 states, the District of Columbia, Puerto Rico, and the U.S. Territories and Freely Associated States of the Pacific region, the REL program brings together the expertise of local communities, top-tier education researchers, and education scientists at IES’s National Center for Education Evaluation and Regional Assistance (NCEE) to address the most vexing problems of education policy and practice in states and regions—on demand and free of charge.

It's not exactly breaking news that many schools in our country struggle to fill vacancies in their teacher workforce. This past fall, IES’s National Center for Education Statistics surveyed public school leaders about staffing challenges as they began the 2022-23 school year. The statistics were sobering: 45 percent of schools reported having at least one vacant position more than one month into the school year, and more than 25 percent of schools reported multiple vacancies. Worryingly, our most underserved students were experiencing this crisis most acutely, with roughly 60 percent of schools in high-poverty neighborhoods or with a high-minority student body reporting at least one vacancy. While all of us anxiously await data on the 2023-24 school year—which should be available later this year—RELs and their state and local partners are working to strengthen all aspects of the teacher pipeline.

In the second of a four-part blog series, we highlight four REL research and development projects that address educator recruitment and retention. Each demonstrates how RELs are leveraging their distinct capacity for innovation, rigorous research, and authentic partnership to deliver locally focused and evidence-based supports to the regions, states, and communities they serve.

REL Northwest: Examining Strategies to Improve Teacher Recruitment and Retention in Rural Alaska

Like many rural school districts across the nation, Alaska’s Lower Kuskokwim School District (LKSD) is experiencing a persistent and pressing need to attract and retain educators. Near the start of the 2022-23 school year, 78 positions—nearly one-quarter of all teaching positions in the district—remained unfilled. New teachers were often recruited to the district from other countries such as the Philippines, leading to low retention rates and a constant churn of new educators. Faced with this persistent, high-stakes problem, leadership at LKSD decided to partner with REL Northwest to discover research-based solutions to their teacher recruitment and retention crisis.

REL Northwest is partnering with LKSD to identify evidence-based strategies and tools to continuously monitor and improve working conditions with the goal of increasing teacher retention. To do that, LKSD plans to implement the recommendation of strengthening teacher working conditions from the state's Teacher Retention and Recruitment Plan. As a first step in this partnership, REL Northwest staff reviewed and summarized research on working conditions and teacher retention to identify eight factors that may influence a teacher's decision to stay or leave.  

Lower Kuskokwim leaders decided to focus on activities to identify how changes related to three of those factors—supportive school leadership, available time for teachers, and community engagement—may improve working conditions. The first activity will adapt existing LKSD data sources and develop a research-informed tool to monitor teacher perceptions of school leadership and collect further data to inform the district’s action plan.

Staff also identified a school leadership responsibility unique to their district that may affect teacher trust: managing teacher housing. Housing is a major challenge, not only in remote villages without road access, but in many areas of the country. REL Northwest led partners through an activity to brainstorm how school leaders could improve housing and develop a theory of action for how those strategies would improve working conditions and promote retention. The strategies included establishing realistic housing expectations for new teachers, revising the process for leader evaluation of housing needs, and changing the budgeting process to make maintenance needs and upgrades easier. As a result of the work, district leaders are designing a program that allows teachers to apply for district funding to make simple housing upgrades, such as changes in lighting or painting.

REL Central: Strengthening the Teacher Pipeline in South Dakota to alleviate Teacher Shortages

Like many states, South Dakota is experiencing a teacher shortage that has worsened in recent years. Late last year, SDDOE partnered with REL Central to support one component of their response to this challenge: developing new pathways into teaching for candidates such as paraprofessionals and other South Dakota residents who have interest in entering the teacher workforce. Initially, the work focused on the design and implementation of a teacher apprenticeship program designed to support paraprofessionals as they acquire their teaching degrees and as they are mentored to become certified teachers.

In March, REL Central began work with SDDOE on a fast-turnaround project to support the development of a survey for paraprofessionals about their interest in the pilot apprenticeship program and the types of supports they seek from mentor teachers. Within a matter of weeks, the survey was developed and administered to paraprofessionals statewide. With survey data in hand indicating that hundreds of South Dakota paraprofessionals were interested in such a program, the pilot was expanded by SDDOE to support additional slots starting in fall 2023. In the coming months, REL Central will work with SDDOE to further refine this program by incorporating research evidence from other “Grow Your Own” teacher workforce programs on the components of effective mentoring and by helping the state to generate, collect, analyze, and use data from participants to inform further improvements to the pilot apprenticeship program.

REL Pacific: Strengthening the teacher workforce in Palau

The Republic of Palau, like many school systems, has experienced challenges in recruiting and retaining enough teachers to provide every student with a high-quality education. The geographic isolation of Palau compounds these challenges. Many local Palauan teachers do not have a four-year college degree or are teaching outside of their area of certification. As was the case in Alaska’s Lower Kuskokwim School District, one solution has been to bring in teachers from out of the country to fill vacancies; however, the turnover rate of these teachers is very high. The Palau Partnership to Support Teacher Effectiveness–– a collaboration with the Palau Ministry of Education (MOE), Palau Community College (PCC), six private schools, and REL Pacific ––is focused on building more sustainable solutions. 

The long-term goals of the partnership include Palau private schools adopting a teacher effectiveness measurement system to support, develop, and retain effective teachers; Palau private schools adopting a systemwide instructional coaching process for improving teacher effectiveness; and Palau MOE and PCC reviewing data on the effects of teacher education programs and making implementation adjustments so that their available resources may be used more effectively and efficiently. 

REL Pacific is supporting partner schools to realize their goals by drawing from resources on indicators of successful teacher recruitment and retention as well as best practices of effective teaching. REL Pacific is providing schools training and coaching on data-driven decision-making and plan-do-study-act (PDSA) cycles in the context of the schools’ goals to improve literacy instruction through developing systemic supports for teachers. By incorporating ways of measuring best practices of effective teaching into the schools’ instructional coaching processes, each school will be better able to enact systemic change to address its specific teacher development and retention needs. Additionally, an applied research study is underway that will describe teacher pathways and certification patterns. The findings from this descriptive study will inform future efforts of Palau’s education community to address the new teacher certification requirements and overall educator shortage crisis.

REL Southwest: Partnering to support early career and aspiring teachers in Louisiana

Louisiana’s educator shortage is compounded with low retention rates for early-career teachers. Teachers with 2–5 years of experience left public schools in 2020 at a rate of 30 percent, compared with 17 percent of teachers with 6–10 years of experience. REL Southwest and the Louisiana Department of Education (LDOE) have formed the Supporting Early Career and Aspiring Teachers (SECAT) partnership to improve supports for novice and aspiring teachers. Through the SECAT partnership, REL Southwest aims to strengthen LDOE’s capacity to use evidence to refine new initiatives that support early career and aspiring teachers, such as the New Teacher Experience program and the Louisiana Pre-Educator Pathway, the state’s largest “Grow Your Own” program. This work builds from a previous partnership between REL Southwest and LDOE focused on exploring and evaluating the early impacts of Louisiana’s Believe and Prepare teacher residency program.  

The work of the SECAT partnership kicked off this past spring. Over the next five years, REL Southwest will work with LDOE and school systems in Louisiana to strengthen their capacity to generate and use evidence to refine existing programs for early career and aspiring teachers. In the first year of the partnership, REL Southwest and LDOE partners plan to focus on technical assistance that builds LDOE’s capacity for evaluating the New Teacher Experience. In future years of the partnership, REL Southwest will study LDOE’s efforts to support new and aspiring teachers. Along the way, REL Southwest will share important takeaways, resources, and policy implications related to teacher recruitment and retention learned through the partnership with Louisiana.

Stay tuned for part three of our “Making a Difference” series, focused on school accountability systems. As always, my (virtual) door is open if you have questions about the work highlighted in this blog, or anything else on REL Program. Just email me at chris.boccanfuso@ed.gov.

The National Assessment of Educational Progress (NAEP) is the largest nationally representative and ongoing assessment of subject knowledge among students in public and private schools in the United States. On the 2017 eighth grade mathematics assessment, 38% of students without disabilities scored at the NAEP Proficient level or above while 25% scored below the NAEP Basic level. However, for students with disabilities, math achievement levels were much worse. Only about 9% of students with disabilities scored at the NAEP Proficient level or above whereas 69% scored below the NAEP Basic level. In response to this gap, in 2021, the National Center for Special Education Research (NCSER) released a funding opportunity to coincide with the release of the 2017 Grade 8 NAEP Mathematics response process data. NCSER intended to support research that explores how learners with disabilities interact with the NAEP digital assessment to better support these learners in test-taking environments and determine whether and how that information could be used to inform instructional practices. There is much to learn from research on NAEP process data for understanding test-taking behaviors and achievement of learners with disabilities. Below we showcase the latest findings from currently funded research and encourage more investigators to conduct research with newly released process data.

Since 2017, administrations of NAEP have captured a variety of response process data, including keystrokes as learners progress through the assessment, how learners use the available tools (such as the calculator), and how accommodations (for example, text-to-speech or more time to complete the assessment) affect performance. Besides score data, NAEP datasets also include survey data from learners, teachers, and schools, and information on test item characteristics and student demographics (including disability). Together, these data provide a unique opportunity for researchers to conduct an in-depth investigation of the test-taking behavior and the mathematics competencies of learners with disabilities compared to their peers without disabilities.  

In July 2021, IES awarded two grants to conduct research using NAEP process data. The results of these projects are expected to improve the future development and administration of digital learning assessments, identify needed enhancements to mathematics instruction, and highlight areas where further research is needed.  Although these projects are ongoing, we would like to highlight findings from one of the funded projects awarded to SRI International and led by principal investigator Xin Wei  entitled Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities.

The findings from this study, recently published in Autism, is an example of the power of process data to shed new light on learners with disabilities. Focusing on autistic students, Xin Wei and her team analyzed data from 15 items on the NAEP math assessment, their response time in seconds, their score on the items (including partially correct scoring), and survey data related to their enjoyment, interest, and persistence in math. They also analyzed the content of each item using Flesch Reading Ease scores to measure the reading difficulty level of the item. Finally, they rated each item based on the complexity of any social context of the item, as prior research has shown that these contexts can be more challenging for autistic students. They conducted statistical analyses to compare the performance of autistic students with extended time accommodations, autistic students without accommodations, and general education peers. The researchers were not only looking for any areas of weakness, but also areas of strength. Previous studies have demonstrated that autistic people frequently excel in abstract spatial reasoning and calculation tasks, relying more on visual-mental representations than verbal ones.

The findings showed that in comparison to their general education peers, unaccommodated autistic students scored higher and solved math problems involving the identification of figures more quickly. Unaccommodated autistic students were also faster than their general education peers at solving the following types of math items: comparing measures using unit conversions, mentally rotating a triangle, interpreting linear equations, and constructing data analysis plots. Although autistic students who used the extended-time accommodation were lower performing than the other two groups, they had a higher accuracy rate on items involving identifying figures and calculating the diameter of a circle. Both groups of autistic students seem to perform poorer on word problems. Researchers concluded that the linguistic complexity could be one of the reasons that autistic students struggle with math word problems; however, there were two word problems with which they seemed to struggle despite the fact that they were not linguistically complex. It turns out that the items were rated as having substantial social context complexity. The researchers also looked at the student survey data on what types of math they enjoyed more and found they had more enjoyment working with shapes and figures and less enjoyment for solving equations.

The researchers recommend incorporating meta-cognitive and explicit schema instruction during mathematics instruction to aid autistic students in understanding real-life math word problems. They also recommend that assessment developers consider simplifying the language and social context of math word problems to make the assessment more equitable, fair, and accessible for autistic students. Because the autistic student population is particularly heterogenous, more research is required to better understand how to improve instructional strategies for them.

IES plans to release the same type of process data from the 2017 Grade 4 NAEP Mathematics at the end of this summer. We encourage researchers to request these process data to conduct research to understand test-taking behavior and performance of students with disabilities at the elementary school level. For a source of funding for the work, consider applying to the current Special Education Research Grants competition. Here are some important resources to support your proposal writing:

• Conducting research using process data from NAEP
• IES/NCSER Standard Research Grant Program (applications due September 21, 2023)
• Response Process Data From the 2017 NAEP Grade 4 and 8 Mathematics Assessment
• Apply for NAEP restricted-use datasets

This blog was authored by Sarah Brasiel (Sarah.Brasiel@ed.gov), program officer at NCSER, and Juliette Gudknecht, summer data science intern at IES and graduate student at Teachers College, Columbia University. IES encourages special education researchers to use NAEP response process data for research under the Exploration project type within our standard Special Education Research Grants Program funding opportunity.