Inside IES Research

In celebration of IES’s 20^th anniversary, we’re telling the stories of our work and the significant impact that—together—we’ve had on education policy and practice. As IES continues to look towards a future focused on progress, purpose, and performance, Dr. Matthew G. Springer of the University of North Carolina at Chapel Hill, discusses the merit pay debate and why the staying power of IES will continue to matter in the years to come.

Merit Pay for Teachers

There are very few issues that impact Americans more directly or more personally than education. The experience of school leaves people with deep memories, strong opinions, and a highly localized sense of how education works in a vast and diverse country. Bringing scientific rigor to a subject so near to people’s lives is incredibly important — and maddeningly hard. The idea of merit pay inspires strong reactions from politicians and the general public, but for a long time there was vanishingly little academic literature to support either the dismissive attitude of skeptics or the enthusiasm of supporters.

Given the stakes of the merit pay debate—the size of the nation’s teacher corps, the scale of local, state, and federal educational investments, and longstanding inequities in access to highly effective teachers—policymakers desperately need better insight into how teacher compensation might incentivize better outcomes. Critics worry merit pay creates negative competition, causes teachers to focus narrowly on incentivized outcomes, and disrupts the collegiate ethos of teaching and learning. On the other hand, supporters argue that merit pay helps motivate employees, attract and retain employees in the profession, and improve overall productivity.

Generating Evidence: The Role of IES-Funded Research

That’s precisely the kind of need that IES was designed to meet. In 2006, with the support of IES, I joined a group of research colleagues to launch the Project on Incentives in Teaching (POINT) to test some of the major theories around merit pay. We wanted to apply the most rigorous research design—a fully randomized control trial—to the real-world question of whether rewarding teachers for better student test scores would move the needle on student achievement.

But orchestrating a real-world experiment is much harder than doing an observational analysis. Creating a rigorous trial to assess merit pay required enormous diplomacy. There are a lot of stakeholders in American education, and conducting a meaningful test of different teacher pay regimes required signoff from the local branch of the teacher’s union, the national union, the local school board and elected leadership, and the state education bureaucracy all the way up to the governor. Running an experiment in a lab is one thing; running an experiment with real-world teachers leading real, live classrooms is quite another.

With IES support to carry out an independent and scientifically rigorous study, POINT was able to move forward with the support of the Metropolitan Nashville Public Schools. The results were not what most anticipated. 

Despite offering up to $15,000 annually in incentive pay based on a value-added measure of teacher effectiveness, in general, we found no significant difference in student performance between the merit-pay recipients and teachers who weren’t eligible for bonuses. 

As with all good scientific findings, we strongly believed that our results should be the start of a conversation, not presented as the final word. Throughout my career, I’ve seen one-off research findings treated by media and advocacy organizations as irrefutable proof of a particular viewpoint, but that’s not how scientific research works. My colleagues and I took care to publish a detailed account of our study’s implementation, including an honest discussion of its methodological limitations and political constraints. We called for more studies in more places, all in an effort to contest or refine the limited insights we were able to draw in Nashville. “One experiment is not definitive,” we wrote. “More experiments should be conducted to see whether POINT’s findings generalize to other settings.”

How IES-Funded Research Informs Policy and Practice 

Around the time of the release of our findings, the Obama administration was announcing another investment in the Teacher Incentive Fund, a George W. Bush-era competitive grant program that rewarded teachers who proved effective in raising test scores. We were relieved that our study didn’t prompt the federal government to abandon ship on merit pay; rather, they reviewed our study findings and engaged in meaningful dialogue about how to refine their investments. Ultimately, the Teacher Incentive Fund guidelines linked pay incentives with capacity-building opportunities for teachers—things like professional development and professional learning communities—so that the push to get better was matched by resources to make improvement more likely.  

While we did not have education sector-specific research to support that pairing, it proved a critical piece of the merit pay design puzzle. In 2020, for example, I worked with a pair of colleagues on a meta-analysis of the merit pay literature. We synthesized effect sizes across 37 primary studies, 26 of which were conducted in the United States, finding that the merit pay effect is nearly twice as large when paired with professional development. This is by no means the definitive word, but it’s a significant contribution in the incremental advancement of scientific knowledge about American education. It is putting another piece of the puzzle together and moving the education system forward with ways to improve student opportunity and learning.

That’s the spirit IES encourages—open experimentation, modesty in drawing conclusions, and an eagerness to gather more data. Most importantly, we wanted our findings to spark conversation not just among academic researchers but among educators and policymakers, the people ultimately responsible for designing policy and implementing it in America’s classrooms.

Pushing that public conversation in a productive direction is always challenging. Ideological assumptions are powerful, interest groups are vocal, and even the most nuanced research findings can get flattened in the media retelling. We struggled with all of that in the aftermath of the POINT study, which arrived at a moment of increased federal interest in merit pay and other incentive programs.  Fortunately, the Obama administration listened to rigorous research findings and learned from the research funding arm of the U.S. Department of Education.

This is why the staying power of IES matters. The debates around education policy aren’t going away, and the need for stronger empirical grounding grows alongside the value of education and public investment. The POINT experiment didn’t settle the debate over merit pay, but we did complicate it in productive ways that yielded another critically important finding nearly eight years later. That’s a victory for education science, and a mark of progress for education policy.

Matthew G. Springer is the Robena and Walter E. Hussman, Jr. Distinguished Professor of Education Reform at the University of North Carolina at Chapel Hill School of Education.

This blog was produced by Allen Ruby (Allen.Ruby@ed.gov), Associate Commissioner for Policy and Systems Division, NCER.  

In honor of Career and Technical Education (CTE) Month, we asked principal investigator Dr. Lee Kern how her intervention research reduces dropouts and prepares students with emotional and behavioral disorders (EBD) for college and career readiness (CCR). The purpose of her current IES project is to develop and pilot test an intervention, Supported College and Career Readiness (SCCR), that augments typical school-based college and career readiness activities for students at or at risk for EBD.

What motivated you to conduct this research?

Given the high dropout rate among students with EBD, I am interested in strategies that keep them in school. Because post-graduation experiences serve as important indicators of positive educational outcomes, I want to establish a stronger connection between school and life after school to ensure that students are fully prepared. My co-PIs, Jennifer Freeman and Chris Liang, were motivated to collaborate on the current research project as well because of their unique focus on different aspects of CCR, allowing us to address multiple dimensions in the development of our intervention.

Can you provide us with an update on the project? What work have you completed to date on the development of the SCCR program?

We recognized and addressed a gap in the college and career readiness literature with this group of students. During the first 2 years of the project, we completed two literature reviews and two conceptual papers, which are in press, and we are in the process of completing a third literature review. Our completed literature reviews indicated (a) limited attention to CCR for individuals with emotional and behavioral problems, (b) lack of defined components of CCR interventions, (c) the need to evaluate the effectiveness of CCR interventions with students of color, and (d) aspects of CCR interventions that might be important for individuals with diverse sexual identities. These papers helped us develop our multi-component CCR intervention for students with or at risk for EBD.

The development phase was vital to creating our multi-component program. Schools practice different approaches to college and career preparation, so we needed to create a flexible program that could fit the many permutations in course scheduling, career interest assessments, career exposure activities, and other factors. Receiving teacher and student feedback on the program during the second year of the project was helpful and appreciated as we refined SCCR. We initiated a randomized controlled trial and ran the study in four schools this academic year. We will expand the research into four additional schools in the 2023-24 academic year.

What other types of research are needed to move forward in the field of CTE for students with or at risk for EBD?

Although we know that students, especially those with or at risk for EBD, need more preparedness for college or their future careers, research must specify intervention components that result in improved outcomes in these areas. Also, it must determine whether the interventions are effective across diverse groups of students and ascertain adaptations that address the needs of all students. Existing and ongoing research must be conducted to better assess student skills. Identifying assessments directly linked to critical and effective interventions that practitioners can implement will be important for future progress.

NCSER looks forward to learning the results of the pilot study to better understand the promise of the SCCR program for improving the college and career readiness of students with or at risk for EBD. For more highlights on the CTE-related work that IES is supporting, please check out our IES CTE page. 

Dr. Kern is a professor and the director of the Center for Promoting Research to Practice at Lehigh University. She has more than 30 years of experience in special education, mental health, and behavior intervention for students with EBD.

This CTE blog post was produced by Alysa Conway, NCSER student volunteer and University of Maryland, College Park graduate student. Akilah Nelson is the program officer for NCSER’s Career and Technical Education grants.

As part of our 20^th anniversary celebration and in recognition of Black History Month, we asked Dr. Jason Snipes, Director of Applied Research for REL-West at WestEd, to discuss his inspiration for his IES-funded replication study. The purpose of the study is to test the potential of self-affirmation interventions to counteract the harmful effects of negative racial stereotypes on the academic, disciplinary, and psychosocial outcomes of 7^th grade Black and Latino students.

What motivates your research on the effects of self-affirmation interventions on Black and Latino student outcomes?

My mother was a civil rights activist, and I was raised with a clear sense of my history as a Black person and the importance of making a contribution worthy of those that preceded me. Her example inspired me to pursue a career in research on education and youth development and to focus on finding, testing, and understanding strategies for improving outcomes for the Black, Latino, and other students systematically underserved by our education system.

My specific interest in stereotype threat and self-affirmation research stems in part from my own education experiences. I still remember how—despite being in the gifted and talented program at my elementary school—every mistake I made, every question I asked, every idea I expressed was greeted with the snickers and whispers of my peers crudely expressing their doubts about my intelligence. I remember my success slipping away, and before I knew it, being in 8th grade remedial math—failing. My father essentially saved my life. He somehow taught me to truly believe that I could accomplish anything I wanted to. This is not a solution to systemic racism. Still, his support for changing my beliefs about myself, combined with going to a new high school, completely changed my academic trajectory.

I again felt the weight and pressure of low expectations in graduate school, in the subtle and not-so-subtle ways my White professors expressed their doubts about my ability to succeed in a rigorous PhD program.

So, later in my career, when I learned about stereotype threat and self-affirmation, I saw a bit of myself and my life experiences. I saw something else that I found unusual: an intervention with significant effects, even when tested in rigorous randomized trials. I wanted to know more about where, how, and under what circumstances it could be effectively used support Black and Brown children.

What is stereotype threat and how might it impact Black students?

Among Black students, stereotype threat is the fear of confirming negative racial stereotypes about their academic performance and their underlying intelligence. It can be one of the many persistent and pervasive psychological stressors that Black people encounter on a daily basis. Randomized trials show that when prompted to believe a test is an assessment of their intelligence, Black students perform more poorly. The prompt generates physiological and psychological stress responses, reduces available working memory, and results in both fewer questions answered in a given period of time and a lower percentage of correct answers among those given. The same prompt has no effect on White or Asian students. A meta-analysis of 300 studies suggests stereotype threat accounts for a quarter to a third of the Black-White and Latino-White achievement gaps.

How does the self-affirmation intervention you are evaluating address stereotype threat?

The self-affirmation intervention, created by Claude Steele and Joshua Aronson, is designed to respond to stereotype threat. It’s a set of four 15-minute writing exercises administered over the course of a school year. Each exercise provides students with the opportunity to affirm their value by asking them to write about things that are important to them. Experiments with college and middle school students show that self-affirmation improves a variety of academic outcomes for Black students, and that that these effects persist and grow over time.

Our study goes beyond prior research to provide new evidence about the impact of self-affirmation on Black and Latinx students in schools with different demographic compositions and the extent to which its effects generalize across a nationally representative sample of schools. Some studies suggest that self-affirmations effects are smaller in schools with higher concentrations of Black and/or Latino students. We plan to systematically explore this and other questions about moderators. Our findings will have implications for the settings in which self-affirmation ought to be scaled and implemented and how it might be used as a complement to other available supports to bolster the success of Black and Latinx students.

That stereotype threat appears to account for a quarter of observed racial achievement gaps, and that self-affirmation ameliorates this effect makes self-affirmation relevant to larger discussions of educational equity. Self-affirmation, along with other psychosocial interventions, should be investigated as potential tools for reducing racial disparities in education outcomes. That said, it is important to remember that racial inequity is a feature of the education system and the institutions that surround it, not a function of some sort of “flaw” in the attitudes or psychosocial make up of Black and Brown students themselves. While these approaches may help buffer Black and Latino students against the full consequences of racism, bias, and stereotyping, we should never allow ourselves to be confused. The fundamental problem is not Black and Latino students’ ability to cope with these dynamics, but the presence of these dynamics in and of themselves. Furthermore, psychosocial interventions are not a substitute for high quality instruction or solutions to other systemic problems (for example, de facto segregation) that have powerful negative effects on academic outcomes among Black and Latino students.

Self-affirmation is also relevant to discussions of racial equity in education because the intervention reflects a fundamental concept underlying racial equity: personhood. It offers students a chance to affirm their value as human beings, and this may be one of the mechanisms through which it helps disrupts the destructive cycle of stereotype threat. This simple assertion embodies a core idea underlying the civil rights movement: that racial equity requires recognizing and treating Black and Brown people as fully human.   

Your current IES study aims to replicate prior research on self-affirmation. Why is replication important?

Too often, I have seen researchers, policy makers, practitioners, and funders make the mistake of misinterpreting results from a single, even rigorously designed, study as answering the question of whether an intervention or strategy works. Reality is more complex. What we can learn from a single study is usually something closer to the extent to which an approach worked in this place (or places) at this time. Under pressure for answers to pressing policy problems, we may rush to scale approaches or interventions with evidence from one or two well designed studies, only to find out that they don’t work at scale, or in a subsequent implementation, and we don’t know why.

It may be better to ask, “To what extent does an approach generate impacts, under what circumstances does it do so, and why?” Systematically replicating initial causal studies, enables researchers to address these questions more effectively. Rather than guessing at post hoc explanations of the patterns we observe, replication systematically tests hypotheses regarding how implementation, context, and other moderators and mediators affect program impacts. Doing so prior to undertaking massive scaling efforts therefore helps reduce the extent to which money, effort, and, perhaps most importantly, public will, are expended on strategies with fundamental limitations. Replication enables us to more systemically study and understand mechanisms of action, improving the extent to which we implement or scale interventions in contexts and situations in which they are likely to be most effective.

This blog was produced by Katina Stapleton (Katina.Stapleton@ed.gov) and Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER Program Officers.

In recognition of the IES 20th anniversary and Black History Month, we interviewed Dr. Jamie Pearson, an assistant professor of special education at North Carolina State University. Jamie is developing and refining a community-based parent-training intervention, FACES (Fostering Advocacy, Communication, Empowerment, and Support), designed to strengthen Black parents' capacity to access and use special education services and improve the communication and behavior outcomes for their autistic children.

How have your background and experiences shaped your scholarship and career in studying diversity, equity, and inclusion in education?

My early career experiences were as a behavioral interventionist for autistic students in home, school, and community settings. While providing direct support, I noticed that many of the students I supported were white and most came from middle- and upper-class socioeconomic backgrounds. These experiences led me to question whether there were disparities in diagnosis, misdiagnosis, and treatment/service access for children of color, particularly Black autistic children. These early questions were the catalysts for my scholarship.

As a doctoral student, I began exploring Black families’ experiences supporting autistic children. I became very passionate about investigating (a) disparities in the identification of autism and service access for Black autistic students and their families, (b) the implementation and evaluation of culturally responsive family advocacy interventions, and (c) strategies for strengthening partnerships between historically marginalized families and schools. Based on the findings from my early exploratory research, I developed and piloted the FACES intervention.

What advice would you give to emerging scholars from underrepresented, minoritized groups that are pursuing a career in education research?

When I began this work, I distinctly remember a faculty member asking me why it was important to look at the intersections of autism and race/ethnicity. They genuinely didn’t understand. I was passionate about my work, and even though not everyone understood the implications of these disparities at the time, they learned from my early exploratory work. It is important for underrepresented scholars to know that you have a seat at the table! Your knowledge, experiences, and contributions are needed in education research. We need more scholars of color, disabled scholars, and LGBTQIA+ scholars who reflect the populations with whom we conduct educational research and whose diverse perspectives impact how we engage in and interpret education research. My three pieces of advice in a nutshell would be find your passion, follow your passion, and know that you are deserving of a seat at the table. Pull up a chair if you have to!

Tell us about your current IES project focused on FACES. Do you have any updates or preliminary findings you would like to share about supporting Black children with ASD and their families?

The purpose of my IES Early Career project is to develop and test the promise of FACES when delivered by community-based parent educators. So far, two of my doctoral students and I (all Black women) have been the only people to facilitate FACES. To scale the intervention up, we need to design a training for facilitators to know how to implement FACES, train the facilitators, and then test its promise when delivered by facilitators in community-based settings. We are partnering with two community-based organizations who provide parent advocacy and support to achieve these goals.

During phase 1 of this project, we conducted a content analysis of our community partners’ data to better understand the extent to which Black families raising autistic students were seeking support for their child. These findings indicate that Black families are most often seeking specific therapeutic services (such as speech therapy) for their child, followed by school-related support and behavioral support. We then conducted focus groups with community-based providers to better understand their experiences and needs supporting Black families. Findings from these focus groups indicated that community-based providers are serving multiple roles—feeling as though they serve as therapist, teacher, advocate, and more with some families—with limited resources. These findings, combined with emergent themes around racial responsiveness and racial sensitivity, are helping us tailor the train-the-trainer components of the project. For example, we are building a section into our training about the implications of colorblind ideology and how to address facilitator biases. Facilitators will need to complete this training and demonstrate their understanding of the content before they move forward with facilitating the FACES intervention.

What do you see as the greatest research needs to improve the relevance of education research for diverse communities of students and families?

Much of the research around autism disparities has focused on quantifying racial disparities, yet little work has been done to reduce these disparities. Black families raising autistic children need access to parent education and advocacy training to combat the barriers they face in service access and utilization and find spaces where they feel welcome. I strongly believe that community-based parent education sets the foundation for empowering families that have been historically marginalized. We’ve seen FACES families go back to their communities and educate their friends and families about autism, connect them to services, and even create their own support groups. When families have more knowledge about autism and autism services, they feel more empowered. When they feel more empowered, they are better equipped to advocate. This is why it’s critical to engage in this work with historically marginalized families at the community level.

However, families of color still face many systemic barriers, so we still have a lot of work to do with educators and healthcare providers to ensure they are engaging in culturally responsive practices that facilitate effective partnerships with marginalized families. We need both empowered families and culturally responsive providers to effectively address these disparities.

The IES 20th anniversary campaign focuses on the future of IES as well as the most notable IES accomplishments. Follow the campaign on IES social media channels and our website. Join the conversation by using #IESat20 on social media.

This blog was produced by Akilah Nelson, program officer for the National Center for Special Education Research.

As part of the IES 20^th Anniversary celebration, we are highlighting NCER’s investments in field-initiated research. In the Adult Education and Adult Foundational Skills series, NCER will be spotlighting researchers and projects and sharing information about research to improve outcomes for adult learners.

In this opening blog, program officer Dr. Meredith Larson defines NCER’s use of the terms adult education and adult foundational skills, which have specific meaning and scope for IES research. Dr. Larson also describes who the adult learners are, why research in this area is important, and the research NCER has supported.

How does NCER define Adult Education and Adult Foundational Skills?

NCER uses the term adult education to refer to a system legislatively defined in the Workforce Innovation and Opportunity Act (WIOA). The adult education system serves learners who are at least 16 years old and not enrolled in K-12 through programs such as adult basic education, adult secondary education, integrated education training, family literacy, and integrated English language and civics.

By adult foundational skills, NCER means the common academic skills—such as literacy (reading, writing, listening), English language proficiency, and numeracy—that are necessary for participating in college or career. Nowadays, digital literacy and digital skills are emerging as foundational skills.

As researchers and practitioners in this field discuss how to define their work and purpose, NCER’s use of the terms may evolve

How many U.S. adults have low foundational skills?

Data from the 2017 Program for the International Assessment of Adult Competencies (PIAAC) indicate that roughly 114 to 135 million U.S. adults may have significant or moderate skill gaps in reading or numeracy, with approximately 48 million (23 percent) of adults having significantly low literacy and 69 million (33 percent) having significant low numeracy skill. Having low foundational skills may impede adults’ ability to pursue education or training, participate fully in the workforce, or engage civically.

What is the adult education system like?

Although millions of U.S. adults may benefit from building foundational skills, the adult education system and programs focus on adults with significant skill gaps and those who lack a high school diploma or equivalent.

Adult learners who enter into this system vary widely. They can include people from all regions, races, ethnicities, age groups, and levels of academic attainment, including those with no formal education to those with advanced degrees from other countries. They may be parents, workers, or retirees. They may seek out programs to learn how to read, earn a high school diploma, prepare for employment or college, or pass a citizenship test. The educators who teach adult learners are also diverse. Some have teaching certification, but many do not. The majority of them are part-time. There is also a wide variety of program providers (community colleges, community-based organizations, LEAs, etc.), relevant policies (federal, state, local), and funding sources. The National Reporting System for Adult Education includes descriptive data on students and programs reported to the US Department of Education as part of annual reporting requirements for grantees.

Why is research on this area important? What types of work has NCER supported?

Because of the diversity of learners in the system and the complexity of the system itself, the way forward maybe be unclear without a solid research base to inform policy and practice. Research can provide the knowledge, innovations, and evidence that can help learners, educators, and policymakers make informed choices and decision.

The adult education and foundational skills research portfolio at NCER is small but expanding. Between 2004 and 2021, NCER has invested approximately $51.8 million across 27 awards (grants and contracts) relevant to adults with low foundational skills who are in or eligible for adult education services. Please see a list of the NCER adult education and foundational skills projects funded since 2004: Word file or PDF file).

In the early years of NCER, research addressing adult education and foundational skills tended to focus only on reading, and adults were not the primary or sole focus of the study. Over time, this research has grown to encompass additional skills and education policy, and more studies focused primarily on adults served by the adult education system. The earliest NCER study in this set was funded in 2004 and is a direct ancestor to a 2016 grant to validate a reading assessment specifically for adult struggling readers. Other adult education projects have also built off of early NCER work and have inspired other projects. For example, NCER’s first adult education research and development center, the Center for the Study of Adult Skills (CSAL), incorporated work from one of NCER’s first grants in 2002 and led subsequent development research. Multiple awards in the adult education portfolio also use the PIAAC, as a resource for both research (see here, here, and here) and training grants.

What is on the horizon for NCER research?

One significant recent trend in this portfolio has been the expanding role of technology and digital skills. From CSAL to the CREATE Adult Skills Network and to projects exploring adult problem solving in digital environments, NCER researchers are building knowledge and developing interventions that focus on technology and adult learners’ ability to benefit from technology.

In particular, the CREATE Adult Skills Network is bringing together multiple research teams around technology to support reading, writing, math, professional development, civics/history and English language instruction, and assessment. CREATE is also helping to disseminate information about the role of technology in adult education and the importance of developing adults’ foundational skills.

How can people learn more?

Please visit the project homepage for the CREATE Adult Skills Network and sign up for their newsletters. The network also hosts blogs and podcasts, both of which include discussions of research. People can also visit the IES-wide topic page for adult skills, which curates examples of the work happening across IES relevant to adult education or adult foundational skills.

Dr. Meredith Larson (Meredith.Larson@ed.gov) is a research analyst and program officer at NCER. Her focus areas include postsecondary teaching and learning, adult education, and postdoctoral research training. She was trained in cognitive and instructional psychology and psycholinguistics and has served as a volunteer tutor for refugee children and in adult basic and adult secondary education programs.