IES Blog

As part of the IES 20^th Anniversary, NCER is reflecting on the past, present, and future of adult education research. In this blog, Dr. Daphne Greenberg, Distinguished University Professor at Georgia State University, reflects on her connection to adult education and gives advice to researchers interested in this field. Dr. Greenberg has served as the principal investigator (PI) on three NCER grants, including the Center for the Study of Adult Literacy, and is also co-PI on three current NCER grants (1, 2, 3). She helped found the Georgia Adult Literacy Advocacy group and the Literacy Alliance of Metro Atlanta and has tutored adult learners and engaged in public awareness about adult literacy, including giving a TedTalk: Do we care about us? Daphne Greenberg at TEDxPeachtree.

What got you interested in adult education research?

During graduate school, I was a volunteer tutor for a woman who grew up in a southern sharecropper family, did not attend school, and was reading at the first-grade level. Her stories helped me understand why learning was important to her. For example, her sister routinely stole money from her bank account because she couldn’t balance her checkbook.

I began wondering whether adults and children reading at the same level had similar strengths and weaknesses and whether the same word-reading components were equally important for them. I later published an article that became a classic in adult literacy research about this.

Over the years, I have grown to admire adult learners for their unique stories and challenges and am deeply impressed with their “grit” and determination even when faced with difficulties. When I watch a class of native-born adults reading below the 8th grade levels, I am inspired by them and yet deeply conflicted about our K-12 system and how many students aren’t getting what they need from it.

How does your personal experience influence your perspective?

I think my childhood and family planted the seeds. My grandfather ran a grocery store but had only a third-grade education. My parents were immigrants who worked hard to navigate a new culture and language, and I struggled with reading in English and English vocabulary growing up. As a result, I understand how people hide and compensate for academic weaknesses.

Also, my brother has profound autism. As a child, I insisted that I could teach him many skills, and I did. This taught me patience and the joy one feels when even the smallest gain is made.

As an adult, I mess up idioms, use Hebraic sentence structure, and need help with editing. I also have a visual condition that causes me to miss letters when I read and write. These difficulties help me relate to the struggles of adult learners. I often feel deep embarrassment when I make mistakes. But I am very fortunate that I have colleagues who celebrate my strengths and honor my weaknesses. Not all of our adult learners are as fortunate.

What should researchers new to adult education know about the system?

Adult education serves students with significant needs and numerous goals—from preparing for employment or postsecondary education to acquiring skills needed to pass the citizenship exam or helping their children with homework. But the adult education system has less public funding than K-12 or postsecondary systems.

Many of the educators are part-time or volunteers and may not have a background in teaching—or at least in teaching adults. There just aren’t the same level of professional development opportunities, technological and print instructional resources, infrastructure, or supporting evidence-based research that other education systems have. 

What should researchers know about adult learners?

As a starting point, here are three things that I think researchers should know about adult learners:

• What it means to live in poverty. For example, I once worked with a researcher who, when told that adult learners wouldn’t have access to the internet, replied “That’s not an issue. They can take their laptops to a Starbucks to access the Internet.”
• That adult learners are motivated. The fact that they have inconsistent attendance does not mean that they are not motivated. It means that they have difficult lives, and if we were in their shoes, we would also have difficulty attending on a regular basis.
• That adult learners’ oral vocabulary often matches their reading vocabulary. If you want adult learners to understand something, such as informed consents, realize that their oral vocabulary often is very similar to their reading grade equivalencies and consider simplifying complex vocabulary and syntax structure.

What specific advice do you have about conducting research with adult learners?

Testing always takes longer to complete than anticipated. I never pull students out from classes for testing because their class time is so precious. So they have to be available after or before class to participate in research, and this can be problematic. We often need to reschedule an assessment because public transportation is late, a job shift suddenly changes, or a family member is sick.

Finding enough of particular types of students is difficult because sites often attract different demographics. For example, one site may have primarily 16- and 17-year-olds, another site may have mostly non-native speakers, and another site may have either lower- or higher-skilled adult learners.

Having a “clean” comparison group at the same site is challenging because of intervention “leakage” to nonintervention teachers.  Adult education teachers are often so hungry for resources that they may try to peak into classrooms while an intervention is in process, get access to materials, or otherwise learn about the intervention. Their desire for anything that might help students makes contamination a concern.  

What areas of adult education could use more research?

I think that policymakers and practitioners would benefit from many areas of research, but two come to mind.

• How to measure outcomes and demonstrate “return”: Many funding agencies require “grade level” growth, but it can take years for skills to consolidate and manifest as grade level change. In the meantime, adults may have found a job, gotten promoted, feel more comfortable interacting with their children’s schools, voted for the first time, etc. Are we measuring the right things in the right way? Are we measuring the things that matter to students, programs, and society? Should life improvements have equal or even more weight than growth on standardized tests? After how much time should we expect to see the life improvements (months, years, decades)?
• How to create useful self-paced materials for adults who need to “stop-out”: Due to the complexities of our learners’ lives, many have to “stop-out” for a period before resuming class attendance. These adults would benefit from resources that they could use on their own, at their own pace during this time. What is the best practice for delivery of these types of resources? Does this “best practice” depend on the adult’s ability level? Does it depend on the content area? 

Any final words for researchers new to adult education?

I extend a warm welcome to anyone interested in research with adult learners. You will discover that many adult learners are eager to participate in research studies. They feel good helping researchers with their work and are hopeful that their time will help them or be of help to future learners. I highly recommend that you collaborate with researchers and/or practitioners who are familiar with the adult education context to help smooth the bumps you will inevitably experience.

This blog was produced by Dr. Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer at NCER.

In celebration of IES’s 20^th anniversary, we’re telling the stories of our work and the significant impact that—together—we’ve had on education policy and practice. As IES continues to look towards a future focused on progress, purpose, and performance, Dr. Matthew G. Springer of the University of North Carolina at Chapel Hill, discusses the merit pay debate and why the staying power of IES will continue to matter in the years to come.

Merit Pay for Teachers

There are very few issues that impact Americans more directly or more personally than education. The experience of school leaves people with deep memories, strong opinions, and a highly localized sense of how education works in a vast and diverse country. Bringing scientific rigor to a subject so near to people’s lives is incredibly important — and maddeningly hard. The idea of merit pay inspires strong reactions from politicians and the general public, but for a long time there was vanishingly little academic literature to support either the dismissive attitude of skeptics or the enthusiasm of supporters.

Given the stakes of the merit pay debate—the size of the nation’s teacher corps, the scale of local, state, and federal educational investments, and longstanding inequities in access to highly effective teachers—policymakers desperately need better insight into how teacher compensation might incentivize better outcomes. Critics worry merit pay creates negative competition, causes teachers to focus narrowly on incentivized outcomes, and disrupts the collegiate ethos of teaching and learning. On the other hand, supporters argue that merit pay helps motivate employees, attract and retain employees in the profession, and improve overall productivity.

Generating Evidence: The Role of IES-Funded Research

That’s precisely the kind of need that IES was designed to meet. In 2006, with the support of IES, I joined a group of research colleagues to launch the Project on Incentives in Teaching (POINT) to test some of the major theories around merit pay. We wanted to apply the most rigorous research design—a fully randomized control trial—to the real-world question of whether rewarding teachers for better student test scores would move the needle on student achievement.

But orchestrating a real-world experiment is much harder than doing an observational analysis. Creating a rigorous trial to assess merit pay required enormous diplomacy. There are a lot of stakeholders in American education, and conducting a meaningful test of different teacher pay regimes required signoff from the local branch of the teacher’s union, the national union, the local school board and elected leadership, and the state education bureaucracy all the way up to the governor. Running an experiment in a lab is one thing; running an experiment with real-world teachers leading real, live classrooms is quite another.

With IES support to carry out an independent and scientifically rigorous study, POINT was able to move forward with the support of the Metropolitan Nashville Public Schools. The results were not what most anticipated. 

Despite offering up to $15,000 annually in incentive pay based on a value-added measure of teacher effectiveness, in general, we found no significant difference in student performance between the merit-pay recipients and teachers who weren’t eligible for bonuses. 

As with all good scientific findings, we strongly believed that our results should be the start of a conversation, not presented as the final word. Throughout my career, I’ve seen one-off research findings treated by media and advocacy organizations as irrefutable proof of a particular viewpoint, but that’s not how scientific research works. My colleagues and I took care to publish a detailed account of our study’s implementation, including an honest discussion of its methodological limitations and political constraints. We called for more studies in more places, all in an effort to contest or refine the limited insights we were able to draw in Nashville. “One experiment is not definitive,” we wrote. “More experiments should be conducted to see whether POINT’s findings generalize to other settings.”

How IES-Funded Research Informs Policy and Practice 

Around the time of the release of our findings, the Obama administration was announcing another investment in the Teacher Incentive Fund, a George W. Bush-era competitive grant program that rewarded teachers who proved effective in raising test scores. We were relieved that our study didn’t prompt the federal government to abandon ship on merit pay; rather, they reviewed our study findings and engaged in meaningful dialogue about how to refine their investments. Ultimately, the Teacher Incentive Fund guidelines linked pay incentives with capacity-building opportunities for teachers—things like professional development and professional learning communities—so that the push to get better was matched by resources to make improvement more likely.  

While we did not have education sector-specific research to support that pairing, it proved a critical piece of the merit pay design puzzle. In 2020, for example, I worked with a pair of colleagues on a meta-analysis of the merit pay literature. We synthesized effect sizes across 37 primary studies, 26 of which were conducted in the United States, finding that the merit pay effect is nearly twice as large when paired with professional development. This is by no means the definitive word, but it’s a significant contribution in the incremental advancement of scientific knowledge about American education. It is putting another piece of the puzzle together and moving the education system forward with ways to improve student opportunity and learning.

That’s the spirit IES encourages—open experimentation, modesty in drawing conclusions, and an eagerness to gather more data. Most importantly, we wanted our findings to spark conversation not just among academic researchers but among educators and policymakers, the people ultimately responsible for designing policy and implementing it in America’s classrooms.

Pushing that public conversation in a productive direction is always challenging. Ideological assumptions are powerful, interest groups are vocal, and even the most nuanced research findings can get flattened in the media retelling. We struggled with all of that in the aftermath of the POINT study, which arrived at a moment of increased federal interest in merit pay and other incentive programs.  Fortunately, the Obama administration listened to rigorous research findings and learned from the research funding arm of the U.S. Department of Education.

This is why the staying power of IES matters. The debates around education policy aren’t going away, and the need for stronger empirical grounding grows alongside the value of education and public investment. The POINT experiment didn’t settle the debate over merit pay, but we did complicate it in productive ways that yielded another critically important finding nearly eight years later. That’s a victory for education science, and a mark of progress for education policy.

Matthew G. Springer is the Robena and Walter E. Hussman, Jr. Distinguished Professor of Education Reform at the University of North Carolina at Chapel Hill School of Education.

This blog was produced by Allen Ruby (Allen.Ruby@ed.gov), Associate Commissioner for Policy and Systems Division, NCER.  

In a first for ERIC, the supply of education research content the program aspires to index is regularly exceeding its capacity to do so. For the past 15 years, ERIC has consistently indexed 4,000 records a month. This pace has allowed us to index all approved sources without significant delays. However, over the past two years, the volume of content published in our approved sources has doubled. This has resulted in a backlog of publishable content and, as a result, ERIC cannot index new work in a timely fashion.

As part of our periodic collection analyses, we have been investigating potential causes of this backlog. We have found that several journals are publishing far more content than when they were originally selected to be included in ERIC. For example, one journal was publishing fewer than 20 articles per year when approved, but now is publishing over 850 per year. This is close to a 5000% increase in production.

An increased volume of published work in the education sciences—as a whole—is a good thing. However, when an individual journal dramatically increases the number of articles it publishes, it is noteworthy. In those instances, ERIC wants to ensure that the journal is still adhering to the standards and criteria it met when originally included in the index, including rigorously applying the peer-review process, if applicable, and maintaining its original aim and scope. Both are important to ensuring that work contained in ERIC is of high quality and that a wide range of key topics in education can be indexed.

ERIC’s Selection Policy already requires an ongoing review of currently indexed sources, including identifying sources that may no longer meet the Policy’s standard and criteria. As part of that review, ERIC will now identify journals that have published over twice as many articles from the year it was accepted and flag them for further review. As part of that review, ERIC will assess whether the increase is temporary and associated with a unique event, like a special issue, or if the increase reflects an ongoing trend. If the review indicates the increase is persistent, the ERIC team will recommend that indexing of that journal be paused. If a pause is approved, ERIC will stop indexing subsequent issues for a two-year period. The journal will also be removed from ERIC’s Journals List, because this list only contains actively indexed journals.

After the two-year pause, ERIC will re-review the journal. If ERIC reinstates the journal, ERIC will notify the journal and will index any content published during the two-year pause. If the journal is not reinstated, ERIC will not index any issues published during the pause.

This decision has led to a few questions:

• Why can’t ERIC index everything? ERIC is funded by the U.S. Department of Education’s Institute of Education Sciences and has limited resources. It must prioritize indexing the highest quality education research, include content for all topic areas, and can only index a set number of records per month.
• What is the concern with the increased volume? Increased volume may signal that the journal has changed scope to accommodate a broader set of articles or that quality assurance processes have been affected in a rush-to-publish environment. Particularly among journals that were considered to be peer reviewed when originally accepted into ERIC, the publication of articles shortly after submission may signal a substantive change in a journal’s quality assurance process such that they no longer meet the criteria needed to receive that designation. There is also a concern that if a journal greatly produces more records than estimated, the collection will get skewed in a way which would favor one topic area over another.
• How will ERIC identify the journals to pause? As part of the source selection process, ERIC will monitor two years of current publishing and compare the number of articles published to the number published during the year the journal was selected for ERIC.
• Why is the pause for two years? The ERIC Selection Policy says that sources may be reviewed after 2 years (24 months). To be consistent with this policy, we will automatically review paused sources after this same time frame.
• How will I know if my journal is paused? ERIC will email the journal representatives to inform them of this decision in the coming weeks.
• Can my journal appeal the decision to be paused? Journals may not appeal the decision to pause indexing. However, if at the end of the pause period the journal is not automatically reinstated, journals may apply for re-review 24 months later.
• What is ERIC looking for in the automatic review at the end of the pause period? ERIC will conduct a full review of the journal and consider the two years of published content against the criteria set forth in the ERIC Selection Policy.
• Can authors submit their article published in a “paused” journal via online submission during this period? No, authors must wait for a re-review of the journal to be conducted after the two-year pause. If the journal is selected again, the articles from the paused issues will be indexed in ERIC.

In honor of Career and Technical Education (CTE) Month, we asked principal investigator Dr. Lee Kern how her intervention research reduces dropouts and prepares students with emotional and behavioral disorders (EBD) for college and career readiness (CCR). The purpose of her current IES project is to develop and pilot test an intervention, Supported College and Career Readiness (SCCR), that augments typical school-based college and career readiness activities for students at or at risk for EBD.

What motivated you to conduct this research?

Given the high dropout rate among students with EBD, I am interested in strategies that keep them in school. Because post-graduation experiences serve as important indicators of positive educational outcomes, I want to establish a stronger connection between school and life after school to ensure that students are fully prepared. My co-PIs, Jennifer Freeman and Chris Liang, were motivated to collaborate on the current research project as well because of their unique focus on different aspects of CCR, allowing us to address multiple dimensions in the development of our intervention.

Can you provide us with an update on the project? What work have you completed to date on the development of the SCCR program?

We recognized and addressed a gap in the college and career readiness literature with this group of students. During the first 2 years of the project, we completed two literature reviews and two conceptual papers, which are in press, and we are in the process of completing a third literature review. Our completed literature reviews indicated (a) limited attention to CCR for individuals with emotional and behavioral problems, (b) lack of defined components of CCR interventions, (c) the need to evaluate the effectiveness of CCR interventions with students of color, and (d) aspects of CCR interventions that might be important for individuals with diverse sexual identities. These papers helped us develop our multi-component CCR intervention for students with or at risk for EBD.

The development phase was vital to creating our multi-component program. Schools practice different approaches to college and career preparation, so we needed to create a flexible program that could fit the many permutations in course scheduling, career interest assessments, career exposure activities, and other factors. Receiving teacher and student feedback on the program during the second year of the project was helpful and appreciated as we refined SCCR. We initiated a randomized controlled trial and ran the study in four schools this academic year. We will expand the research into four additional schools in the 2023-24 academic year.

What other types of research are needed to move forward in the field of CTE for students with or at risk for EBD?

Although we know that students, especially those with or at risk for EBD, need more preparedness for college or their future careers, research must specify intervention components that result in improved outcomes in these areas. Also, it must determine whether the interventions are effective across diverse groups of students and ascertain adaptations that address the needs of all students. Existing and ongoing research must be conducted to better assess student skills. Identifying assessments directly linked to critical and effective interventions that practitioners can implement will be important for future progress.

NCSER looks forward to learning the results of the pilot study to better understand the promise of the SCCR program for improving the college and career readiness of students with or at risk for EBD. For more highlights on the CTE-related work that IES is supporting, please check out our IES CTE page. 

Dr. Kern is a professor and the director of the Center for Promoting Research to Practice at Lehigh University. She has more than 30 years of experience in special education, mental health, and behavior intervention for students with EBD.

This CTE blog post was produced by Alysa Conway, NCSER student volunteer and University of Maryland, College Park graduate student. Akilah Nelson is the program officer for NCSER’s Career and Technical Education grants.

As part of our 20^th anniversary celebration and in recognition of Black History Month, we asked Dr. Jason Snipes, Director of Applied Research for REL-West at WestEd, to discuss his inspiration for his IES-funded replication study. The purpose of the study is to test the potential of self-affirmation interventions to counteract the harmful effects of negative racial stereotypes on the academic, disciplinary, and psychosocial outcomes of 7^th grade Black and Latino students.

What motivates your research on the effects of self-affirmation interventions on Black and Latino student outcomes?

My mother was a civil rights activist, and I was raised with a clear sense of my history as a Black person and the importance of making a contribution worthy of those that preceded me. Her example inspired me to pursue a career in research on education and youth development and to focus on finding, testing, and understanding strategies for improving outcomes for the Black, Latino, and other students systematically underserved by our education system.

My specific interest in stereotype threat and self-affirmation research stems in part from my own education experiences. I still remember how—despite being in the gifted and talented program at my elementary school—every mistake I made, every question I asked, every idea I expressed was greeted with the snickers and whispers of my peers crudely expressing their doubts about my intelligence. I remember my success slipping away, and before I knew it, being in 8th grade remedial math—failing. My father essentially saved my life. He somehow taught me to truly believe that I could accomplish anything I wanted to. This is not a solution to systemic racism. Still, his support for changing my beliefs about myself, combined with going to a new high school, completely changed my academic trajectory.

I again felt the weight and pressure of low expectations in graduate school, in the subtle and not-so-subtle ways my White professors expressed their doubts about my ability to succeed in a rigorous PhD program.

So, later in my career, when I learned about stereotype threat and self-affirmation, I saw a bit of myself and my life experiences. I saw something else that I found unusual: an intervention with significant effects, even when tested in rigorous randomized trials. I wanted to know more about where, how, and under what circumstances it could be effectively used support Black and Brown children.

What is stereotype threat and how might it impact Black students?

Among Black students, stereotype threat is the fear of confirming negative racial stereotypes about their academic performance and their underlying intelligence. It can be one of the many persistent and pervasive psychological stressors that Black people encounter on a daily basis. Randomized trials show that when prompted to believe a test is an assessment of their intelligence, Black students perform more poorly. The prompt generates physiological and psychological stress responses, reduces available working memory, and results in both fewer questions answered in a given period of time and a lower percentage of correct answers among those given. The same prompt has no effect on White or Asian students. A meta-analysis of 300 studies suggests stereotype threat accounts for a quarter to a third of the Black-White and Latino-White achievement gaps.

How does the self-affirmation intervention you are evaluating address stereotype threat?

The self-affirmation intervention, created by Claude Steele and Joshua Aronson, is designed to respond to stereotype threat. It’s a set of four 15-minute writing exercises administered over the course of a school year. Each exercise provides students with the opportunity to affirm their value by asking them to write about things that are important to them. Experiments with college and middle school students show that self-affirmation improves a variety of academic outcomes for Black students, and that that these effects persist and grow over time.

Our study goes beyond prior research to provide new evidence about the impact of self-affirmation on Black and Latinx students in schools with different demographic compositions and the extent to which its effects generalize across a nationally representative sample of schools. Some studies suggest that self-affirmations effects are smaller in schools with higher concentrations of Black and/or Latino students. We plan to systematically explore this and other questions about moderators. Our findings will have implications for the settings in which self-affirmation ought to be scaled and implemented and how it might be used as a complement to other available supports to bolster the success of Black and Latinx students.

That stereotype threat appears to account for a quarter of observed racial achievement gaps, and that self-affirmation ameliorates this effect makes self-affirmation relevant to larger discussions of educational equity. Self-affirmation, along with other psychosocial interventions, should be investigated as potential tools for reducing racial disparities in education outcomes. That said, it is important to remember that racial inequity is a feature of the education system and the institutions that surround it, not a function of some sort of “flaw” in the attitudes or psychosocial make up of Black and Brown students themselves. While these approaches may help buffer Black and Latino students against the full consequences of racism, bias, and stereotyping, we should never allow ourselves to be confused. The fundamental problem is not Black and Latino students’ ability to cope with these dynamics, but the presence of these dynamics in and of themselves. Furthermore, psychosocial interventions are not a substitute for high quality instruction or solutions to other systemic problems (for example, de facto segregation) that have powerful negative effects on academic outcomes among Black and Latino students.

Self-affirmation is also relevant to discussions of racial equity in education because the intervention reflects a fundamental concept underlying racial equity: personhood. It offers students a chance to affirm their value as human beings, and this may be one of the mechanisms through which it helps disrupts the destructive cycle of stereotype threat. This simple assertion embodies a core idea underlying the civil rights movement: that racial equity requires recognizing and treating Black and Brown people as fully human.   

Your current IES study aims to replicate prior research on self-affirmation. Why is replication important?

Too often, I have seen researchers, policy makers, practitioners, and funders make the mistake of misinterpreting results from a single, even rigorously designed, study as answering the question of whether an intervention or strategy works. Reality is more complex. What we can learn from a single study is usually something closer to the extent to which an approach worked in this place (or places) at this time. Under pressure for answers to pressing policy problems, we may rush to scale approaches or interventions with evidence from one or two well designed studies, only to find out that they don’t work at scale, or in a subsequent implementation, and we don’t know why.

It may be better to ask, “To what extent does an approach generate impacts, under what circumstances does it do so, and why?” Systematically replicating initial causal studies, enables researchers to address these questions more effectively. Rather than guessing at post hoc explanations of the patterns we observe, replication systematically tests hypotheses regarding how implementation, context, and other moderators and mediators affect program impacts. Doing so prior to undertaking massive scaling efforts therefore helps reduce the extent to which money, effort, and, perhaps most importantly, public will, are expended on strategies with fundamental limitations. Replication enables us to more systemically study and understand mechanisms of action, improving the extent to which we implement or scale interventions in contexts and situations in which they are likely to be most effective.

This blog was produced by Katina Stapleton (Katina.Stapleton@ed.gov) and Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER Program Officers.