IES Blog

Institute of Education Sciences

When “More Research is Needed” Is the Key Finding: Improving the Evidence Base for Nonacademic Interventions for Postsecondary Success in Rural and High-Poverty Contexts

Stakeholders in rural and high-poverty districts in Regional Educational Laboratory (REL) Appalachia’s region have noticed a troubling trend: many students graduate from high school academically well prepared but fail to enroll in college or enroll in college only to struggle and drop out within the first year. Stakeholders believe these high-performing students may face nonacademic challenges to postsecondary success, such as completing financial aid paperwork, securing transportation and housing at colleges far from home, or adjusting to campus life. To address these challenges, education leaders are looking for interventions that address nonacademic competencies: the knowledge, skills, and behaviors that enable students to navigate the social, cultural, and other implicit demands of postsecondary study.

To fill this need, REL Appalachia researchers conducted a review of the existing evidence of the impact of nonacademic interventions – that is, those designed to address nonacademic competencies – on postsecondary enrollment, persistence, and completion. The review had a particular focus on identifying interventions that also have evidence of effectiveness in communities serving students similar to those in Appalachia—high-poverty, rural students. Only one intervention, Upward Bound, demonstrated impact in rural, high-poverty communities. The review showed that Upward Bound, as implemented in the early 1990s, benefited high-poverty rural students’ college enrollment, with no demonstrated impact on persistence or completion.

Schools and communities need access to nonacademic interventions that benefit students served in high-poverty rural communities. Researchers: read on to learn more about the methods used in the evidence review, its findings, and steps you can take to support rural and high-poverty communities in improving enrollment and success in postsecondary education!

Nonacademic challenges to postsecondary success for rural students

All students face nonacademic challenges to postsecondary success, but rural populations and high-poverty populations in particular may benefit from interventions addressing those challenges because they enroll in and complete college at significantly lower rates than their nonrural or low-poverty peers. Although academic challenges contribute to this gap, rural and high-poverty populations also face unique nonacademic challenges to postsecondary enrollment and success. For example, rural students are less likely to encounter college-educated role models and high-poverty students often face inadequate college counseling at their schools (see research here, here, and here). As a result, rural and high-poverty students may have inadequate access to knowledgeable adults who can help them understand the steps needed to enroll or prepare them for the challenges of persisting in postsecondary education.  Nonacademic interventions can support students in developing the knowledge, skills, and behaviors necessary to overcome these challenges and improve postsecondary enrollment and success for rural and high-poverty students.

The need for evidence-based interventions

To support decisionmakers at rural and high-poverty schools in identifying evidence-based nonacademic interventions, researchers at REL Appalachia conducted an extensive search of the published research. The search looked for rigorous studies of nonacademic interventions with evidence of positive impact on college enrollment, persistence, performance, and completion for students attending rural schools or who were identified as high poverty. The purpose of the project was to identify a suite of interventions to recommend to these education leaders.

The results of our review indicate there may be gaps in the evidence available to all decisionmakers who are trying to help their students succeed in postsecondary education. The search first identified any studies that focused on postsecondary outcomes of nonacademic interventions serving students ages 5–19. Of the 1,777 studies with the relevant keywords, only 65 focused on the postsecondary outcomes of nonacademic interventions. Next, we evaluated these 65 studies against the What Works Clearinghouse (WWC) design standards, which assess the quality of evaluation study designs. Only 17 studies met WWC’s rigorous study design standards with or without reservations. Finally, researchers from REL Appalachia identified studies that showed positive impacts on students overall, and studies that looked at rural students and students identified as high poverty in particular. Only eight studies showed positive, statistically significant impacts on students’ postsecondary enrollment or success overall. Of the eight studies that showed positive impacts of nonacademic interventions on postsecondary outcomes, only three focused on high-poverty populations, and only one reported specifically on rural populations.

This figure shows the number of studies remaining at each stage of screening. The original searches returned 1,777 unique studies. Of these, 65 focused on postsecondary outcomes of nonacademic interventions with students ages 5 to 19. At the next stage, 17 studies remained that met these criteria and also met WWC standards. At the final stage, 8 studies remained that met all criteria and had a positive effect on postsecondary outcomes.

 Without additional research that focuses on low-income and rural contexts, schools and districts are left to implement programs with limited or no evidence of effectiveness. For example, the Quantum Opportunity Program (QOP) provides mentors to students as part of a long-term after school program. However, WWC reviews of QOP studies (here and here) showed indeterminate effects of the program on postsecondary outcomes. The lack of evidence should not detract from the important role QOP has in serving students, but it leaves open the question of whether those efforts are having the intended effects. With few clear alternatives, schools and districts continue to implement programs with limited evidence of effectiveness.

Action steps

Nationwide, 19 percent of U.S. public school students are enrolled in a rural school, and 24 percent are enrolled in a high-poverty school. To help districts and schools provide effective supports to those students, researchers can provide high-quality evidence on the effectiveness of nonacademic interventions in these contexts.

Carry out more studies on specific interventions designed to improve nonacademic competencies. REL Appalachia’s review found that the research on nonacademic competencies often focuses on defining the competencies themselves, rather than on studying interventions designed to develop the competencies. Of the 1,777 unique studies identified in our review, only 65 (3 percent) studied outcomes of interventions designed to improve nonacademic competencies. From these, we identified only 17 studies, representing nine interventions, with sufficiently rigorous designs to examine evidence of effectiveness.

The limited availability of rigorous evaluations of interventions suggests that, as researchers, we need to increase our focus on evaluating new interventions as they are developed or tested. Decisionmakers rarely design their own programs or interventions from scratch; they need to be able to identify existing programs and policies that are within their power to implement and have been proven effective in similar communities. Researchers can help decisionmakers select and implement successful interventions by providing evidence on whether interventions that develop students’ nonacademic competencies have positive effects on students’ postsecondary outcomes.

Design studies to generalize to rural and high-poverty populations. As researchers, we can also increase our focus on rural and high-poverty populations. REL Appalachia’s review found only three studies that focused on a high-poverty population and one that focused on a rural population. As researchers, we can address this gap in two ways: (a) we can carry out more studies specifically focused on rural and high-poverty areas; and (b) when using large national datasets or multi-site studies, we can consider rural and high-poverty populations in our sampling and disaggregate our results for these populations.

Summary

Stakeholders in rural and high-poverty contexts are looking for nonacademic interventions that will be effective with their students. To that end, REL Appalachia carried out an extensive review of evidence-based interventions. The review found few rigorous studies of nonacademic interventions, and even fewer that examined findings for students identified as high poverty or in rural settings. Without additional research, schools and districts serving rural and high-poverty populations may implement interventions that are not designed for their circumstances and may not achieve intended outcomes. As a result, resources may be wasted while rural and high-poverty students receive inadequate support for postsecondary success.  In addition to investing in rigorous studies, which can take a long time to complete, researchers and practitioners can also collaborate to implement short-term research methods to identify early indicators of the success of these programs. For example, researchers may be able to support schools and districts in developing descriptive studies examining change over time or change in formative assessment outcomes.

 

 

 Researchers have a role in helping more high school graduates from rural communities enroll, persist, and succeed in postsecondary education.

 

Rural and high-poverty schools and districts have unique strengths and challenges, and the lack of information about how interventions perform in those contexts presents a dilemma for decisionmakers: do nothing, or else muddle through with existing evidence, investing in interventions that don’t address local needs. As researchers, we can help resolve this dilemma by providing rigorous evidence about effective interventions tailored to rural and high-poverty contexts, as well as supporting practitioners in using more accessible methods to investigate the short-term outcomes of the programs they are already implementing.

 

by Rebecca A. Schmidt and CJ Park, Regional Educational Laboratory Appalachia

 

Towards a Better Understanding of Middle-Schoolers’ Argumentation Skills

What is the difference between fact and opinion? How do you find relevant evidence and use it to support a position? Every day, teachers help students practice these skills by fostering critical discussions, a form of argumentation that encourages students to use reasoning to resolve differences of opinion.

In their IES-funded study, Exploring and Assessing the Development of Students' Argumentation Skills, Yi Song and her colleagues are uncovering activities (both teacher led and technology supported) that can improve middle-school students’ ability to generate better oral and written arguments.

This project began in 2019 and is working in classrooms and with teachers and students. The researchers have created a series of videos that describe their work. In this series, Dr. Song and her co-PIs, Dr. Ralph Ferretti and Dr. John Sabatini, discuss why the project is important to education, how they will conduct the research plan, and how educators can apply what they are learning in classrooms.

 

 


For questions and more information, contact Meredith Larson (Meredith.Larson@ed.gov), Program Officer, NCER

NCSER Principal Investigators Receive 2021 CEC Awards

This week, the Council for Exceptional Children (CEC) is holding its annual Convention and Expo virtually. Several scholars are being presented with awards during the event to recognize their research contributions to the field. IES-funded investigators Linda Mason, Nicholas Gage, and Patricia Snyder are among those recognized.

Photo of Dr. Linda Mason

Dr. Linda Mason, Endowed Director of the Kellar Institute for Human disAbilities at George Mason University, received the CEC Special Education Research Award. This award is given to an individual or team whose research has made significant contributions to the education of children and youth with exceptionalities. Dr. Mason’s research focuses on content reading comprehension and writing interventions to support students with learning disabilities in inclusive classrooms. She has been the Principal Investigator (PI) on two National Center for Special Education Research (NCSER) grants. For one, she developed a writing intervention for middle schoolers with behavior disorders. Currently, Dr. Mason is the PI for a project that is exploring the relationships among teacher use of evidence-based practices, teacher experience with and attitudes about adapting instruction for students with disabilities, and student writing outcomes.

Photo of Dr. Nicholas Gage

Dr. Nicholas Gage, Associate Professor at the School of Special Education, School Psychology, and Early Childhood Studies at the University of Florida, received the CEC Division for Research Martin J. Kauffman Distinguished Early Career Research Award. This award is given to an individual in recognition of outstanding scientific contributions in special education research within the first 10 years after receiving doctoral degree. Dr. Gage’s research focuses on identification of policies and practices to support the academic, social, and behavioral needs of students with or at risk for emotional or behavioral disorders. A former IES post-doctoral fellow, he has also served as the PI on a NCSER grant that developed a technology-based intervention to support students with visual impairments in locating key information in math word problems that include graphics. Currently, Dr. Gage serves as a mentor for an early career grant.

Photo of Dr. Patricia Snyder

Dr. Patricia Snyder, a Distinguished Professor and Director of the Anita Zucker Center for Excellence in Early Childhood Studies at the University of Florida, received the CEC Division for Research Kauffman-Hallahan-Pullen Distinguished Researcher Award. This award is given to an individual in recognition of research resulting in more effective services or education for exceptional individuals. Dr. Snyder’s research focuses on developing, validating, and evaluating interventions for infants, toddlers, and preschoolers with or at risk for disabilities or experiencing social and behavioral challenges. Dr. Snyder has been the Principal Investigator number of NCSER-funded research grants. She and her colleagues designed a professional development program, Tools for Teachers, to support teachers in using embedded instruction with preschool children with disabilities. Following up on the initial grant, she and her colleagues recently completed an efficacy trial of Tools for Teachers to examine the impact of the professional development intervention on teacher practices and child outcomes and she is serving as a co-PI for a grant developing Tools for Families, a program for teachers to engage families in embedded instruction for learning across school and home. Dr. Snyder has also been a co-PI for a study examining the efficacy of the Pyramid Model, a class-wide model aimed at promoting social-emotional development and positive behavior for preschool children; a project that developed Embedded Practices and Intervention with Caregivers, an early intervention program aimed at coaching caregivers of infants and toddlers to embed learning opportunities in every day routines; and a current project developing a professional development intervention focused on teaching vocabulary to children at risk for communication difficulties. Finally, she has been actively involved in training the next generation of researchers, serving as PI on a postdoctoral training grant focused on preparing postdoctoral fellows to conduct research on improving outcomes for young children with or at risk for disabilities and serving as a mentor for an Early Career program grant.

Congratulations to the award recipients!

This blog was authored by Alice Bravo (University of Washington), IES intern through the Virtual Student Federal Service, and Amy Sussman, Program Officer at NCSER.

"Boys Have It; Girls Have to Work for It": Examining Gender Stereotypes in Mathematics Achievement

In 2020, Andrei Cimpian, along with co-PIs Sapna Cheryan, Joseph Cimpian, and Sarah Lubienski, were awarded a grant for “Boys Have It; Girls Have to Work for It”: The Development and Consequences of Gender Stereotypes About Natural Talent vs. Effort in Mathematics. The goal of this project is threefold: 1) to explore the origins of the gender stereotype that girls achieve in math due to effort and boys achieve in math due to natural talent, 2) to investigate the consequences of these stereotypes, and 3) to identify ways of reducing the negative effects of these stereotypes on mathematics outcomes. In this blog, we interviewed Dr. Andrei Cimpian on his inspiration and insights on this research, as well as his plans to disseminate the findings to education practitioners.

 

Dr. Andrew CimpianWhat spurred your research, and what prior research was foundational for this current study? 

The co-PIs and I were inspired to do this research because we were struck by the contrast between two sets of facts. On the one hand, girls do better in school than boys from kindergarten to grade 12. Women also obtain more bachelor’s and graduate degrees than men. On the other hand, we as a society still think of men as more brilliant and genius-like than women. For example, participants in a 2018 study referred more male than female acquaintances for a job that they were told requires natural smarts1. When the same job was said to require a strong work ethic instead, participants referred equal numbers of women and men.

Of course, societal views of women and men have changed quite a bit over the last century. With respect to general competence, women are now equal with men in the eyes of the American public. But the stereotype that associates “raw,” high-level intellectual talent with men more than women seems to have resisted change. Why?

Our research is testing a promising hypothesis: It is possible that people give different explanations for women’s and men’s intellectual successes, explaining men’s competence as being due primarily to their inborn intellectual talent and women’s as being due to their efforts. This effort-vs.-talent stereotype “explains away” women’s achievements by attributing them to a quality—perseverance—that is less valued in American culture than natural ability.

Versions of this explanatory stereotype have been documented in adults, but our project will provide its first systematic investigation among children. In particular, we will investigate the effort-vs.-talent stereotype in the domain of mathematics because innate ability is particularly valued in this domain1, which might make this stereotype especially consequential.

 

What are some examples of language or behavior that might suggest an individual holds a particular stereotype? Are there potential ways of mitigating the negative effects of stereotypes?

The best example of this stereotype that I can think of—and this is in fact the anecdote that crystallized our team’s interest in this topic—was recounted by co-PI Joseph Cimpian in a recent piece for The Brookings Institution (emphasis is mine):

About five years ago, while Sarah [Lubienski] and I were faculty at the University of Illinois, we gathered a small group of elementary teachers together to help us think through […] how we could intervene on the notion that girls were innately less capable than boys. One of the teachers pulled a stack of papers out of her tote bag, and spreading them on the conference table, said, “Now, I don’t even understand why you’re looking at girls’ math achievement. These are my students’ standardized test scores, and there are absolutely no gender differences. See, the girls can do just as well as the boys if they work hard enough.” Then, without anyone reacting, it was as if a light bulb went on. She gasped and continued, “Oh my gosh, I just did exactly what you said teachers are doing,” which is attributing girls’ success in math to hard work while attributing boys’ success to innate ability. She concluded, “I see now why you’re studying this.”

In terms of what can be done to mitigate the effects of this stereotype, our project will investigate a potential strategy: normalizing effort by making it clear to students that everyone (not just particular groups) needs to work hard to learn math. This message reframes what is viewed as necessary for success in math away from the belief that natural talent is key, thereby undercutting the power of effort-vs.-talent stereotypes.

 

The current study focuses on elementary school students in grades 1 through 4. What was the motivation for choosing this specific age group?

In general, gender stereotypes about intellectual ability seem to emerge quite early. For instance, girls as young as 6 and 7 are less likely than boys to associate being “really, really smart” with members of their own gender. For this reason, we think it is really important to focus on young children—we need to understand when effort-vs.-talent stereotypes first take root!

“Catching” these stereotypes when they first arise is also important for intervention purposes. If left unchecked, the effects of effort-vs.-talent stereotypes may snowball over time (for example, differences in the types of careers that young women and men are motivated to pursue).

 

What plans do you have to disseminate the findings of this research in ways that will be useful for education practitioners? 

We are mindful of the importance of getting this research into the hands of teachers so that they can use it in practice. We hope to write articles on this work for media outlets that draw educationally oriented audiences. To reach parents as well, we will coordinate with popular media outlets to disseminate the results of this work to general audiences. More generally, we will make every effort to ensure that the findings have maximal societal impact, raising awareness of effort-vs.-talent stereotypes among parents, educators, and the general public.

 


Andrei Cimpian, PhD (@AndreiCimpian), Professor of Psychology at New York University, has conducted extensive research on children’s conceptual development, explanations, and motivation in school.

Written by Yuri Lin (ylin010101@g.ucla.edu), intern for the Institute of Education Sciences.

Photo credit: Brian Stauffer


1The full PDF and resources are available at https://www.cimpianlab.com/motivation.

Advancing High-Quality Data and Evidence at the U.S. Department of Education

March 5, 2021: A post from Greg Fortelny, Chief Data Officer and Matt Soldner, Evaluation Officer, U.S. Department of Education

Last year, the education landscape changed dramatically as the effects of the coronavirus swept across the country. Overnight, families were confronted with the twin challenges of keeping their children, loved ones, and communities safe while establishing learning environments which enabled students to succeed and achieve. With each passing day, our schools are one step nearer recovery. But here at the U.S. Department of Education (ED), our work is far from done. Among the many lessons learned in the wake of the pandemic is that we must take full advantage of every opportunity to strengthen education systems and improve outcomes for all learners. From where we sit, making the most of those opportunities depends on two things: high-quality data and evidence.

Basing education policy and practice in strong evidence that is rooted in high-quality data can accelerate learning for all students, speeding efforts to recover from the pandemic’s effects. As the stewards of education data and evidence at ED, it is our charge from the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) to improve the collection, analysis, and use of high-quality data and evidence. By doing so, we hope to help educators and policymakers at the federal, state, and local levels make the most effective decisions possible on behalf of the learners, families, and communities they serve.

In the two years since the passage of the Evidence Act, the Department’s Office of the Chief Data Officer (OCDO) has made progress in supporting ED’s mission to improve education outcomes by effectively leveraging data to support evidence-based policy and data-driven decision-making.  The Department’s Data Governance Board (DGB) was created to lead these efforts and, with the launch of its inaugural Data Strategy in December 2020, ED has established guidance and goals to go further to improve data quality and enable evidence-building in service of our nation’s learners.

The work of evidence-building is a collaborative effort, coordinated by ED’s Evaluation Officer housed at the Department’s Institute of Education Sciences. In this first phase of Evidence Act implementation, the Department has published a new agency-wide evaluation policy that governs the generation of its most rigorous evidence and is preparing to release its inaugural Annual Evaluation Plan. As part of the agency’s strategic planning process, ED will also develop and publish its first-ever Learning Agenda, documenting its evidence-building priorities for the next four years.

Even prior to the passage of the Evidence Act, ED has made data and evidence a priority. For decades, ED has been collecting and publishing data on students, teachers, schools, colleges, grants, student aid and more.  Now, with the launch of the ED’s Open Data Platform (ODP) in December 2020, educators, researchers, stakeholders, decision-makers, and the public can explore the array of taxpayer-funded education data and profiles through a user-friendly interface, with all data accessible from one central online repository.

At OCDO, we developed the ODP to link to research and ED data tools that serve to engage and inform the public through various displays of that publicly available data.  One rich example of these tools is the recently enhanced College Scorecard.  Visited by more than 1.4 million users in 2020, ED’s College Scorecard now enables students and their advocates to more easily search field of study identifiers and compare similar fields of study within an institution or across different institutions.  And with recent updates including  information on loan repayment rates and parent PLUS loan debt, prospective students now have even more data to make more informed enrollment decisions and to find the right postsecondary fit. 

In addition to making existing data more accessible to decision-makers, the Department invests in new discoveries in the education sciences that have the potential to dramatically improve student outcomes and strengthen education systems. For nearly 20 years, the Department’s Institute of Education Sciences (IES) has worked to bring rigorous, independent, and objective education statistics, research, and evaluation to bear on challenges from early childhood to adult and postsecondary education.   

Through its National Center for Education Evaluation and Regional Assistance (NCEE), IES supports several programs dedicated to improving the use of data and evidence in education practice. NCEE’s Regional Educational Laboratories (REL) program works in partnership with state and local educators and policymakers to develop and use research that improves academic outcomes for students. It’s What Works Clearinghouse™ reviews existing research on education programs, practices, and policies in education to help families, teachers, and leaders answer the question “what works” in the nation’s schools, colleges, and universities. And, through its Evaluation Division, NCEE conducts independent, high-quality evaluations of education programs supported by federal funds.

In recent months, much of the work of both OCDO and IES has pivoted to address the effects of the coronavirus pandemic. At IES, we have developed a wide range of COVID-related resources for families, educators, and policymakers. And our National Center for Education Statistics has recently announced a new survey designed to collect vital data on schools’ approaches to learning during the pandemic, critical to safely reopening America’s schools and promoting educational equity.  

OCDO also has also created valuable new resources in response to the pandemic. The new Education Stabilization Fund Public Transparency Portal provides public transparency and accountability for the over $30 billion in Elementary and Secondary School Emergency Relief, the Governor's Emergency Education Relief, and the Higher Education Emergency Relief funds established through the Coronavirus Aid, Relief, and Economic (CARES) Act. The grant funds were awarded to states, schools, and institutions of higher education last spring. Continuously updated to reflect new activity, this portal provides the public with accurate, reliable, and accessible data on one of the largest federal investments in education in our country’s history.  The portal will soon include similar accounting of the awards made to states, districts, and colleges through the $81.9 billion in Education Stabilization Funds authorized through the Coronavirus Response and Relief Supplemental Appropriations (CRRSA) Act, 2021. 

Despite the challenges we face, there is optimism, like the spark of an engaged student or the light of an inspired educator; we are eager to continue the work to serve learners through data.  The critical data priorities of ED are to empower users and leverage the data to address education equity gaps too often borne by our nation’s underprivileged students.  Rigorous evaluation identifies effective policies and practices, open and transparent data furthers research and public trust. Leveraging data to inform decisions not only improves ED operations but also helps guide schools and families in their efforts to support students and improve education outcomes. 

Your feedback is welcome, you can email us at data@ed.gov.

Subscribe to the Data Matters Blog at https://www.ed.gov/subscriptions , the NCEE blog at https://ies.ed.gov/blogs/ncee/, and follow OCDO on LinkedIn.

Yours Truly in Data and Evaluation,

Greg and Matt

P.S. Happy International Open Data Day Eve!