IES Blog

Institute of Education Sciences

Is believing in yourself enough? Growth mindset and social belonging interventions for postsecondary students

The WWC recently reviewed the strength of evidence for two types of interventions designed to help students succeed in college: one report focuses on growth mindset interventions and another on social belonging. The WWC found that (1) neither type of intervention had a discernible effect on full-time college enrollment, (2) social belonging interventions had mixed effects on progressing in college and academic achievement, and (3) growth mindset interventions had potentially positive effects on academic achievement. We asked Greg Walton, an Associate Professor at Stanford University, IES-funded researcher, and expert on these kinds of interventions, to discuss what college faculty, administrators, and students should make of these findings.  

Can you walk through how growth mindset interventions and social belonging interventions with postsecondary students work? Were the interventions reviewed by the WWC typical interventions in this space?

Growth mindset interventions focus on the underlying “implicit” beliefs students have about the nature of intelligence: Is intelligence fixed or can it grow? These beliefs inform how students make sense of everyday academic challenges in school. If you think that intelligence is fixed, that you either have it or you don’t, then a setback like a poor grade can seem to be evidence that you don’t have what it takes. That can make students avoid academic challenges, withdraw, and ultimately learn and achieve less. Growth mindset interventions offer students the view that intelligence can grow with effort, hard work, good strategies, and help from others. The theory is that that mindset can help students see setbacks simply as evidence that they haven’t learned the material yet, or that their strategies haven’t been successful yet, and thus to sustain their efforts. These interventions typically start by sharing information from neuroscience about how the brain grows “like a muscle” during learning, especially when students work on challenging material. Then students might read stories from older students who used a growth mindset to persist through challenges. Finally, they may be asked to describe this idea to help younger students struggling in school, a technique termed “saying-is-believing.” That makes the experience active rather than passive and positions students as benefactors rather than beneficiaries, which would be potentially stigmatizing.

Social-belonging interventions target “belonging uncertainty,” a persistent doubt students can feel about whether “people like me” can belong in a school setting. This doubt arises most strongly for people who belong to groups that have historically faced exclusion in school settings, negative stereotypes that pose them as less able and less deserving of educational opportunities, or who are underrepresented in a school context. When students experience this doubt, everyday challenges such as feeling lonely, being excluded, or getting critical feedback can seem like evidence that they don’t belong in general. Social-belonging interventions share stories from older students who describe how they worried at first about whether they belonged in a new school and how these worries dissipated with time as they developed friendships and study partners, joined student groups, and formed mentor relationships. Belonging interventions offer students the view that it’s normal to worry about belonging at first in a new school but this gets better with time. Like growth mindset interventions, belonging interventions use written exercises to give students the opportunity to reflect on the intervention message and advocate for it to younger students. The theory is that this message can help students sustain a sense of belonging and stay engaged in school even when they face challenges, and that that helps students develop friendships and mentor relationships that support higher rates of achievement.

Social-belonging interventions were designed specifically to address circumstances in which people face underrepresentation or negative stereotypes in school. Even if all students have reasons to worry whether they belong in school, only some students have reason to question whether “people like me” belong. I am a White person whose parents both graduated from college. So, when I went to college, I felt homesick but I didn’t wonder whether “people like me” could belong.

That said, belonging concerns are felt by almost everyone, and in some cases belonging interventions have produced main effects (benefits for all students) rather than interactions predicated on group identity (e.g., Borman et al., 2019 for evidence from students in grade 6). However, most trials find greater benefits for students who face underrepresentation or negative stereotypes in specific settings. One study found that women in more gender-diverse engineering majors (averaging 33% women) showed no achievement gap with men in the first year and no benefit from a belonging intervention. But women in male-dominated majors (averaging 10% women) showed a large achievement gap in first year performance, but that gap was closed by the intervention (Walton et al., 2015; see also Binning et al., 2020) [Editor’s note: These two latter studies did not meet WWC standards for internal validity. Although this suggests caution in drawing conclusions from the studies, failing to meet WWC standards does not imply that an intervention is ineffective.]

Taken together, a fixed-mindset of intelligence and belonging uncertainty can be like a toxic tornado for students, swirling into each other and creating cascading self-doubt. I’m describing these interventions separately because they grew up independently in the literature, and the WWC’s two reports look at each separately. But for students, they are often experienced together.

It’s also important to state that, although the interventions reviewed by the WWC are typical of those conducted with postsecondary students, these are highly active areas with new trials reported regularly. Studies have explored new populations and college contexts (e.g., Murphy et al., 2020) and are increasingly focused on identifying boundary conditions that determine where we should and should not predict effects (see Bryan, Tipton, & Yeager, 2020). It is also noteworthy how few studies have examined the critical question of progress in college (3 in each report). We need much more research here, exploring effectiveness, implementation strategies, and boundary conditions. Further, research is increasingly complementing direct-to-student interventions by exploring how we can support practices in school that support growth mindset and belonging (Murphy et al., 2021). For example, recent research shows that highlighting pro-diversity peer norms—namely that most students endorse diversity—can facilitate more inclusive treatment among college students and, in turn, reduce achievement gaps between privileged and marginalized students (Murrar et al., 2020).

What are the key components that are needed for a social belonging or growth mindset intervention to have a good chance of working? What elements need to be in place to help students improve academically or to stay enrolled in college?

I would distinguish two layers of this question.

One layer is what does it take for a discrete exercise focused on belonging or growth-mindset—such as the focus of the trials reviewed by WWC—to help students. In general, we should consider what, how, when, and where.

What is it you want to offer students? It should give students an authentic and adaptive way to make sense of common challenges they face, a way of thinking they can use to achieve their goals in college. Simple exhortations such as, “I know you can do it” or “You belong!” do not effectively impart a growth mindset or a sense of belonging, as Carol Dweck and I have written. Instead, it is useful to use high-quality materials developed and validated in research. Examples of materials available online are here and here.

How will you convey this? The goal of these interventions is to address foundational beliefs students have about school, such as “Can I do it?” and “Can people like me belong here?” It’s not to do something else, like to build a skill. That means the experience need not take long—typically, interventions last 30-60 minutes—but it should be immersive and interactive. You want students to deeply reflect on the ideas you present and connect these ideas to their lived experience.

That said, the more you can implement approaches that are scalable within an institutional context the more students you can potentially help. That’s one reason recent trials that reach large samples have focused on online modules (e.g., LaCosse et al., 2020; Yeager, Walton, & Brady et al., 2016). Students can log-on individually and complete materials at near-zero marginal cost. However, these approaches also have challenges, as online modules may not be as engrossing as in-person experiences. As we have moved from delivering these interventions in one-on-one, in-person experiences to larger studies with materials delivered online, we have found that students spend less time on the same materials and write less in response to prompts. Another alternative is having students meet in-person in groups to participate in these interventions or discuss their content (see Binning et al., 2020; Murphy et al., 2020), but that may be more difficult to implement on a large scale. So, there can be trade-offs between reaching scale and creating deep and impactful experiences.

When should you do this? In general, it is valuable if an intervention happens earlier rather than later, so it can alter trajectories going forward. However, it may be optimal to deliver interventions soon after students have encountered some challenges, but before they have taken steps in response to those challenges that are hard to reverse (e.g., dropping out). In general, social-psychological interventions are more sensitive to timing than to dosage. Growth mindset and belonging interventions have been delivered from the summer before college (Yeager, Walton, Brady, et al., 2016), to the first academic term (Walton et al., 2015), to the second (Walton & Cohen, 2011).

Where should you deliver interventions? This brings us to the second layer. So far, I’ve addressed the first layer, where you are focused on a discrete experience or set of experiences. But the second layer is that, growth mindset and belonging interventions will be most effective in contexts in which (1) the message offered is legitimate and authentic (locally true) and (2) students have real opportunities to get academic support and to develop a sense of belonging. In the end, to produce the most robust change, we must create cultures in schools in which adaptive ideas about ability and belonging are normal and reinforced. There are many ways that institutions signal to students, even inadvertently, messages about the nature of intelligence and who belongs. In welcoming a new class to campus, do we extol the past achievements of a few, which may only heighten imposter syndrome among everyone else? Can we instead talk about what students can do in the future and who they can become? In welcoming students to class, do faculty communicate that they expect to weed out large numbers of students? Or do they design assignments and evaluations to support students’ learning and growth (Canning et al., 2019)? Another question involves how well colleges foster opportunities for students to develop in-group pride and identity. Tiffany Brannon at UCLA finds that African American students do better in college when they have more opportunities to participate in events that celebrate and explore Black culture (Brannon & Lin, 2021). Some resources to help researchers and practitioners create cultures of growth and belonging for all students are available at the Student Experience Project, co-led by the College Transition Collaborative (https://collegetransitioncollaborative.org/student-experience/).

Recently, you and your colleagues have distinguished between people with different characteristics - and environments with different characteristics. You’ve argued that researchers should be looking more closely at the contexts, or what you’ve called “psychological affordances” in which these interventions might have different effects. Why is this work important? Why should educators be paying attention?

Social-psychological interventions operate within complex systems. Those systems invariably determine the specific effect any intervention has. To understand this, my colleagues and I have found it useful to consider the affordances of a school context: What does a context make possible (Walton & Yeager, 2020)? For instance, no psychological intervention will help English-language speakers learn Chinese if they aren’t receiving instruction in Chinese.

We distinguish two kinds of affordances. One is structural: What is it that different institutions make possible for students to do? As an example, in a forthcoming study, Shannon Brady, Parker Goyer, David Yeager, and I tracked college outcomes of students randomly assigned to a social belonging intervention or a control condition at the end of high school. The intervention raised the rate of bachelor’s degree completion for students who first enrolled in more selective 4-year institutions from 26% to 43%. These are institutions that tend to have higher retention and graduation rates and tend to spend more per student on instruction and student services than less selective 4-year institutions. They thus afford higher 4-year completion rates. At the same time, the same belonging intervention had no effect on bachelor’s degree completion rates for students who first enrolled in less selective 4-year institutions.

The second kind of affordance is psychological: What is it that students can believe in a school context? Does the cultural context in which an intervention is delivered one in which the way of thinking offered by the intervention can take hold and thrive? Or is it one that makes that way of thinking illegitimate, inauthentic, or not useful?  A large-scale social-belonging intervention delivered online to students in 21 diverse colleges and universities increased first-year full-time completion rates for students from historically underperforming groups, but only in colleges that afforded, or fostered, a sense of belonging to members of those groups. Let’s break this down: In some college contexts, students from historically underperforming groups (who were not exposed to the intervention) realized a high sense of belonging by the end of the first year. Here the belonging message was “locally true” (true here, for people like me). Although we don’t know exactly why this was the case, presumably in these schools students from the given group had more opportunities to develop friendships, to join student groups, and to form meaningful relationships with instructors. In other colleges, students did not attain this high sense of belonging by the end of the first year. Only in the first case did the belonging intervention raise first-year completion rates (Walton, Murphy et al., in prep; described in Walton & Yeager, 2020).

In both cases, the belonging intervention helped students take advantage of opportunities available to them, whether to graduate or to belong. An important implication is that it may be necessary to address both students’ beliefs and whether contexts support more positive beliefs. That’s helpful, because it gives us a precise way to think about how to make contexts more supportive: To what extent do they make adaptive beliefs about intelligence and belonging legitimate and authentic and, if they do not, what can we do about this?

It sounds like you’re saying postsecondary leaders who want to foster greater student success and reduce gaps in retention and academic performance may want to consider these kinds of interventions, in part because they are relatively inexpensive to deliver to large numbers of students. But they should also consider how hospitable their campus is to students who might initially struggle in college.

For example, to reinforce a growth mindset, universities need to make academic support resources truly accessible; to reinforce a sense of belonging, universities might look for multiple ways to communicate that successful students of all kinds of backgrounds have initially experienced self-doubt, and that feeling like you don’t belong is a fairly normal and temporary part of adjusting to college.

That’s right. Growth mindset and belonging are about both student beliefs or ways of thinking and institutional practices—either alone may not be enough. So, to support a growth mindset, institutions should both (1) convey that all students can learn and grow with effort, good strategies, and support from others and (2) back that up by creating learning environments designed to support growth, including adequate academic supports, and classes that focus on fostering growth rather than identifying who is allegedly smart and who is not. To support belonging, institutions should (1) acknowledge that nearly all new college students worry at first about whether they belong, that this is normal and improves with time and (2) create classroom and out-of-classroom environments in which all of the diverse students we serve can develop strong friendships and mentoring relationships and find communities in which they belong.

Thanks very much, Greg.

 

Read the WWC’s summary of evidence for these interventions in the Growth Mindset Intervention Report and the Social Belonging Intervention Report. Find related resources at the The College Transition Collaborative (https://collegetransitioncollaborative.org/) or the Project for Education Research That Scales (https://www.perts.net/)

 

Carter Epstein, Senior Associate at Abt Associates, produced this blog with Greg Walton, Associate Professor of Psychology at Stanford University.

 

Note: The discussion above reflects the opinions of Greg Walton and does not necessarily reflect the opinions of the Institute of Education Sciences or the What Works Clearinghouse. Some of the studies cited above have not been reviewed by the What Works Clearinghouse.

 

REFERENCES

Binning, K.R., Kaufmann, N., McGreevy, E.M., Fotuhi, O., Chen, S., Marshman, E., Kalender, Z.Y., Limeri, L., Betancur, L., & Singh, C. (2020). Changing social contexts to foster equity in college science courses: An ecological-belonging intervention. Psychological Science, 31,1059-1070. https://doi.org/10.1177/0956797620929984

Borman, G.D., Rozek, C.S., Pyne, J., & Hanselman, P. (2019). Reappraising academic and social adversity improves middle school students’ academic achievement, behavior, and well-being. Proceedings of the National Academy of Sciences 116 (33), 16286-16291. https://doi.org/10.1073/pnas.1820317116

Brady, S. T., Walton, G. M., Goyer, J. P., & Yeager, D. S. (in prep). [Where does a brief belonging intervention increase the attainment of a college degree? The role of institutional affordances.] Manuscript in preparation.

Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature human behaviour, 5(8), 980–989. https://doi.org/10.1038/s41562-021-01143-3

 Bryk, A. S., Grunow, A., Gomez, L. M., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better.  Harvard Education Press.

Canning, E. A., Muenks, K. ,Green, D.J., & Murphy, M.C. (2019). STEM faculty who believe ability is fixed have larger racial achievement gaps and inspire less student motivation in their classes. Science Advances, 5(2). https://www.science.org/doi/10.1126/sciadv.aau4734 

Dweck, C. (2016, January 11). Recognizing and overcoming false growth mindset. Edutopia. https://www.edutopia.org/blog/recognizing-overcoming-false-growth-mindset-carol-dweck

Murphy, M.C., Fryberg, S.A., Brady, L.M, Canning, E.A., & Hecht, C.A. ( 2021, August 25). Global Mindset Initiative Paper 1: Growth mindset cultures and teacher practices. https://ssrn.com/abstract=3911594

Murrar, S., Campbell, M.R. & Brauer, M. (2020). Exposure to peers’ pro-diversity attitudes increases inclusion and reduces the achievement gap. Nature Human Behavior 4, 889–897 . https://doi.org/10.1038/s41562-020-0899-5

Walton, G.M. (2021, November 9). Stop telling students, “You belong!” Three ways to make a sense of belonging real and valuable. Education Week. https://www.edweek.org/leadership/opinion-stop-telling-students-you-belong/2021/11

Walton, G. M., Logel, C., Peach, J. M., Spencer, S. J., & Zanna, M. P. (2015). Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering. Journal of Educational Psychology, 107(2), 468–485. https://eric.ed.gov/?id=EJ1061905

Walton, G. M., Murphy, M. C., Logel, C., Yeager, D. S., Goyer, J. P., Brady, S. T., . . . Krol, N. (in preparation). Where and with whom does a brief social-belonging intervention raise college achievement? Manuscript in preparation.

Walton, G. M. & Yeager, D. S. (2020). Seed and soil: Psychological affordances in contexts help to explain where wise interventions succeed or fail. Current Directions in Psychological Science, 29, 219-226. http://gregorywalton-stanford.weebly.com/uploads/4/9/4/4/49448111/waltonyeager_2020.pdf

Yeager, D. S., Walton, G. M., Brady, S. T., Akcinar, E. N., Paunesku, D., Keane, L., Kamentz, D., Ritter, G., Duckworth, A. L., Urstein, R., Gomez, E. M., Markus, H. R., Cohen, G. L., & Dweck, C. S. (2016). Teaching a lay theory before college narrows achievement gaps at scale. Proceedings of the National Academy of Sciences of the United States of America, 113(24), E3341-E3348. https://doi.org/10.1073/pnas.1524360113

 

 

Exploring the Growing Impact of Career Pathways

Career pathways programs for workforce development are spreading across the country at both the secondary and postsecondary levels. Based on a synthesis of studies examining career pathways programs that integrate postsecondary career-technical education (CTE), the What Works Clearinghouse (WWC)’s Designing and Delivering Career Pathways at Community Colleges practice guide presents five recommendations for implementing evidence-based practices:

Cover of advising practice guide
  1. Intentionally design and structure career pathways to enable students to further their education, secure a job, and advance in employment.
  2. Deliver contextualized or integrated basic skills instruction to accelerate students’ entry into and successful completion of career pathways.
  3. Offer flexible instructional delivery schedules and models to improve credit accumulation and completion of non-degree credentials along career pathways.
  4. Provide coordinated comprehensive student supports to improve credit accumulation and completion of non-degree credentials along career pathways.
  5. Develop and continuously leverage partnerships to prepare students and advance their labor market success.

Led by the WWC’s postsecondary contractor, Abt Associates, this practice guide was created by an expert panel of researchers and practitioners to provide examples of career pathways strategies and components and guidance to implement them; advise on strategies to overcome potential obstacles; and summarize evidence associated with rigorous research studies that met WWC standards.

As a long-time researcher of postsecondary CTE and many other important aspects of community college education, I welcome the opportunity to reflect on these five recommendations. I hope that my blog will help readers understand how this new practice guide fits into a larger landscape of research focusing on programs, policies, and practices aligned with the career pathways framework. Far from new, the notion of career pathways goes back several decades; thus, it is not surprising that we see an evolution in research to measure students’ education and employment outcomes. And still, there is a need for more rigorous studies of career pathways.

The Abt team located about 16,000 studies that were potentially relevant to the practice guide. Those studies used a wide variety of methods, data (quantitative and qualitative), and analysis procedures. Only 61 of them were eligible for review against the WWC standards, however; and only 21 of those met the WWC standards. Interestingly, most of those 21 studies focused on non-degree postsecondary credentials, rather than on college degrees, with policies and programs associated with workforce development and adult education well represented. Thus, lessons from the practice guide speak more directly to career pathways programs that culminate in credentials below the associate degree level than about those programs leading to the associate or baccalaureate degree level.

This dearth of rigorous career pathways research is problematic, as educational institutions of all types, including community colleges, seek to deliver positive, equitable outcomes to students during and beyond the COVID-19 pandemic.

Focus on Career Pathways

After examining the evidence from the studies that met the WWC standards, it was clear that the evidence converged around career pathways programs following requirements in the Strengthening Career and Technical Education for the 21st Century Act and Workforce Innovation and Opportunity Act (WIOA). In alignment with the WIOA definition of career pathways, the set of studies in the practice guide examine a “combination of rigorous and high-quality education, training, and other services” that align with the skill needs of industries in the region or state and accelerate participants’ educational and career advancement, to the extent practicable.

As defined by WIOA, career pathways support learners in pursuing their education and career goals, lead to at least one postsecondary credential, and provide entry or advancement in a particular occupation or occupational cluster. Because a growing number of community colleges employ a career pathways approach, as advocated by the federal legislation, it made sense to focus the practice guide on rigorous results and evidence-based recommendations that may help to move career pathway design and implementation forward.

The Five Recommendations

Recommendation 1: Intentionally design and structure career pathways to enable students to further their education, secure a job, and advance in employment. Our panel advocated for the intentional design and structure of career pathways for good reason. Whereas all educational institutions enroll students in courses and programs, career pathways prioritize the student’s entire educational experience, from access and entry, to completion and credentialing, and on to employment and career advancement. This purposeful approach to supporting student attainment is theorized to lead to positive student outcomes.

Applying the meta-analysis process required by the WWC, we determined from the 21 studies whether career pathways were achieving this crucial goal. We found nine of the studies showed overall statistically significant, positive results on industry-recognized credential attainment. Of the 12 studies supporting this recommendation, most  measured non-degree credentials; only two measured degree attainment—an important point to recognize, because these are the studies that have been conducted thus far.

This very small number of rigorous studies measuring degree attainment leaves open the question of whether career pathways increase postsecondary degree attainment—specifically the predominant credential in the community college context, the associate degree—and calls for greater investment in research on student completion of associate degrees (as well as baccalaureate degrees, a growing phenomenon in the United States).

Recommendation 2. Deliver contextualized or integrated basic skills instruction to accelerate students’ entry into and successful completion of career pathways. Studies that met WWC standards showed a positive impact of career pathways on college credit accumulation and industry-recognized credential attainment. Only one study measured postsecondary degree attainment relative to contextualized and basic skills instruction and it reported statistically significant and negative results. However, descriptive and correlational studies suggest that contextualized and basic skills instruction contribute to positive educational outcomes for students enrolled in Adult Basic Education in addition to postsecondary CTE and workforce training.

That results of rigorous research complement descriptive studies, some of which provide rich details on program implementation, is information useful for scaling up community college career pathways. Having said this, we still need to know more about how contextualized, basic skills instruction—and other applied instructional interventions—affect the outcomes of students, especially those from racial minoritized groups, with low incomes, and who are the first generation to attend college, all purported to be well served by the career pathways approach.

Recommendation 3: Offer flexible instructional delivery schedules and models to improve credit accumulation and completion of non-degree credentials along career pathways. Studies supporting this recommendation focused on five education outcomes: industry-recognized credential attainment, academic performance, technical skill proficiency, credit accumulation, and postsecondary degree attainment. As seen with the previous two recommendations, results on industry-recognized credential attainment were statistically significant and positive. Results on academic performance, technical skill proficiency, and credit accumulation were indeterminate, meaning findings could be positive or negative but were not statistically significant.

What is important to reiterate here is that nearly all the studies that met the WWC standards focused on non-degree credentials, providing limited information about results on the education outcome of postsecondary degree attainment. To be clear, our panel is not saying career pathways should focus exclusively on non-degree credentials; rather that results on postsecondary degree attainment are not definitive. Even so, that findings linking flexible scheduling and non-degree credential attainment are positive is important to know now, when the country is dealing with the pandemic.

Community colleges nationwide are rethinking instructional delivery to better meet students’ dire health, family, and employment needs. Rigorous research on career pathways interventions, such as flexible delivery, is needed, particularly studies involving diverse student populations. In times of economic and social struggle, it is essential that community college career pathways produce the equitable outcomes they purport to provide.

Recommendation 4: Provide coordinated comprehensive student supports to improve credit accumulation and completion of non-degree credentials along career pathways. The rigorous studies meeting WWC standards and measuring outcomes relative to comprehensive student supports focused on the education outcome domain only. Similar to the previous recommendation on flexible scheduling, findings on industry-recognized credential attainment were statistically significant and positive. However, on supports, findings on credit accumulation were statistically significant and positive, reinforcing findings generated by other studies showing holistic supports improve student outcomes. For example, a meta-analysis of studies of the Trade Adjustment Assistance Community College and Career Training grants that used rigorous evaluation designs reported favorable results for holistic supports in counseling and advising, case management, and various other support services and educational outcomes.

Consistent with the recommendations in this practice guide, a growing body of evidence favors integrating comprehensive student supports with career pathways. These supports are intended to meet the needs of the diverse population of students who attend community colleges; so, they should demonstrate equitable results on educational outcomes. More rigorous research is needed to measure whether and how career pathways provide access, opportunity, and outcomes for racially minoritized, low-income, and other underserved student groups. These studies should ascertain the impact of student supports on both education and employment outcomes, recognizing that students seek a high-quality credential and a good job that offers economic security and career mobility.

Recommendation 5: Develop and continuously leverage partnerships to prepare students and advance their labor market success. This recommendation specifically emphasizes labor market success, based on studies that examine labor market outcomes only. Supporting this recommendation were findings from studies of four labor market outcomes: short-term employment, short-term earnings, medium-term employment, and medium-term earnings. (The studies did not include long-term findings.)

Overall, statistically significant and positive outcomes were found in the meta-analysis for short-term employment, short-term earnings, and medium-term earnings. However, for medium-term employment, the meta-analysis results were indeterminate. To clarify, this does not mean employment-focused partnerships do not lead to labor market success; instead it points to a dearth of research that tracks students through training and into employment for long enough to measure long-term outcomes.

Even so, these initial findings from the meta-analysis are promising and suggest that developing and leveraging such partnerships may help move the needle on short- and medium-term employment outcomes. Longitudinal research that tracks students for periods sufficient to know whether long-term employment and earnings are affected should be a priority in the future.

Moving Forward

As I reflect on the research that I have conducted on career pathways over the years, I am gratified to see mounting evidence of positive student outcomes. As a first-generation college student myself, it has always made sense to me to demystify the college education process. Helping learners understand the entire educational journey, from start to finish, is bound to help them see how what they are learning may contribute to future education and career choices. I went to college not knowing what it would be like or whether I would be able to succeed, and I benefited from faculty and advisors who helped me see how my future could progress.

For other students like me who enter college without the benefit of family members sharing their stories of college-going, and for those who have to balance school with work and family care-taking responsibilities, it is important to know how a college education, including postsecondary CTE, can lead to positive educational and employment outcomes. Student groups underserved by postsecondary education deserve our most resolute and far-reaching efforts.

To this end, additional rigorous evidence on the impact of postsecondary CTE on college degree attainment could help to inform career pathways design, funding, and implementation. Also, as I reflected on the five recommendations, I was struck by the modest amount of research on medium-term labor market outcomes and the lack of any studies of long-term labor market outcomes. When the focus of career pathways is creating a path to living-wage employment and career advancement over the long term, it isn’t enough to know that students’ immediate employment outcomes were improved. When many students attending community colleges are already working, it isn’t even clear what immediate employment means.

If the outcome of interest for the majority of community college students who are adults and working is whether they get a better job and higher salary than they were getting pre-education, more nuanced measures and longer follow-up periods are needed than those provided by any of the research reviewed for this practice guide. It seems to me that finding more evidence of medium- and long-term outcomes could also provide more useful evidence of how career pathways work for diverse learner groups who are under-studied at the present time.

I was honored to help develop the practice guide with Hope Cotner, Grant Goold, Eric Heiser, Darlene Miller, and Michelle Van Noy. What an enormously gratifying experience it was to work with these professionals, the WWC team at Abt, and the Institute of Education Sciences staff. Working on this practice guide has left me feeling more optimistic about what we could learn with a more sizeable federal investment in research on postsecondary CTE in general, and on career pathways specifically. Rigorous evidence is needed to test models, explore interventions, and understand results for the plethora of learner groups who attend community colleges.

As the nation struggles to pull out of the pandemic that continues to rage in pockets across the country, it is the right time to invest in research that helps prepare students for good jobs that advance living-wage careers over a lifetime. A true commitment to equity in CTE programming is necessary for the nation, and now is the time to invest.

_____________________________________________________________________________________________________________

Debra D. Bragg, PhD, is president of Bragg & Associates, Inc., and the founder of research centers focusing on community college education at the University of Illinois at Urbana-Champaign and the University of Washington. She spent the first 15 years of her career in academe studying postsecondary CTE for federally funded research centers, having devoted her entire research agenda to improving education- and employment-focused policies, programs, and practices to create more equitable outcomes for community college students. She served as an expert panelist for the What Works Clearinghouse (WWC)’s Designing and Delivering Career Pathways at Community Colleges practice guide.

 

 

Teachers Should Not Be Left Wondering What Works

The past two school years have posed many new and unexpected challenges for students and teachers. One thing that has not changed much is that educators continue to need quick access to evidence on strategies that can best support students. The What Works Clearinghouse (WWC), an initiative of the U.S. Department of Education’s Institute of Education Sciences, aims to meet these needs with ready-to-use practices supported by evidence. The WWC Practice Guides describe these practices and how to implement them, most recently in the new guide for assisting students struggling in mathematics. These Practice Guides contain the classroom strategies and tips that are most likely to help improve student outcomes.

More than two dozen free Practice Guides address challenges educators face in teaching math, reading, and writing; supporting positive student behavior; and preventing dropout. The recommendations in Practice Guides are based on evidence from well-designed and well-implemented studies, the experiences of practitioners, and the expert opinions of a panel of nationally recognized experts.

Ann Jolly, an instructional program manager at the Charlotte-Mecklenburg Schools’ Program for Exceptional Children, has used WWC Practice Guides for years. She describes her experiences using the WWC resources below. Her experiences may help teachers or instructional leaders understand how to better incorporate evidence-based practices into their own practice.


The COVID-19 pandemic has us all wondering where the time goes. We want to use the most promising evidence-based practices to support our students. However, as expressed by one teacher who understands how easy it is to forget about trying out something new in the face of day-to-day demands, “Yeah, you just get busy teaching…

Whether you are a new teacher trying to figure out how to balance teaching, lesson planning, grading, and other duties, or a veteran who is “busy teaching,” you should check out the WWC. The WWC, created by the U.S. Department of Education, is an easy-to-navigate website with valuable resources. I know that, as teachers, we are constantly seeking out resources that will enable us to provide the best instruction to our students. The WWC can help by searching for research, reviewing studies for quality, and summarizing findings, so that busy teachers like us can focus on our students! Here’s a quick look at some of the WWC resources I have used to make a difference in my school and district as an instructional leader collaborating with teachers and families.

When I needed help boosting reading comprehension among my special education students, I used the WWC Practice Guide Improving Reading Comprehension in Kindergarten Through 3rd Grade. This guide provided me with recommendations of practices and other relevant information that the WWC gathered to support classroom instruction. For example, I was able to quickly see that teaching students how to use reading comprehension strategies had the strongest evidence, so I knew to focus on that. The guide gave me easy-to-understand resources about how to bring the strategies into my classroom, plus videos and reference tools with examples. These were easy to digest and I was able to immediately implement the recommendations in my classroom.

When I needed strategies to support literacy at home and in school, I used the WWC Practice Guide Foundational Skills to Support Reading for Understanding in Kindergarten Through 3rd Grade and its supplemental resources. Not only does the guide include a wealth of information for teachers, but companion documents include a summary of recommendations, a Professional Learning Communities Facilitator’s Guide, and Tips for Supporting Reading Skills at Home. I used the last tool to develop a presentation for parents. Parents took notes and asked questions as they made connections between the guide and the practices they could use at home with their children. Finding opportunities like this one to build relationships between teachers and parents may be even more important now, during a pandemic, than it was when I held this workshop. 

When my school was looking for strategies to improve student behavior, I facilitated a book club with school staff using the WWC Practice Guide Reducing Behavior Problems in the Elementary School Classroom. I began the club after noticing that other teachers were coming to me for suggestions about a common pattern of behaviors interfering with student learning.  This WWC guide offered several strategies to share. Although we started by discussing a specific behavioral issue and a recommended practice to address it, we eventually worked through the whole guide, chapter by chapter. The WWC Practice Guide gave us a free resource with powerful evidence-based strategies and practices for us to try. Teachers across grade levels and content areas actively collaborated through the book club and were able to build a common language and understanding about schoolwide practices. One of the great embedded features in WWC Practice Guides are the “Obstacles” or “Roadblocks.” This feature acknowledges perceived and actual barriers to implementing evidence-based practices and suggests solutions to overcome them!

The WWC has created a wide range of other Practice Guides, covering students from early childhood through high school graduation (and beyond). The most recent products include Assisting Students Struggling with Mathematics: Intervention in the Elementary Grades, a Practice Guide for educators in grades K to 6 that provides ready-to-use strategies for assisting struggling students. Some of my colleagues have used the guides on Teaching Secondary Students to Write Effectively, Teaching Math to Young Children, and Using Student Achievement Data to Support Instructional Decision Making. So many more Practice Guides are available!

I also encourage you to sign up now for the WWC News Flash and add the WWC to your social media network on Twitter, Facebook, or YouTube to easily keep up with the most current information. Research evidence on “what works” in education is there just for you. When you have a question, rely on the WWC…and don’t be left wondering what works!

This blog was written by Ann C. Jolly, Instructional Program Manager, Programs for Exceptional Children at Charlotte-Mecklenburg Schools with Data Rotz, Mathematica. 

“The How” of “What Works:” The Importance of Core Components in Education Research

Twenty-some odd years ago as a college junior, I screamed in horror watching a friend open a running dishwasher. She wanted to slip in a lightly used fork. I jumped to stop her, yelling “don’t open it, can’t you tell it’s full of water?” She paused briefly, turning to look at me with a “have you lost your mind” grimace, and yanked open the door.

Much to my surprise, nothing happened. A puff of steam. An errant drip, perhaps? But no cascade of soapy water. She slid the fork into the basket, closed the door, and hit a button. The machine started back up with a gurgle, and the kitchen floor was none the wetter.

Until that point in my life, I had no idea how a dishwasher worked. I had been around a dishwasher, but the house I lived in growing up didn’t have one. To me, washing the dishes meant filling the sink with soapy water, something akin to a washer in a laundry. I assumed dishwashers worked on the same principle, using gallons of water to slosh the dishes clean. Who knew?

Lest you think me completely inept, a counterpoint. My first car was a 1979 Ford Mustang. And I quickly learned how that very used car worked when the Mustang’s automatic choke conked out. As it happens, although a choke is necessary to start and run a gasoline engine, that it be “automatic” is not. My father Rube Goldberg-ed up a manual choke in about 15 minutes rather than paying to have it fixed.

My 14-year-old self learned how to tweak that choke “just so” so that I could get to school each morning. First, pull the choke all the way out to start the car, adjusting the fuel-air mixture ever so slightly. Then gingerly slide it back in, micron by micron, as the car warms up and you hit the road. A car doesn’t actually run on liquid gasoline, you see. Cars run on fuel vapor. And before the advent of fuel injection, fuel vapor was courtesy your carburetor and its choke. Not a soul alive who didn’t know how a manual choke worked could have started that car.

You would be forgiven if, by now, you were wondering where I am going with all of this and how it relates to the evaluation of education interventions. To that end, I offer three thoughts for your consideration:

  1. Knowing that something works is different from knowing how something works.

 

  1. Knowing how something works is necessary to put that something to its best use.

 

  1. Most education research ignores the how of interventions, dramatically diminishing the usefulness of research to practitioners.

My first argument—that there is a distinction between knowing what works and how something works—is straightforward. Since it began, the What Works Clearinghouse™ has focused on identifying “what works” for educators and other stakeholders, mounting a full-court press on behalf of internal validity. Taken together, Version 4.1 of the WWC Standards and Procedures Handbooks total some 192 pages. As a result, we have substantially greater confidence today than we did a decade ago that when an intervention developer or researcher reports that something worked for a particular group of students, we know that it actually did.

In contrast, WWC standards do not, and as far as I can tell have not ever, addressed the how of an intervention. By “the how” of an intervention, I’m referring to the parts of it that must be working, sometimes “just so,” if its efficacy claims are to be realized. For a dishwasher, it is something like: “a motor turns a wash arm, which sprays dishes with soapy water.” (It is not, as I had thought, “the dishwasher fills with soapy water that washes the mac and cheese down the drain.”) In the case of my Mustang, it was: “the choke controls the amount of air that mixes with fuel from the throttle, before heading to the cylinders.”

If you have been following the evolution of IES’ Standards for Excellence in Education Research, or SEER, and its principles, you recognize “the how” as core components. Most interventions consist of multiple core components that are—and perhaps must—be arrayed in a certain manner if the whole of the thing is to “work.” Depicted visually, core components and their relationships to one another and to the outcomes they are meant to affect form something between a logic model (often too simplistic) and a theory of change (often too complex).

(A word of caution: knowing how somethings works is also different from knowing why something works. I have been known to ask at work about “what’s in the arrows” that connect various boxes in a logic model. The why lives in those arrows. In the social sciences, those arrows are where theory resides.)  

My second argument is that knowing how something works matters, at least if you want to use it as effectively as possible. This isn’t quite as axiomatic as the distinction between “it works” and “how it works,” I realize.

This morning, when starting my car, I didn’t have to think about the complex series of events leading up to me pulling out of the driveway. Key turn, foot down, car go. But when the key turns and the car doesn’t go, then knowing something about how the parts of a car are meant to work together is very, very helpful. Conveniently, most things in our lives, if they work at all, simply do.  

Inconveniently, we don’t have that same confidence when it comes to things in education. There are currently 10,677 individual studies in the What Works Clearinghouse (WWC) database. Of those, only about 11 percent meet the WWC’s internal validity standards. Among them, only 445 have at least one statistically significant positive finding. Because the WWC doesn’t consider results from studies that don’t have strong internal validity, it isn’t quite as simple as saying “only about 4 percent of things work in education.” Instead, we’re left with “89 percent of things aren’t tested rigorously enough to have confidence about whether they work, and when tested rigorously, only about 38 percent do.” Between the “file drawer” problem that plagues research generally and our own review of the results from IES efficacy trials, we have reason to believe the true efficacy rate of “what works” in education is much lower.

Many things cause an intervention to fail. Some interventions are simply wrong-headed. Some interventions do work, but for only some students. And other interventions would work, if only they were implemented well.

Knowing an intervention’s core components and the relationships among them would, I submit, be helpful in at least that third case. If you don’t know that a dishwasher’s wash arm spins, the large skillet on the bottom rack with its handle jutting to the sky might not strike you as the proximate cause of dirty glasses on the top rack. If you don’t know that a core component of multi-tiered systems of support is progress monitoring, you might not connect the dots between a decision to cut back on periodic student assessments and suboptimal student outcomes.

My third and final argument, that most education research ignores the how of interventions, is based in at least some empiricism. The argument itself is a bit of a journey. One that starts with a caveat, wends its way to dismay, and ends in disappointment.

Here’s the caveat: My take on the relative lack of how in most education research comes from my recent experience trying to surface “what works” in remote learning. This specific segment of education research may well be an outlier. But I somehow doubt it.

Why dismay? Well, as regular readers might recall, in late March I announced plans to support a rapid evidence synthesis on effective practices in remote learning. It seemed simple enough: crowd-source research relevant to the task, conduct WWC reviews of the highest-quality submissions, and then make those reviews available to meta-analysts and other researchers to surface generalizable principles that could be useful to educators and families.

My stated goal had been to release study reviews on June 1. That date has passed, and the focus of this post is not “New WWC Reviews of Remote Learning Released.” As such, you may have gathered something about my plan has gone awry. You would be right.

Simply, things are taking longer than hoped. It is not for lack of effort. Our teams identified more than 930 studies, screened more than 700 of those studies, and surfaced 250 randomized trials or quasi-experiments. We have prioritized 35 of this last group for review. (For those of you who are thinking some version of “wow, it seems like it might be a waste to not look at 96 percent of the studies that were originally located,” I have some thoughts about that. We’ll have to save that discussion, though, for another blog.)

Our best guess for when those reviews will be widely available is now August 15. Why things are taking as long as they are is, as they say, “complicated.” The June 1 date was unlikely from the start, dependent as it was upon a series of best-case situations in times that are anything but. And at least some of the delay is driven by our emphasis on rigor and steps we take to ensure the quality of our work, something we would not short-change in any event.  

Not giving in to my dismay, however, I dug in to the 930 studies in our remote learning database to see what I might be able to learn in the meantime. I found that 22 of those studies had already been reviewed by the WWC. “Good news,” I said to myself. “There are lessons to be learned among them, I’m sure.”

And indeed, there was a lesson to be learned—just not the one I was looking for. After reviewing the lot, there was virtually no actionable evidence to be found. That’s not entirely fair. One of the 22 records was a duplicate, two were not relevant, two were not locatable, and one was behind a paywall that even my federal government IP address couldn’t get behind. Because fifteen of the sixteen remaining studies reviewed name-brand products, there was one action I could take in most cases: buy the product the researcher had evaluated.

I went through each article, this time making an imperfect determination about whether the researcher described the intervention’s core components and, if so, arrayed them in a logic model. My codes for core components included one “yes,” two “bordering on yes,” six “yes-ish,” one “not really,” and six “no.” Not surprisingly, logic models were uncommon, with two studies earning a “yes” and two more tallied as “yes-ish.” (You can see now why I am not a qualitative researcher.)

In case there’s any doubt, herein lies my disappointment: if an educator had turned to one of these articles to eke out a tip or two about “what works” in remote learning, they would have been, on average, out of luck. If they did luck out and find an article that described the core components of the tested intervention, there was a vanishingly small chance there would be information on how to put those components together to form a whole. As for surfacing generalizable principles for educators and families across multiple studies? Not without some serious effort, I can assure you.

I have never been more convinced of the importance of core components being well-documented in education research than I am today. As they currently stand, the SEER principles for core components ask:

  • Did the researcher document the core components of an intervention, including its essential practices, structural elements, and the contexts in which it was implemented and tested?
  • Did the researcher offer a clear description of how the core components of an intervention are hypothesized to affect outcomes?
  • Did the researcher's analysis help us understand which components are most important in achieving impact?

More often than not, the singular answer to the questions above is a resounding “no.” That is to the detriment of consumers of research, no doubt. Educators, or even other researchers, cannot turn to the average journal article or research report and divine enough information about what was actually studied to draw lessons for classroom practice. (There are many reasons for this, of course. I welcome your thoughts on the matter.) More importantly, though, it is to the detriment of the supposed beneficiaries of research: our students. We must do better. If our work isn’t ultimately serving them, who is it serving, really?  

Matthew Soldner
Commissioner, National Center for Education Evaluation and Regional Assistance
Agency Evaluation Officer, U.S. Department of Education

An Evidence-Based Response to COVID-19: What We’re Learning

Several weeks ago, I announced the What Works Clearinghouse’s™ first ever rapid evidence synthesis project: a quick look at “what works” in distance education. I asked families and educators to send us their questions about how to adapt to learning at home, from early childhood to adult basic education. I posed a different challenge to researchers and technologists, asking them to nominate high-quality studies of distance and on-line learning that could begin to answer those questions.

Between public nominations and our own databases, we’ve now surfaced more than 900 studies. I was happy to see the full-text of about 300 studies were already available in ERIC, our own bibliographic database—and that many submitters whose work isn’t yet found there pledged to submit to ERIC, making sure it will be freely available to the public in the future. I was a little less happy to learn that only a few dozen of those 900 had already been reviewed by the WWC. This could mean either that (1) there is not a lot of rigorous research on distance learning, or (2) rigorous research exists, but we are systematically missing it. The truth is probably “both-and,” not “either-or.” Rigorous research exists, but more is needed … and the WWC needs to be more planful in capturing it.

The next step for the WWC team is to screen nominated studies to see which are likely to meet our evidence standards. As I’ve said elsewhere, we’ll be lucky if a small fraction—maybe 50—do. Full WWC reviews of the most actionable studies among them will be posted to the WWC website by June 1st, and at that time it is my hope that meta-analysts and technical assistance providers from across the country pitch in to create the products teachers and families desperately need. (Are you a researcher or content producer who wants to join that effort? If so, email me at matthew.soldner@ed.gov.)

Whether this approach actually works is an open question. Will it reduce the time it takes to create products that are both useful and used? All told, our time on the effort will amount to about two months. I had begun this process hoping for something even quicker. My early thinking was that IES would only put out a call for studies, leaving study reviews and product development to individual research teams. My team was convinced, however, that the value of a full WWC review for studies outweighed the potential benefit of quicker products. They were, of course, correct: IES’ comparative advantage stems from our commitment to quality and rigor.

I am willing to stipulate that these are unusual times: the WWC’s evidence synthesis infrastructure hasn’t typically needed to turn on a dime, and I hope that continues to be the case. That said, there may be lessons to be learned from this moment, about both how the WWC does its own work and how it supports the work of the field. To that end, I’d offer a few thoughts.

The WWC could support partners in research and content creation who can act nimbly, maintaining pressure for rigorous work.

Educators have questions that span every facet of their work, every subject, and every age band. And there’s a lot of education research out there, from complex, multi-site RCTs to small, qualitative case studies. The WWC doesn’t have the capacity to either answer every question that deserves answering or synthesize every study we’re interested in synthesizing. (Not to mention the many types of studies we don’t have good methods for synthesizing today.)

This suggests to me there is a potential market for researchers and technical assistance providers who can quickly identify high-quality evidence, accurately synthesize it, and create educator-facing materials that can make a difference in classroom practice. Some folks have begun to fill the gap, including both familiar faces and not-so-familiar ones. Opportunities for collaboration abound, and partners like these can be sources of inspiration and innovation for one another and for the WWC. Where there are gaps in our understanding of how to do this work well that can be filled through systematic inquiry, IES can offer financial support via our Statistical and Research Methodology in Education grant program.   

The WWC could consider adding new products to its mix, including rigorous rapid evidence syntheses.

Anyone who has visited us at whatworks.ed.gov recently knows the WWC offers two types of syntheses: Intervention Reports and Practice Guides. Neither are meant to be quick-turnaround products.

As their name implies, Intervention Reports are systematic reviews of a single, typically brand-name, intervention. They are fairly short, no longer than 15 pages. And they don’t take too long to produce, since they’re focused on a single product. Despite having done nearly 600 of them, we often hear we haven’t reviewed the specific product a stakeholder reports needing information on. Similarly, we often hear from stakeholders that they aren’t in a position to buy a product. Instead, they’re looking for the “secret sauce” they could use in their state, district, building, or classroom.

Practice Guides are our effort to identify generalizable practices across programs and products that can make a difference in student outcomes. Educators download our most popular Guides tens of thousands of times a year, and they are easily the best thing we create. But it is fair to say they are labors of love. Each Guide is the product of the hard work of researchers, practitioners, and other subject matter experts over about 18 months.  

Something seems to be missing from our product mix. What could the WWC produce that is as useful as a Practice Guide but as lean as an Intervention Report? 

Our very wise colleagues at the UK’s Education Endowment Foundation have a model that is potentially promising: Rapid Evidence Assessments based on pre-existing meta-analyses. I am particularly excited about their work because—despite not coordinating our efforts—they are also focusing on Distance Learning and released a rapid assessment on the topic on April 22nd. There are plusses and minuses to their approach, and they do not share our requirement for rigorous peer review. But there is certainly something to be learned from how they do their work.

The WWC could expand its “what works” remit to include “what’s innovative,” adding forward-looking horizon scanning to here-and-now (and sometimes yesterday) meta-analysis.

Meta-analyses play a critical role in efforts to bring evidence to persistent problems of practice, helping to sort through multiple, sometimes conflicting studies to yield a robust estimate of whether an intervention works. The inputs to any meta-analysis are what is already known—or at least what has already been published—about programs, practices, and policies. They are therefore backward-looking by design. Given how slowly most things change in education, that is typically fine.

But what help is meta-analysis when a problem is novel, or when the best solution isn’t a well-studied intervention but instead a new innovation? In these cases, practitioners are craving evidence before it has been synthesized and, sometimes, before it has even been generated. Present experience demonstrates that any of us can be made to grasp for anything that even smacks of evidence, if the circumstances are precarious enough. The challenge to an organization like the WWC, which relies on traditional conceptions of rigorous evidence of efficacy and effectiveness, is a serious one.

How might the WWC become aware of potentially promising solutions to today’s problems before much if anything is known about their efficacy, and how might we surface those problems that are nascent today but could explode across the landscape tomorrow? 

One model I’m intensely interested in is the Health Care Horizon Scanning System at PCORI. In their words, it “provides a systematic process to identify healthcare interventions that have a high potential to alter the standard of care.” Adapted to the WWC use case, this sort of system would alert us to novel solutions: practices that merited monitoring and might cause us to build and/or share early evidence broadly to relevant stakeholders. This same approach could surface innovations designed to solve novel problems that weren’t already the subject of multiple research efforts and well-represented in the literature. We’d be ahead of—or at least tracking alongside—the curve, not behind.  

Wrapping Up

The WWC’s current Rapid Evidence Synthesis focused on distance learning is an experiment of sorts. It represents a new way of interacting with our key stakeholders, a new way to gather evidence, and a new way to see our reviews synthesized into products that can improve practice. To the extent that it has pushed us to try new models and has identified hundreds of “new” (or “new to us”) studies, it is already a success. Of course, we still hope for more.

As I hope you can see from this blog, it has also spurred us to consider other ways we can further strengthen an already strong program. I welcome your thoughts and feedback – just email me at matthew.soldner@ed.gov.