IES Blog

Institute of Education Sciences

The Enduring Friendship of the MOCCA Team: How Camaraderie Benefits Research

This blog is a part of our Spotlight on IES Training Programs series and was written by Shirley Liu, a virtual intern for NCER.

One of the important—though sometimes overlooked—benefits of the IES training programs is friendship. When you think of what makes a good research team, friendship is probably not your first answer. However, the researchers behind the Multiple-Choice Online Causal Comprehension Assessment (MOCCA) demonstrate just how crucial strong bonds are. This blog shares how three long-time friends and members of the MOCCA team–Drs. Gina Biancarosa, Sarah Carlson, and Ben Seipel–have benefited from friendship.

 

(From top left: Sarah, Gina; Ben, Sarah; Gina, Ben)

 

How they met

Sarah and Ben first met during their IES predoctoral program at the University of Minnesota. During a grant-writing course, they developed a proposal for what would eventually become MOCCA. When Sarah attended the University of Oregon for her IES postdoctoral program, she met Gina, who ultimately joined in the MOCCA research.

The three scholars shared a passion for reading comprehension and assessment and a love for trading jokes. The team’s love of cute animal stories, especially otters, as another reason they get along well. “Every otter story that was in the news got shared multiple times,” Ben said as others laughed in agreement. Over the years, they have continued to invest in their shared interests and in one another.

How their friendship benefits their work

The three credit their friendship as contributing to their personal and professional growth in three key ways.

Combatting loneliness. According to Gina, “[Socialization] just gets you out of your head. That is not only good for your emotional health and mental health, but also for stimulating new ideas and improving the rigor of old ideas.” Whether it is visiting cool restaurants, taking pictures of each other with funny filters after long conferences, or going on retreats, the MOCCA team makes sure to create time for non-research related activities. Even during the pandemic, MOCCA still prioritizes the socialization aspect of their research by meeting online instead.

Creating a supportive atmosphere that encourages taking risks. The MOCCA team has found that their friendship creates an open-minded and supportive atmosphere for their research. This environment encourages risk taking and helps researchers voice their opinions. In turn, this stimulates innovation and intellectual diversity. “It makes it easier to float ideas that you think might not be all there and not have to risk rejection. They’ll tell you if it’s not all there, but you’re not going to feel crushed,” explained Gina. “It makes you take more risks.”

Fostering growth and personal development. The MOCCA team has also found that friendship leads them to see one another as more than just experts. Instead, they acknowledge their individual strengths while encouraging one another to grow intellectually as complex and constantly learning individuals. As a result, each member of the MOCCA team contributes to the research in unique and equally appreciated ways. “We all have that creative energy, but we have different types of creative energy,” said Ben. “Sarah is really the dreamer: What can this look like? What can it do for teachers? And I really am an innovator: I take things that are different, make them new, and get at things that we have not been able to get at in the past. But Gina really brings that maker aspect: How can we actually make this work? What are the things that function in our toolbox to make it happen?”

The value of friendship

Although the MOCCA team’s bond seems like a uniquely serendipitous union of like-minded people, all of us can reap the benefits of friendship in research and in everyday life. This past year has taught us the value of community and personal relationships in times of isolation. Researchers like the MOCCA team have known this for years.


Dr. Carlson and Dr. Seipel were predoctoral fellows in the Minnesota Interdisciplinary Training in Education Research program, Dr. Biancarosa was a postdoctoral fellow in Stanford University’s Postdoctoral Research Training in the Education Sciences program, and Dr. Carlson was a postdoctoral fellow in the Preparing Education Scientist training program. For more information about MOCCA, please visit the MOCCA webpages (here and here). 

The MOCCA team has been awarded three IES grants to support their measurement work: Multiple-choice Online Cloze Comprehension Assessment (MOCCA) (R305A140185); Multiple-choice Online Causal Comprehension Assessment for Postsecondary Students (MOCCA-College) (R305A180417); Multiple-choice Online Causal Comprehension Assessment Refinement (R305A190393).

By Shirley Liu, virtual intern for NCER and an English/Anthropology & Sociology double major at Lafayette College.

National Developmental Disabilities Awareness Month

March has been National Developmental Disabilities Awareness Month since 1987. President Reagan’s goal for this annual observation was to increase “public awareness of the needs and the potential of Americans with developmental disabilities” and to provide the opportunities and supports individuals with developmental disabilities may need to lead productive lives and reach their full potential.

Special education research is improving ways in which educators help realize this goal by enhancing teaching strategies for working with students with developmental disabilities and their families from early childhood through the transition to young adulthood. Below are examples of current NCSER-funded projects that are focused on supporting educators who work with students with developmental disabilities across childhood and adolescence.

Early Childhood

Early childhood and education (ECCE) providers play an important role in the development and well-being of children; however, training opportunities focused on working with children with developmental disabilities are often limited for ECCE providers. Dr. Rebecca Landa at the Kennedy Krieger Institute is developing a professional development program for ECCE providers to implement Early Achievements (originally developed for young children with autism spectrum disorders) with children with developmental disabilities. The program will train ECCE providers to implement the three evidence-based practices of Early Achievements—explicit targeting of language, social, and cognitive development; strategies to enhance meaning (such as themes and hand-on learning); and naturalistic developmental behavior strategies (such as prompts and natural reinforcers).

Elementary Level

At Purdue University, Dr. Rose Mason and colleagues are developing Para-Impact, a PD package for educators who work with elementary students with developmental disabilities. Para-Impact trains special educators to use practice-based coaching to support paraprofessional implementation of systematic instruction. Paraprofessionals often have access to few formal training opportunities on how to implement evidence-based practices, and special educators often have limited experience supervising and training paraprofessionals to implement such practices. Dr. Mason’s work addresses this gap by supporting educators in training and supervising paraprofessionals in the use of systematic instruction with students with developmental disabilities. The ultimate goals of this work are to increase the engagement of students with developmental disabilities in the classroom and to increase student progress on their individualized education goals.

High School

At the University of Massachusetts Boston, Dr. Allison Hall is exploring whether and how the information special educators provide to parents about transfer rights and guardianship may support or limit transition outcomes for students with developmental disabilities. Special education regulations state that parental decision-making rights will transfer to students at the age of 18 unless parents obtain guardianship. During transition planning, special educators frequently encourage parents to seek guardianship despite the growing array of available formal and informal alternatives to guardianship, such as supported decision making. Dr. Hall and her research team are examining the factors that affect how special educators provide this information to families and the ways in which this information may impact transition outcomes, such as parent expectations and student self-determination.

We look forward to seeing how these projects support students with developmental disabilities in leading productive lives and achieving their full potential.

This blog was authored by Alice Bravo (University of Washington), IES intern through the Virtual Student Federal Service.

Middle Level Education Month: Celebrating the Early Adolescent Years

March is Middle Level Education Month, a month in which we recognize the important early adolescent years for learning and growth. For young adolescents, particularly adolescents with or at risk for disabilities, these can be important years to further develop executive functioning and self-regulation skills as opportunities for independence expand and the academic and behavioral expectations increase. For middle school teachers of students with or at risk for disabilities, classroom management and effective behavior supports may be particularly important. Many middle school teachers experience student disengagement and disruptive behavior. To support middle school teachers and students, several researchers are exploring ways to promote appropriate behavior, support executive functioning and self-regulation skills, and enhance academic engagement in middle schoolers with or at risk for disabilities through grants from the National Center for Special Education Research (NCSER).

Two NCSER-funded researchers are currently developing interventions to improve learning and behavior for middle school students. At the University of Wisconsin-Madison, Dr. Andrew Garbacz has an Early Career Development and Mentoring grant to iteratively adapt and test the Conjoint Behavioral Consultation (CBC) service model for middle school students with or at risk for serious emotional disturbance. CBC (also known as Teachers and Parents as Partners) is an indirect service delivery model, previously tested with younger students, that partners parents, educators, and other key stakeholders in data-driven, collaborative problem-solving and implementation of evidence-based interventions to address challenging behavior. At the Cincinnati Children’s Hospital Medical Center, Dr. Leanne Tamm is adapting and testing the Teaching Academic Skills to Kids—School-based intervention. This intervention was originally developed for individuals with attention-deficit hyperactivity disorder and is currently being adapted to focus on the specific needs of middle schoolers with high-functioning autism spectrum disorder. The adapted intervention will target executive functioning skills, academic behaviors (such as homework completion), and academic outcomes.

In addition to this work on developing interventions, three NCSER-funded researchers are testing the efficacy of existing interventions that aim to improve the behavior, engagement, school adjustment, and academic outcomes of middle school students with or at risk for emotional and behavioral disorders (EBD). At the University of Florida, Dr. Stephen Smith is examining the efficacy of I Control, an intervention for middle school students with EBD that targets executive functioning skills. At the Oregon Social Learning Center, Dr. Rohanna Buchanan is evaluating the efficacy of the Students with Involved Families and Teachers (SWIFT) program, which is intended to improve school adjustment for students with EBD who are transitioning between school placements or who are at risk of being placed in a more restrictive setting (such as a residential facility). The goal of SWIFT is to promote successful student transitions and to increase parental involvement in schools. At the University of Kansas, Dr. Howard Wills is evaluating the efficacy of Class-Wide Function-Based Intervention Teams Middle School (CW-FIT MS). CW-FIT MS aims to improve engagement, academic outcomes, and socially appropriate behaviors of middle school students with or at risk for EBD while improving teacher classroom management practices.

IES is committed to improving learning opportunities and outcomes for middle school students with and without disabilities, and we look forward to seeing how these projects will help support this goal.

This blog was authored by Alice Bravo (University of Washington), IES intern through the Virtual Student Federal Service.

When “More Research is Needed” Is the Key Finding: Improving the Evidence Base for Nonacademic Interventions for Postsecondary Success in Rural and High-Poverty Contexts

Stakeholders in rural and high-poverty districts in Regional Educational Laboratory (REL) Appalachia’s region have noticed a troubling trend: many students graduate from high school academically well prepared but fail to enroll in college or enroll in college only to struggle and drop out within the first year. Stakeholders believe these high-performing students may face nonacademic challenges to postsecondary success, such as completing financial aid paperwork, securing transportation and housing at colleges far from home, or adjusting to campus life. To address these challenges, education leaders are looking for interventions that address nonacademic competencies: the knowledge, skills, and behaviors that enable students to navigate the social, cultural, and other implicit demands of postsecondary study.

To fill this need, REL Appalachia researchers conducted a review of the existing evidence of the impact of nonacademic interventions – that is, those designed to address nonacademic competencies – on postsecondary enrollment, persistence, and completion. The review had a particular focus on identifying interventions that also have evidence of effectiveness in communities serving students similar to those in Appalachia—high-poverty, rural students. Only one intervention, Upward Bound, demonstrated impact in rural, high-poverty communities. The review showed that Upward Bound, as implemented in the early 1990s, benefited high-poverty rural students’ college enrollment, with no demonstrated impact on persistence or completion.

Schools and communities need access to nonacademic interventions that benefit students served in high-poverty rural communities. Researchers: read on to learn more about the methods used in the evidence review, its findings, and steps you can take to support rural and high-poverty communities in improving enrollment and success in postsecondary education!

Nonacademic challenges to postsecondary success for rural students

All students face nonacademic challenges to postsecondary success, but rural populations and high-poverty populations in particular may benefit from interventions addressing those challenges because they enroll in and complete college at significantly lower rates than their nonrural or low-poverty peers. Although academic challenges contribute to this gap, rural and high-poverty populations also face unique nonacademic challenges to postsecondary enrollment and success. For example, rural students are less likely to encounter college-educated role models and high-poverty students often face inadequate college counseling at their schools (see research here, here, and here). As a result, rural and high-poverty students may have inadequate access to knowledgeable adults who can help them understand the steps needed to enroll or prepare them for the challenges of persisting in postsecondary education.  Nonacademic interventions can support students in developing the knowledge, skills, and behaviors necessary to overcome these challenges and improve postsecondary enrollment and success for rural and high-poverty students.

The need for evidence-based interventions

To support decisionmakers at rural and high-poverty schools in identifying evidence-based nonacademic interventions, researchers at REL Appalachia conducted an extensive search of the published research. The search looked for rigorous studies of nonacademic interventions with evidence of positive impact on college enrollment, persistence, performance, and completion for students attending rural schools or who were identified as high poverty. The purpose of the project was to identify a suite of interventions to recommend to these education leaders.

The results of our review indicate there may be gaps in the evidence available to all decisionmakers who are trying to help their students succeed in postsecondary education. The search first identified any studies that focused on postsecondary outcomes of nonacademic interventions serving students ages 5–19. Of the 1,777 studies with the relevant keywords, only 65 focused on the postsecondary outcomes of nonacademic interventions. Next, we evaluated these 65 studies against the What Works Clearinghouse (WWC) design standards, which assess the quality of evaluation study designs. Only 17 studies met WWC’s rigorous study design standards with or without reservations. Finally, researchers from REL Appalachia identified studies that showed positive impacts on students overall, and studies that looked at rural students and students identified as high poverty in particular. Only eight studies showed positive, statistically significant impacts on students’ postsecondary enrollment or success overall. Of the eight studies that showed positive impacts of nonacademic interventions on postsecondary outcomes, only three focused on high-poverty populations, and only one reported specifically on rural populations.

This figure shows the number of studies remaining at each stage of screening. The original searches returned 1,777 unique studies. Of these, 65 focused on postsecondary outcomes of nonacademic interventions with students ages 5 to 19. At the next stage, 17 studies remained that met these criteria and also met WWC standards. At the final stage, 8 studies remained that met all criteria and had a positive effect on postsecondary outcomes.

 Without additional research that focuses on low-income and rural contexts, schools and districts are left to implement programs with limited or no evidence of effectiveness. For example, the Quantum Opportunity Program (QOP) provides mentors to students as part of a long-term after school program. However, WWC reviews of QOP studies (here and here) showed indeterminate effects of the program on postsecondary outcomes. The lack of evidence should not detract from the important role QOP has in serving students, but it leaves open the question of whether those efforts are having the intended effects. With few clear alternatives, schools and districts continue to implement programs with limited evidence of effectiveness.

Action steps

Nationwide, 19 percent of U.S. public school students are enrolled in a rural school, and 24 percent are enrolled in a high-poverty school. To help districts and schools provide effective supports to those students, researchers can provide high-quality evidence on the effectiveness of nonacademic interventions in these contexts.

Carry out more studies on specific interventions designed to improve nonacademic competencies. REL Appalachia’s review found that the research on nonacademic competencies often focuses on defining the competencies themselves, rather than on studying interventions designed to develop the competencies. Of the 1,777 unique studies identified in our review, only 65 (3 percent) studied outcomes of interventions designed to improve nonacademic competencies. From these, we identified only 17 studies, representing nine interventions, with sufficiently rigorous designs to examine evidence of effectiveness.

The limited availability of rigorous evaluations of interventions suggests that, as researchers, we need to increase our focus on evaluating new interventions as they are developed or tested. Decisionmakers rarely design their own programs or interventions from scratch; they need to be able to identify existing programs and policies that are within their power to implement and have been proven effective in similar communities. Researchers can help decisionmakers select and implement successful interventions by providing evidence on whether interventions that develop students’ nonacademic competencies have positive effects on students’ postsecondary outcomes.

Design studies to generalize to rural and high-poverty populations. As researchers, we can also increase our focus on rural and high-poverty populations. REL Appalachia’s review found only three studies that focused on a high-poverty population and one that focused on a rural population. As researchers, we can address this gap in two ways: (a) we can carry out more studies specifically focused on rural and high-poverty areas; and (b) when using large national datasets or multi-site studies, we can consider rural and high-poverty populations in our sampling and disaggregate our results for these populations.

Summary

Stakeholders in rural and high-poverty contexts are looking for nonacademic interventions that will be effective with their students. To that end, REL Appalachia carried out an extensive review of evidence-based interventions. The review found few rigorous studies of nonacademic interventions, and even fewer that examined findings for students identified as high poverty or in rural settings. Without additional research, schools and districts serving rural and high-poverty populations may implement interventions that are not designed for their circumstances and may not achieve intended outcomes. As a result, resources may be wasted while rural and high-poverty students receive inadequate support for postsecondary success.  In addition to investing in rigorous studies, which can take a long time to complete, researchers and practitioners can also collaborate to implement short-term research methods to identify early indicators of the success of these programs. For example, researchers may be able to support schools and districts in developing descriptive studies examining change over time or change in formative assessment outcomes.

 

 

 Researchers have a role in helping more high school graduates from rural communities enroll, persist, and succeed in postsecondary education.

 

Rural and high-poverty schools and districts have unique strengths and challenges, and the lack of information about how interventions perform in those contexts presents a dilemma for decisionmakers: do nothing, or else muddle through with existing evidence, investing in interventions that don’t address local needs. As researchers, we can help resolve this dilemma by providing rigorous evidence about effective interventions tailored to rural and high-poverty contexts, as well as supporting practitioners in using more accessible methods to investigate the short-term outcomes of the programs they are already implementing.

 

by Rebecca A. Schmidt and CJ Park, Regional Educational Laboratory Appalachia

 

Towards a Better Understanding of Middle-Schoolers’ Argumentation Skills

What is the difference between fact and opinion? How do you find relevant evidence and use it to support a position? Every day, teachers help students practice these skills by fostering critical discussions, a form of argumentation that encourages students to use reasoning to resolve differences of opinion.

In their IES-funded study, Exploring and Assessing the Development of Students' Argumentation Skills, Yi Song and her colleagues are uncovering activities (both teacher led and technology supported) that can improve middle-school students’ ability to generate better oral and written arguments.

This project began in 2019 and is working in classrooms and with teachers and students. The researchers have created a series of videos that describe their work. In this series, Dr. Song and her co-PIs, Dr. Ralph Ferretti and Dr. John Sabatini, discuss why the project is important to education, how they will conduct the research plan, and how educators can apply what they are learning in classrooms.

 

 


For questions and more information, contact Meredith Larson (Meredith.Larson@ed.gov), Program Officer, NCER