Inside IES Research

This year, Inside IES Research is publishing a series of interviews showcasing a diverse group of IES-funded education researchers that are making significant contributions to education research, policy, and practice.

The process of education transmits sociocultural values to learners in addition to information and knowledge. How individuals are represented in curricula and instructional materials can teach students about their place in the world. This can either perpetuate existing systemic inequalities or, conversely, provide a crucial counternarrative to them. With an exploration grant from IES, Anjali Adukia (University of Chicago) and Alex Eble (Teachers College, Columbia University) are exploring how  representation and messages about gender and race in elementary school books may influence student’s education outcomes over time. The researchers will develop and use machine-learning tools that leverage text and image analysis techniques to identify gender- and race-based messages in commonly used elementary-school books.

Interview with Anjali Adukia, University of Chicago

Tell us how your research contributes to a better understanding of the importance of diversity, equity, and/or inclusion in education.

In my work, I seek to understand how to reduce inequalities such that children from historically (or contemporaneously) marginalized backgrounds have equal opportunities to fully develop their potential. I examine factors that motivate and shape behavior, preferences, and educational decision-making, with a particular focus on early-life influences. Proceeding from the notion that children are less likely to be able to focus on learning until their basic needs are met, my research uses both econometric methods and qualitative approaches to understand the specific roles different basic needs play in making these decisions. My research, for example, has explored the role of safety and health (sanitation, violence), economic security (road construction, workfare), justice (restorative practices), and representation (children’s books), particularly for marginalized groups.

As a woman and a minority, how has your background and experiences shaped your career?

My research is informed and influenced by my own experiences. When I was a child, I never understood why there weren’t more characters that looked like me or when there were, why they had such limited storylines. For me personally, the motivation underlying our IES-funded project was borne out of my lived experience of always searching for content that reflected who I was. I think of representation as a fundamental need: if you don’t see yourself represented in the world around you, it can limit what you see as your potential; and similarly, if you don’t see others represented, it can limit what you see as their potential; and if you only see certain people represented, then this shapes your subconscious defaults.

It was a real watershed moment when I realized that academia allowed me to pursue many of my larger goals in life, in which I hope to meaningfully improve access to opportunities and outcomes for children – especially those from disadvantaged backgrounds. I hope to accomplish this in various ways, paying forward the many kindnesses generously given to me by: (1) producing rigorous policy-relevant evidence that expands our understanding of big questions and opens new avenues for inquiry; (2) translating my research such that it helps inform policymakers and practitioners in the design of school policies and practices; (3) understanding issues with a depth and sophistication that comes from “on-the-ground” insights, knowledge cultivated in multiple disciplines using different methodologies, introspection, humility, and courage; (4) directly working with government agencies, non-profit organizations, and community groups to positively inform policies; (5) contributing to the larger public discourse; and (6) by training and advising students to have the fortitude to ask hard questions, to be able to defend different perspectives on issues, to learn that knowledge brings more questions than answers, and to be willing to take risks and fail (and in the process, I will certainly learn more from them than they will ever be able to learn from me).

What has been the biggest challenge you encountered and how did you overcome the challenge?

Life is always filled with challenges, but one challenge starting from when I was young was to feel comfortable in my own skin and to find legitimacy in my own voice. I grew up as an Indian-American daughter of Hindu immigrants in a rural, predominantly white and Christian setting. I was different from the other kids and did not always feel like I fit in. I remember literally trying to erase my skin hoping that it would make it lighter. I found the helpers, as my parents (and Mr. Rogers) would suggest, and tried to focus on the voices that lifted me up – my family, teachers, other mentors, those friends who loved me no matter my differences. My mother always told me to find the kindness, the good, the love in people; to find the common ground and to embrace and learn from the differences. I surrounded myself with love, focusing on what I had and on what I could do rather than what society was telling me I couldn’t do. I turned to concentrating on things that mattered to me, that drove me. I don’t think there is a single challenge in life that I overcame alone. I have been very lucky, and I am deeply grateful for the many gifts in my life, the many loved ones – family, friends, colleagues, mentors, healthcare workers – who have lifted me up, and the opportunities that came my way.

How can the broader education research community better support the needs of underrepresented, minority scholars?

The notion of what is considered to be an important question is often driven by the senior scholars in a field, for example, the people considered to be “giants.” Demographically, this small set of leading scholars has historically consisted of people from the most highly represented groups (particularly in economics). And because the field is thus shaped mainly by researchers from a “dominant” group background, the key questions being pursued may not always reflect the experiences or concerns of people from underrepresented backgrounds. Education research has pockets where these different perspectives are being considered, but it can continue to evolve by becoming more open to approaches thought to be less traditional or to questions not typically asked (or asked from a different point of view). Expanding the notion of what is considered important, rigorous research can be difficult and cause growing pains, but it will help expand our knowledge to incorporate more voices.

What advice would you give to emerging scholars from underrepresented, minority backgrounds that are pursuing a career in education research?

Keep a journal of questions that arise and topics that pique your curiosity and interest. Soon, you will find questions in the fabric of everyday life, and you will start to articulate the wonder you see in the world around you and what inspires you to action, to understand the universe further. I find that when I return to past writings and journal entries, I am reminded of questions that have ignited my fires and see some of the common themes that emerge over time. Find your voice and know that your voice and views will grow and evolve over time. There are so many interesting and important questions one can pursue. Most importantly, you have to be true to yourself, your own truth. Find circles of trust in which you can be vulnerable. Draw strength from your struggle. There is deep truth and knowledge within you.

Dr. Anjali Adukia is an Assistant Professor at the University of Chicago Harris School of Public Policy and the College.

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov), Program Officer, National Center for Education Research.

The more you already know about a topic, the easier it may be to comprehend and learn from texts about that topic. But knowledge has to start somewhere. So how can we help students learn from texts when they may have low background knowledge?

In their exploratory study, researchers from ETS found that lack of knowledge is not necessarily a barrier to comprehension. Rather, they suggest that students who can identify their lack of background knowledge are more likely to comprehend and learn new information than students who do not acknowledge they lack background knowledge. In other words, knowing that you might not know may lead to better outcomes.

To determine the role of background knowledge, the researchers pretested middle and high school students’ background knowledge through questions related to topics the students may have some but not complete knowledge of, such as ecology, immigration, and wind power. The pretest included an “I don’t know” option, along with correct and incorrect responses.

Students then took a scenario-based assessment in which they read multiple sources about each of the topics. This type of assessment mirrors real-world learning by encouraging readers to build their own interpretations of a topic, which helps researchers determine whether students comprehend what they read.

They found that students who selected “I don’t know” when answering background knowledge questions had better understanding of the content than those who provided wrong answers on these questions. In fact, students who selected “I don’t know” rather than answering incorrectly were nearly three times as likely to learn from sources that provided the correct information than students who had answered the pretest incorrectly. Students who selected “I don’t know” may also learn more than students who had a comparable level of weak background knowledge. The researchers suggest that the “I don’t know” readers may have set different reading goals prior to engaging with the sources than those who guessed incorrectly.

Possible Implications for Teaching and Learning

The results from this work support the idea that having and building background knowledge is key. Thus, teachers may want to assess existing knowledge and address knowledge gaps prior to instruction.

Teachers may also want to provide an “I don’t know” option or options that allow students to rate their level of certainty. Doing so may help teachers distinguish between students who recognize their own gaps in knowledge from those who may not be aware that they are wrong or that they simply do not know. This latter group of students may need more help in determining the accuracy of their judgments or may have incorrect knowledge that could interfere with learning.

The researchers further suggest that teachers may want to go beyond the role of background knowledge by teaching students how to set appropriate reading goals and use strategic reading approaches to learn new facts or correct existing misunderstandings.

The research reported here was conducted under NCER grant R305A150176: What Types of Knowledge Matters for What Types of Comprehension? Exploring the Role of Background Knowledge on Students' Ability to Learn from Multiple Texts.

This blog was written by Dr. Meredith Larson. Contact her for more information about this project.

Under an IES grant, the RAND Corporation, in collaboration with NWEA, is developing strategies for schools and districts to address the impacts of COVID-19 disruptions on student assessment programs. The goal is to provide empirical evidence of the strengths and limitations of strategies for making decisions in the absence of assessment data. Jonathan Schweig, Andrew McEachin, and Megan Kuhfeld describe early findings from surveys and structured interviews regarding key concerns of districts and schools. 

As a first step, we surveyed assessment and research coordinators from 23 school districts (from a sample of 100 districts) and completed follow-up interviews with seven of them on a variety of topics, including the re-entry scenario for their district, the planning activities that they were not able to perform this year due to coronavirus-based disruptions to spring 2020 assessments, and the strategies they were employing to support instructional planning in the absence of assessment data. While the research is preliminary and the sample of respondents is not nationally representative, the survey and interview responses identified two key concerns arising from the lack of spring 2020 assessment data which has made it challenging to examine student or school status and change over time, especially as COVID-19 has differential impacts on student subgroups:

• Making course placement decisions. Administrators typically rely on spring assessment scores—often in conjunction with other assessment information, course grades, and teacher recommendations—to make determinations for course placements, such as who should enroll in accelerated or advanced mathematics classes. 
• Evaluating programs or district-wide initiatives. Many districts monitor the success of these programs internally by looking at year-to-year change or growth for schools or subgroups of interest. 

How are school systems responding to these challenges? Not surprisingly, the responses vary depending on local contexts and resources. Where online assessments were not feasible in spring 2020, some school districts used older testing data to make course recommendations, either from the winter or from the previous school year. Some districts relaxed typical practice and provided more autonomy to individual schools, relying on school staff to exercise local judgment around course placements and using metrics like grades and teacher recommendations. Other districts reported projecting student scores based on student assessment histories. Relatedly, some districts were already prepared for this decision because they had recently experienced difficulties with adopting an online assessment system and had to address similar problems caused by large numbers of missing or invalid tests.

School districts also raised concerns about whether assessments administered during the 2020-21 school year would be valid and comparable so that they could be used in student placement and program evaluation decisions. These concerns included the following:

• Several respondents raised concerns about the trustworthiness of remote assessment data collected this fall and the extent to which results could be interpreted as valid indicators of student achievement or understanding.
• Particularly for districts that started the 2020-21 school year remotely, respondents were concerned about student engagement and motivation and the possibility of students rushing assessments, running into technological or internet barriers, or seeking assistance from guardians or other resources. 
• Respondents raised questions about the extent to which available assessment scores are representative of school or district performance as a whole. Given that vulnerable students (for example, students with disabilities, students experiencing homelessness) may be the least likely to have access to remote instruction and assessments, it is likely that the students who are not assessed this year are different from students who are able to be assessed.
• Other respondents noted that they encountered resistance from parents around fall assessment because they prioritized student well-being (for example, safety, sense of community, and social and emotional well-being) more so than academics. This is a perspective that resonates with recent findings from a nationally representative sample of teachers and school leaders drawn from RAND’s American Educator Panel (AEP).

In the next phase of the work, the research team plans to:

• Conduct a series of simulation and empirical studies regarding the most common strategies that the district respondents indicated they were using to make course placement decisions and to evaluate programs or district-wide initiatives.
• Provide a framework to help guide local research on the intended (and unintended) consequences for school and school system decision making when standardized test scores are not available.

We welcome individuals to reach out to RAND with additional recommendations or considerations. We are also interested in hearing how districts are approaching course placement, accountability, and program evaluation across the country. Connect with the research team via email at jschweig@rand.org.

Jonathan Schweig is a social scientist at the nonprofit, nonpartisan RAND Corporation.

Andrew McEachin is a senior policy researcher at the nonprofit, nonpartisan RAND Corporation.

Megan Kuhfeld is a researcher at NWEA.

Many elementary school students experience anxiety that interferes with learning and achievement, but few receive services. To expand the network of support for these young students, IES-funded researchers have turned to school nurses as a potential front-line resource. The Child Anxiety Learning Modules (CALM) intervention incorporates cognitive-behavioral therapy (CBT) and other evidence-based strategies for school nurses to use when a child has vague somatic complaints that often signal underlying anxiety.

In 2014, IES funded a Development and Innovation grant to support the development of CALM to enhance the capacity of elementary school nurses to help children with anxiety. Based on promising findings of feasibility and reduced anxiety and fewer school absences, the development team is launching an initial efficacy trial this fall to investigate the scale up potential of the CALM intervention.

We asked the developers of CALM—Golda Ginsburg (University of Connecticut School of Medicine) and Kelly Drake (Founder/Director of the Anxiety Treatment Center of Maryland; Johns Hopkins University School of Medicine)—to answer a few questions for our blog. Here’s what they answered.

Can you describe how the CALM intervention was developed? What led you to develop an intervention for school nurses to implement?

We have been developing and evaluating psychosocial interventions for youth with anxiety for the last two decades, and we’ve learned a lot about effective, evidence-based strategies. We know that CBT, which consists of coping strategies that target the physical, cognitive, and behavioral manifestations of anxiety, is effective in helping youth manage and reduce anxiety. Unfortunately, we’ve also learned that most youth do not receive these—or any—services to help them. To address this gap in service utilization, our efforts have focused on ways of improving access to these therapeutic strategies by broadening the pool of potential providers. Given that early interventions can reduce the long-term consequences of untreated anxiety AND that youth with anxiety often complain of troublesome physical symptoms at school, we naturally thought of school nurses as a key provider with enormous potential. However, although nurses reported spending a lot of time addressing mental health issues, they received minimal training in doing so. That’s when the idea of the CALM intervention was born. We developed the initial CALM intervention using an iterative process in which versions of the intervention and its implementation procedures were sequentially refined in response to feedback from expert consultants, school nurses, children, parents, and school personnel until it was usable in the school environment by school nurses.

Was it part of the original plan to develop an intervention that could one day be used at scale in schools?

Yes—absolutely! Members of the National Association of School Nurses have been on our advisory team throughout to help us plan for how to scale up the intervention if we find it helps students.

What was critical to consider during the research to practice process?

A central focus was to minimize burden on school staff and to integrate the intervention within the goals and mission of schools’ interdisciplinary teams. Therefore, using a multidisciplinary support team was critical in taking the intervention from a research idea to an intervention that school nurses could delivered in their real-world practice setting—schools! As clinical psychologists, we also relied on our multidisciplinary team to ensure the intervention was usable by school nurses in terms of content and flexible and feasible for their busy school day. Indeed, school nurses and school nurse organizations provided critical support for the development of CALM with a focus on feasible strategies and methods for nurses to implement. They also provided invaluable feedback regarding perceived barriers to successful implementation of the intervention and adoption by nurses and school systems, and solutions to potential barriers and options for scaling up the intervention. We also relied on experts in school-based mental health programs and those with expertise in designing, evaluating, and implementing evidence-based prevention programs in schools. We also leveraged state-level expertise by consulting with school health experts in the Connecticut State Department of Education and the Connecticut Nurses Association regarding mental health education for nurses.

What model are you using for dissemination and sustainability?

A wide variety of methods will be used to disseminate findings from the current study to reach different stakeholders. We will present and publish findings at 1) national scientific and practitioner-oriented conferences, 2) Maryland and Connecticut State Departments of Education and participating school districts, and 3) in relevant peer-reviewed journals. In addition, should the findings reveal a beneficial impact of the intervention, we will have the final empirically supported training and intervention materials available for broad scale implementation. The CALM intervention will be packaged to include a training seminar, training videos, nurse intervention manual, child intervention handouts, consultation/coaching plan, and assessment materials. The research team will offer training seminars with all supporting materials to school nurse organizations at the national, state, and local levels. We will also engage nurse supervisors to identify nurses—or volunteer themselves—to become trainers for newly hired nurses in the future. Finally, our current Advisory Board, which consists of members of the National Association of School Nurses (NASN), school nurses, and researchers with expertise in large scale school-based mental health program implementation and evaluation, will assist in broad dissemination and sustainability efforts.

Golda S. Ginsburg, Ph.D., Professor of Psychiatry, University of Connecticut School of Medicine and Adjunct Professor at The Johns Hopkins University School of Medicine, has over 25 years of experience developing and evaluating school-based interventions including school-based interventions for anxiety delivered by school clinicians, teachers, and nurses.

Kelly Drake, Ph.D., Founder/Director of the Anxiety Treatment Center of Maryland, Research Consultant with UConn, and Assistant Professor of Psychiatry in the JHU School of Medicine has extensive training and experience in clinical research with anxious youth and training clinicians in delivering CBT for children.

This interview was produced by Emily Doolittle (Emily.doolittle@ed.gov) of the Institute of Education Sciences. This is part of an ongoing interview series with education researchers, developers, and partners who have successfully advanced IES-funded education research from the university laboratory to practice at scale.

During spring 2020, the COVID-19 pandemic forced the closure of millions of U.S. schools. As schools reopened this fall, conversations have revolved around using this unique situation as a chance to rethink education and how students learn. When we think about innovative ways to improve education, ideas tend to gravitate towards radical changes to the classroom experience, expensive interventions, and costly professional development. Everyone is looking for the next “big” idea, but perhaps part of the solution lies in a more subtle, inexpensive, and less disruptive change that may be as impactful as a completely new education approach: strategic revisions to the materials teachers and students already use in their classrooms (whether in person or virtual).

Textbooks (or ebooks) and supplemental education materials are central to providing students with the content knowledge and practice experiences to support mastery of academic skills. Textbook developers spend significant time and effort to ensure that the content in those textbooks aligns to standards and provides students with the information and examples needed to understand key concepts. However, even with age-appropriate content and high-quality practice exercises, textbooks may not be effective as learning tools if they present and sequence information in a way that is not aligned to what we know about how people learn.

You may be wondering how much room there is for improvement—textbooks seem pretty good at delivering content as is, right? Actually, findings from three IES-funded projects demonstrate that there are multiple ways to improve texts and student understanding of key concepts. Here are a few of those ways:

Present a wide range of fraction practice problems. Textbooks focused on fractions learning tend to present more problems with equal denominators for addition and subtraction problems than for multiplication problems. Why does this matter? In IES-funded research, David Braithwaite and Bob Siegler showed that students pick up on this bias. As a result, students are more likely to make errors on equal denominator fractions multiplication problems because they are so used to seeing those problems when practicing fractions arithmetic and subtraction. The recommended minor change is to include a wider range of fractions practice problems, including equal denominator multiplication problems, to ensure that students do not form irrelevant associations between superficial features of a practice problem and the solution strategies they are practicing.

Provide students with a mix of practice problems that require different strategies rather than practice problems of the same type. Typical math practice involves solving the same type of problem repeatedly to practice the specific solution strategy a student just learned. However, across numerous IES-funded studies, Douglas Rohrer and his research team have shown that students benefit substantially more from math practice that involves a mix of problems that require different strategies (those learned in previous lessons mixed with those just learned). One of the major benefits of this approach is that students get practice choosing which strategy to use for a particular problem. Rohrer and his team found that across 13,505 practice problems from six popular math textbooks, only 9.7% of those problems were mixed up in this way. The recommended minor change is to simply mix up the problem sets so that students have more experiences encountering different types of problems in a single sitting.

Where and how you place visuals on textbook pages matters, especially when you want students to compare them. Textbooks typically use visuals such as diagrams and photos to help reinforce key concepts. In an IES-funded study, Bryan Matlen and colleagues examined anatomy and evolution chapters within three popular middle school science textbooks and found an average of 1.8 visuals per page. Students were expected to make comparisons using about a third of those visuals. Of those they had to compare, about half were positioned in suboptimal ways—that is, the images were not presented in a way that made it easy to identify how the elements of one image compare to the elements of the other. For example, imagine a student is asked to compare two x-ray images of hands to identify a bone that is missing from one of them. This task is much harder if one hand is shown upside down and the other is right-side up or perpendicular to the first image. Consistent with this example, Matlen and colleagues have conducted studies showing that visual comparisons are more effective when the features of the visuals that need to be compared are spatially aligned. The recommended minor change is to be intentional about the placement of visuals that students are supposed to be comparing; make sure they are placed in optimal alignment to each other so that it is easier for students to see how the features of one correspond to those of the other.

In sum, transformative, radical ideas about how to improve education are interesting to brainstorm about, but sometimes the key to improvement is identifying small changes that can deliver big results.

Written by Erin Higgins (Erin.Higgins@ed.gov), Program Officer for the Cognition and Student Learning Program, National Center for Education Research.