Inside IES Research

In honor of Autism Awareness Month, we would like to share an interview with Dr. Stephanie Shire about her Early Career Development and Mentoring project. Dr. Shire, associate professor of Early Childhood Special Education at the University of Oregon, focuses her current research on young children with autism and their families. In this interview, she discusses this project as well as her prior experiences in early intervention and special education and advice for other early career researchers.

Please tell us about your IES Early Career project.

My IES Early Career project is titled LIFT: Leveraging Autism Interventions for Families through Telehealth. The idea behind this project—exploring the technology-assisted delivery of an established evidence-based, in-person, one-on-one, caregiver-mediated social communication intervention—began even prior to the pandemic, before the field had to shift service delivery to online family-mediated services for young children with autism. The project focuses on helping caregivers use the existing intervention strategies to advance their children’s social communication and play skills. We’re not changing or testing the established intervention for the children, but rather the way in which we support caregivers in their learning.

The project is being conducted in partnership with early intervention and early childhood special education community practitioners and leaders. Our partners were fundamental in the development and revision of the online intervention program, which took an intervention manual of several hundred pages designed for clinicians and turned it into a series of brief online modules that families can read or listen to at their own pace. Our partners also shaped the implementation strategies that we are now testing in a pilot randomized trial. Families enrolled in the trial are being served by their local early intervention and early childhood special education practitioners in their home communities in Oregon.

How did you become interested in research on interventions to help young children with autism?

I was introduced to young children with autism as a high school student volunteering in a hospital playroom and as a special education classroom volunteer in my first year as an undergraduate student. In both cases, these preschool and school-age children had few or no words. I watched the practitioners try to connect and engage with the children with mixed success. I then spent the next several years as an undergraduate student working as an in-home intervention aide delivering services to young children with autism, many of whom had few or no words. I found myself failing to support the children’s progress, particularly with their communication skills. My desire to do more for these children prompted me to pursue additional resources and learn more about practices to better support them. This led me on a path to graduate school, first at the master’s level and then doctoral-level training, focused on intervention science to learn more about the development and testing of interventions to maximize communication development for young children with autism.

What do you find most rewarding about conducting research with young children with autism and their families?

Children and their families are at the heart of all my research team’s projects. Celebrating the moments when a child shows us a new idea in play, makes a joke, or points to something to share it with us lights up my entire lab! The greatest reward is seeing children shine and experience victories, big and small.

What are your next steps in this line of research?

We’re taking what we’re learning now, as well as the training that I’ve received in implementation science, to work on the next steps in this research project. We need to understand how to personalize implementation strategies for caregivers to help more families advance their children’s social communication skills through play and daily activities. Because this intervention has an adaptive component, we are now looking at combining sequences of supports for caregivers based on their individual progress halfway through implementing the intervention.

What advice do you have for other early career researchers?

Persist. In special education and early intervention, we are still acutely feeling the effects of the pandemic on a system that was already experiencing many challenges. There will be bumps along the way, but children show us every day that they can keep accomplishing small victories even in the face of obstacles. Let’s follow their lead and do the work in partnership with their caregivers and educators to keep building toward big victories for all children and their families. 

Thank you, Dr. Stephanie Shire, for sharing your early career research experience!

This blog was produced by Skyler Fesagaiga, a Virtual Student Federal Service intern for NCSER and graduate student at the University of California, San Diego. KatieTaylor, NCSER program officer, manages grants funded under The Early Career Development and Mentoring Program Program.

April is National Bilingual/Multilingual Learner Advocacy Month and Mathematics and Statistics Awareness Month. We asked Drs. Haiwen Chu and Leslie Hamburger, secondary mathematics researchers at the IES-funded National Research & Development Center to Improve Education for Secondary English Learners (EL R&D Center), to share how classroom observations are critical to analyzing and improving learning opportunities for English learners.

Could you tell us about your IES-funded project?

Haiwen: As part of the EL R&D Center portfolio of work, we developed RAMPUP, or Reimagining and Amplifying Mathematics Participation, Understanding, and Practices. RAMPUP is a summer bridge course for rising ninth graders. The three-week course is designed to challenge and support English learners to learn ambitious mathematics and generative language simultaneously. We will conduct a pilot study during summer 2024, with preliminary findings in fall 2024.

What motivated you to do this work?

Haiwen: English learners are frequently denied opportunities to engage in conceptually rich mathematics learning. We want to transform these patterns of low challenge and low support by offering a summer enrichment course that focuses on cross-cutting concepts uniting algebra, geometry, and statistics. We also designed active and engaged participation to be central to the development of ideas and practices in mathematics. English learners learn by talking and interacting with one another in ways that are both sustained and reciprocal.

Leslie: In addition, we wanted to offer broader approaches to developing language with English learners. As we have refined the summer program, we have explicitly built in meaningful opportunities for English learners to grow in their ability to describe, argue, and explain critical mathematics concepts in English This language development happens simultaneously with the development of conceptual understanding.

What have you observed among English learners so far in RAMPUP study classrooms?

Leslie: Over the past two summers, I have observed RAMPUP in two districts for two weeks total. The classrooms reflect America’s wide diversity, including refugee newcomers and students who were entirely educated in the United States. I was able to see both teachers facilitating and students learning. I observed how students developed diverse approaches to solving problems.

Through talk, students built upon each other’s ideas, offered details, and expanded descriptions of data distributions. Over time, their descriptions of data became more precise, as they attended to similarities and differences and developed labels. I also observed how teachers assisted students by giving hints without telling them what to do.

Haiwen: As we observed, we wanted to understand how English learners engaged in the activities we had designed, as well as how their conceptual understandings and language developed simultaneously. I have spent two summers immersed in three districts over seven weeks with diverse students as they developed relationships, deep understandings, and language practices.

I was honestly surprised by the complex relationships between how students wrote and the development of their ideas and language. Sometimes, students wrote to collect their thoughts, which they then shared orally with others, to collectively compose a common way to describe a pattern. Other times, writing was a way to reflect and give each other feedback on what was working well and how peers could improve their work. Writing was also multi-representational as students incorporated diagrams, tables, and other representations as they wrote.

From closely observing students as they wrote, I also gained valuable insight into how they think. For example, they often looked back at their past work and then went on to write, stretching their understanding.

Why are your observations important to your project?

Haiwen: RAMPUP is an iterative design and development project: our observations were driven by descriptive questions (how students learned) and improvement questions (how to refine activities and materials). By observing each summer what worked well for students, and what fell flat, we have been able to iteratively improve the flow and sequencing of activities.

We have learned that observations matter most when they directly inform broader, ongoing efforts at quality learning.

Now, in our final phase, we are working to incorporate educative examples of what quality interactions looked and sounded like to enhance the teacher materials. Beyond the shorter episodes confined within a class period, we are also describing patterns of growth over time, including vignettes and portfolios of sample student work.

Leslie: Indeed, I think that wisdom comes both in practice and learning by looking back on practice. Our observations will enable teachers to better anticipate what approaches their students might take. Our educative materials will offer teachers a variety of real-life approaches that actual students similar to their own may take. This deep pedagogical knowledge includes knowing when, if, and how to intervene to give the just-right hints.

To receive regular updates and findings from the Center, as well as webinar and conference opportunities, subscribe to Where the Evidence Leads newsletter.

This blog was produced by Helyn Kim (Helyn.Kim@ed.gov), program officer for the Policies, Practices, and Programs to Support English Learners portfolio at NCER.

Every April, we observe Mathematics and Statistics Awareness month to increase public understanding of math and stats and to celebrate the unique role that math and stats play in solving critical real-world problems. In that spirit, we want to share some exciting progress that SEERNet has made in supporting math education research over the past three years.

In 2021, IES established SEERNet, a network of platform developers, researchers, and education stakeholders, to create and expand the capacity of digital learning platforms (DLPs) to enable equity-focused and rigorous education research at scale. Since then, SEERNet has made significant progress, and we are starting to see examples of how researchers can use this new research infrastructure.

Recently, IES held two rounds of a competition to identify research teams to join SEERNet to conduct a study or series of studies using one of the five DLPs within the SEERNet network. Two research teams joined the network from the first round, and the second round of applications are now under review. We want to highlight the two research teams that joined SEERNet and the important questions about math education that they are addressing.

• Now I See It: Supporting Flexible Problem Solving in Mathematics through Perceptual Scaffolding in ASSISTments – Dr. Avery Closser and her team are working with the E-Trials/ASSISTments team. ASSISTments is a free tool to support math learning, which has been used by over 1 million students and 30,000 teachers across the nation. IES has supported its development and efficacy since 2003. E-Trials is the tool that researchers can use to develop studies to be implemented within ASSISTments. The research team’s studies are designed to test whether perceptual scaffolding in mathematics notation (for example, using color to highlight key terms such as the inverse operators in an expression) leads learners to pause and notice structural patterns and ultimately practice more flexible and efficient problem solving. This project will yield evidence on how, when, and for whom perceptual scaffolding works to inform classroom practice, which has implications for the development of materials for digital learning platforms.
• Investigating the Impact of Metacognitive Supports on Students' Mathematics Knowledge and Motivation in MATHia – Dr. Cristina Zepeda and her team are working with the Upgrade/MATHia team. MATHia is an adaptive software program used in middle and high schools across the country. UpGrade is an open-source A/B testing platform that facilitates randomized experiments within educational software, including MATHia. The research team will conduct a series of studies focused on supporting students’ metacognitive skills, which are essential for learning in mathematics but not typically integrated into instruction. The studies will seek to identify supports that can be implemented during mathematics learning in MATHia that improve metacognition, mathematics knowledge, and motivation in middle school.

Both research teams are conducting studies that will have clear implications for curriculum design within DLPs focused on math instruction for K-12 students. The value of conducting these studies through existing DLPs rather than through individual researcher-designed tools and methods includes—

1. Time and cost savings – Without the need to create materials from scratch, the research teams can immediately get to work on the specific instructional features they intend to test. Additionally, since the intervention and pre/post assessments can be administered through the online tool, the need to travel to study sites is reduced.
2. Access to large sample sizes – Studies like the ones described above are frequently administered in laboratory settings or in a handful of schools. Since over 100k students use these DLPs, there is the potential to recruit a larger and more diverse sample of students for studies. This provides more opportunities to study what works for whom under what conditions.
3. Tighter feedback loops between developers and researchers – Because the research teams need to work directly with the platform developers to administer their studies, the studies need to be designed in ways that will work within the platform and with the platform content. This ensures their relevance to the platform and means that the platform developers will be knowledgeable about what is being tested. They will be interested to hear the study’s findings and likely to use that information to inform future design decisions.

We look forward to seeing how other education researchers take advantage of this new research infrastructure. For math education researchers in particular, we hope these two example projects inspire you to consider how you might use a DLP in the future to address critical questions for math education.

This blog was written by Erin Higgins (Erin.Higgins@ed.gov), Program Officer, Accelerate, Transform, Scale Initiative.

In honor of Developmental Disabilities Month, NCSER would like to highlight research that supports young children with complex communication needs. Many children with disabilities, including those with autism and other developmental disabilities, may be described as having complex communication needs because they are unable to use speech to meet their needs in daily interactions. Augmentative and alternative communication (AAC) systems provide such individuals a way to communicate that does not require vocal speech. Examples include low-tech systems like manual signs or picture cards and high-tech systems like electronic speech generating devices. For non-speaking children, access to AAC is critical for expressing their needs and wants, developing relationships, and participating in academic instruction. In school settings, paraeducators work frequently with students to support their communication needs.

With a NCSER-funded grant, Dr. Sarah Douglas (Michigan State University) has been developing and piloting an online training program, the POWR System, for paraeducators and their supervising teachers to improve communication skills of children with complex communication needs. We recently caught up with Dr. Douglas to learn more about the POWR System, what led her to conduct this research, and future directions.

What inspired you to conduct this research?

My exposure to children who use AAC began when I was a child myself. In elementary school, a new school was built in my neighborhood. Unlike other schools during the late 80s and early 90s, this school had special education rooms at the center of the school. Each time I went to various activities around school, the children were visible. The teacher in the classroom for children with extensive support needs, Mrs. Smith, was an advocate for inclusion and socialization for her students so each of the children spent time in general education classrooms. She began inviting general education students to spend recess in her classroom playing games and cooking with students. I took her up on this offer and got to interact with them while they used their AAC. I learned that communication could come in many forms—not just through speech. These early experiences led me to become a special education teacher supporting children with complex communication needs. In that class I worked with a lot of paraeducators. When I pursued my PhD, I focused on paraeducators and AAC. My dissertation topic laid the foundation for this NCSER grant project. During my dissertation I implemented an intervention to teach paraeducators how to best support children who use AAC. So, I guess you could say this has been something I’ve been working on for decades. 😊

What do the results from your research say about communication outcomes for young children with complex communication needs? What are the outcomes for educators that support student communication?

We’ve learned so much from this work. Findings from our study indicate that, for children who use AAC, the kinds of support and communication opportunities that paraprofessionals provide really matter. Providing meaningful, motivating opportunities to communicate is critical for young children who use AAC. One of our studies highlighted that young children who use AAC are most likely to respond after being provided with a choice or a question. These results suggest that certain types of supports make it more clear to young children that a response from them is expected. We also learned that waiting for them to communicate is critical. Generally, 5-7 seconds is sufficient wait time, but for children who have motor challenges more time is likely necessary. Also, paraeducators modeling the use of an AAC device can be really supportive, as our research found that children were more likely to communicate after a model of AAC by paraeducators. We all need models when we are learning new skills and children who use AAC are no different. We also learned that most paraeducators we worked with were very responsive to child communication, so teachers should continue to support and encourage that. Teachers can provide great supervision and support to paraeducators as they implement AAC strategies.

Based on these results, what are the implications for practice and policy?

Districts could do more to support teachers in knowing how to oversee and provide feedback to paraeducators. Not all teachers were comfortable with this role at first. We also feel strongly that, based on this work, more team members should be involved in interventions focused on AAC strategies. Perhaps the teacher and paraeducator are the main implementers, but speech-language pathologists (SLPs) and district level personnel have important roles in supporting this work and should also be involved in understanding these interventions and guiding implementation.

What are the next steps in your research on AAC for children with developmental disabilities?

We continue to do a lot of work to know how to best support child communication through communication partners such as siblings, parents, SLPs, teachers, and paraeducators. We recently obtained a new grant from IES to develop a professional development and training intervention for school-based SLPs to support family member implementation of communication strategies with children who use AAC. We are really excited about this project. It is only the first year, but we already have most of the intervention developed and are conducting focus groups with SLPs and family members to get feedback and make revisions.

How can educators find more information about the POWR system and implementing augmentative and alternative communication (AAC) systems in their classrooms?

The intervention is available and can be accessed by reaching out to me at sdouglas@msu.edu.

Is there anything else you would like to add?

I am just so grateful for the early experiences I had that led me to this important work and excited to support all the children, families, and educational teams.

A special thanks to Dr. Douglas and the POWR research team for all their hard work supporting communication for students using AAC. We look forward to seeing the impact your current project will have on the field!

This blog was written by Shanna Bodenhamer, virtual student federal service intern at NCSER and doctoral candidate at Texas A&M University. Emily Weaver, NCSER PO, monitors a portfolio of grants that covers both paraeducators and students with autism.

For nearly 15 years, NCER has supported the development and improvement of innovative methodological and statistical tools and approaches that will better enable applied education researchers to conduct high-quality, rigorous education research. This blog spotlights the work of Andrew Gelman, a professor of statistics and political science at Columbia University, and Sophia Rabe-Hesketh, a professor of statistics at the School of Education at the University of California, Berkeley. IES has supported their research on hierarchical modeling and Bayesian computation has for many years. In this interview blog, Drs. Gelman and Rabe-Hesketh reflect on how Bayesian modeling applies to educational data and describe the general principles and advantages of Bayesian analysis.

What motivates your research on hierarchical modeling and Bayesian computation?

Education data can be messy. We need to adjust for covariates in experiments and observational studies, and we need to be able to generalize from non-random, non-representative samples to populations of interest.

The general motivation for multilevel modeling is that we are interested in local parameters, such as public opinion by states, small-area disease incidence rates, individual performance in sports, school-district-level learning loss, and other quantities that vary among people, across locations, and over time. In non-Bayesian settings, the local parameters are called random effects, varying intercepts/slopes, or latent variables.

Bayesian and non-Bayesian models differ in how completely the researcher using them must specify the probability distributions of the parameters. In non-Bayesian models, typically only the data model (also called the likelihood function) must be specified. The underlying parameters, such as the variances of random intercepts, are treated as unknown constants. On the other hand, the Bayesian approach requires specifying a full probability model for all parameters.  

A researcher using Bayesian inference encodes additional assumptions about all parameters into prior distributions, then combines information about the parameters from the data model with information from the prior distributions. This results in a posterior distribution for each parameter, which, compared to non-Bayesian model results, provides more information about the appropriateness of the model and supports more complex inferences.

What advantages are there to the Bayesian approach?

Compared to other estimates, Bayesian estimates are based on many more assumptions. One advantage of this is greater stability at small sample sizes. Another advantage is that Bayesian modeling can be used to produce flexible, practice-relevant summaries from a fitted model that other approaches cannot produce. For instance, when modeling school effectiveness, researchers using the Bayesian approach can rely on the full probability model to justifiably obtain the rankings of schools or the probabilities that COVID-related declines in NAEP mean test scores for a district or state have exceeded three points, along with estimates for the variability of these summaries. 

Further, Bayesian inference supports generalizability and replicability by freely allowing uncertainty from multiple sources to be integrated into models. Without allowing for uncertainty, it’s difficult to understand what works for whom and why. A familiar example is predicting student grades in college courses. A regression model can be fit to obtain a forecast with uncertainty based on past data on the students, and then this can be combined with student-specific information. Uncertainties in the forecasts for individual students or groups of students will be dependent and can be captured by a joint probability model, as implemented by posterior simulations. This contrasts with likelihood-based (non-Bayesian) inference where predictions and their uncertainty are typically considered only conditionally on the model parameters, with maximum likelihood estimates plugged in. Ignoring uncertainty leads to standard error estimates that are too small on average (see this introduction to Bayesian multilevel regression for a detailed demonstration and discussion of this phenomenon).

What’s an important disadvantage to the Bayesian approach?

Specifying a Bayesian model requires the user to make more decisions than specifying a non-Bayesian model. Until recently, many of these decisions had to be implemented using custom programming, so the Bayesian approach had a steep learning curve. Users who were not up to the programming and debugging task had to work within some restricted class of models that had already been set up with existing software. 

This disadvantage is especially challenging in education research, where we often need to adapt and expand our models beyond a restricted class to deal with statistical challenges such as imperfect treatment assignments, nonlinear relations, spatial correlations, and mixtures, along with data issues such as missingness, students changing schools, guessing on tests, and predictors measured with error.

How did your IES-funded work address this disadvantage?

In 2011, we developed Stan, our open-source Bayesian software, with funding from a Department of Energy grant on large-scale computing. With additional support from the National Science Foundation and IES, we have developed model types, workflows, and case studies for education researchers and also improved Stan’s computational efficiency.

By combining a state-of-the-art inference engine with an expressive modeling language, Stan allows education researchers to build their own models, starting with basic linear and logistic regressions and then adding components of variation and uncertainty and expanding as needed to capture challenges that arise in applied problems at hand.  We recommend the use of Stan as part of a Bayesian workflow of model building, checking, and expansion, making use of graphs of data and fitted models.

Stan can be accessed using R, Python, Stata, Julia, and other software. We recommend getting started by looking at the Stan case studies. We also have a page on Stan for education research and a YouTube channel.

In terms of dealing with the issues that arise in complex educational data, where do we stand today?

Put all this together, and we are in the business of fitting complex models in an open-ended space that goes beyond any existing menu of statistical procedures. Bayesian inference is a flexible way to fit such models, and Stan is a flexible tool that we have developed, allowing general models to be fit in reasonable time using advanced algorithms for statistical computing.  As always with research, there are many loose ends and there is more work to be done, but we can now routinely fit, check, and display models of much greater generality than was before possible, facilitating the goals of understanding processes in education.

This blog was produced by Charles Laurin (Charles.Laurin@ed.gov), NCER program officer for the Statistical and Research Methodology in Education grant program.