Inside IES Research

Notes from NCER & NCSER

Addressing COVID-19’s Disruption of Student Assessment

Under an IES grant, the RAND Corporation, in collaboration with NWEA, is developing strategies for schools and districts to address the impacts of COVID-19 disruptions on student assessment programs. The goal is to provide empirical evidence of the strengths and limitations of strategies for making decisions in the absence of assessment data. Jonathan Schweig, Andrew McEachin, and Megan Kuhfeld describe early findings from surveys and structured interviews regarding key concerns of districts and schools. 

 

As a first step, we surveyed assessment and research coordinators from 23 school districts (from a sample of 100 districts) and completed follow-up interviews with seven of them on a variety of topics, including the re-entry scenario for their district, the planning activities that they were not able to perform this year due to coronavirus-based disruptions to spring 2020 assessments, and the strategies they were employing to support instructional planning in the absence of assessment data. While the research is preliminary and the sample of respondents is not nationally representative, the survey and interview responses identified two key concerns arising from the lack of spring 2020 assessment data which has made it challenging to examine student or school status and change over time, especially as COVID-19 has differential impacts on student subgroups:

 

  • Making course placement decisions. Administrators typically rely on spring assessment scores—often in conjunction with other assessment information, course grades, and teacher recommendations—to make determinations for course placements, such as who should enroll in accelerated or advanced mathematics classes. 
  • Evaluating programs or district-wide initiatives. Many districts monitor the success of these programs internally by looking at year-to-year change or growth for schools or subgroups of interest. 

 

How are school systems responding to these challenges? Not surprisingly, the responses vary depending on local contexts and resources. Where online assessments were not feasible in spring 2020, some school districts used older testing data to make course recommendations, either from the winter or from the previous school year. Some districts relaxed typical practice and provided more autonomy to individual schools, relying on school staff to exercise local judgment around course placements and using metrics like grades and teacher recommendations. Other districts reported projecting student scores based on student assessment histories. Relatedly, some districts were already prepared for this decision because they had recently experienced difficulties with adopting an online assessment system and had to address similar problems caused by large numbers of missing or invalid tests.

 

School districts also raised concerns about whether assessments administered during the 2020-21 school year would be valid and comparable so that they could be used in student placement and program evaluation decisions. These concerns included the following:

  • Several respondents raised concerns about the trustworthiness of remote assessment data collected this fall and the extent to which results could be interpreted as valid indicators of student achievement or understanding.
  • Particularly for districts that started the 2020-21 school year remotely, respondents were concerned about student engagement and motivation and the possibility of students rushing assessments, running into technological or internet barriers, or seeking assistance from guardians or other resources. 
  • Respondents raised questions about the extent to which available assessment scores are representative of school or district performance as a whole. Given that vulnerable students (for example, students with disabilities, students experiencing homelessness) may be the least likely to have access to remote instruction and assessments, it is likely that the students who are not assessed this year are different from students who are able to be assessed.
  • Other respondents noted that they encountered resistance from parents around fall assessment because they prioritized student well-being (for example, safety, sense of community, and social and emotional well-being) more so than academics. This is a perspective that resonates with recent findings from a nationally representative sample of teachers and school leaders drawn from RAND’s American Educator Panel (AEP).

 

In the next phase of the work, the research team plans to:

  • Conduct a series of simulation and empirical studies regarding the most common strategies that the district respondents indicated they were using to make course placement decisions and to evaluate programs or district-wide initiatives.
  • Provide a framework to help guide local research on the intended (and unintended) consequences for school and school system decision making when standardized test scores are not available.

 

We welcome individuals to reach out to RAND with additional recommendations or considerations. We are also interested in hearing how districts are approaching course placement, accountability, and program evaluation across the country. Connect with the research team via email at jschweig@rand.org.

 


Jonathan Schweig is a social scientist at the nonprofit, nonpartisan RAND Corporation.

Andrew McEachin is a senior policy researcher at the nonprofit, nonpartisan RAND Corporation.

Megan Kuhfeld is a researcher at NWEA.

From Fellow to Funded: Former IES Postdoctoral Fellows Funded as Principal Investigators

As part of our Spotlight on IES Training Programs series, IES is proud to showcase five former IES postdoctoral fellows who are now principal investigators for grants funded in FY 2020. The goal of the NCER and NCSER postdoctoral training programs is to prepare scholars to conduct rigorous, relevant education and special education research. As the following examples demonstrate, IES fellows are contributing to evidence-based education in a wide range of academic domains and are addressing the needs of students, teachers, and families through their innovative measurement, exploratory, development, and evaluation work.

 

Dr. Crystal Bishop (IES Fellow at the University of Florida until 2016) will lead Tools for Families. This project will develop and pilot test a new component for an existing intervention that aims to improve outcomes for young children with disabilities in preschool programs. The existing program is called Evaluating Embedded Instruction for Early Learning (EIEL) and already includes tools to help teachers. In this new study, Dr. Bishop will create an additional component that helps teachers engage students’ families in implementing EIEL strategies.

 

Dr. Joseph Nese (IES Fellow at the University of Oregon until 2011) will lead A Comprehensive Measure of Reading Fluency: Uniting and Scaling Accuracy, Rate, and Prosody. This project aims to develop and validate an automated scoring system of oral reading fluency for students in grades 2 to 4 to better identify students in need of reading interventions and better evaluate reading interventions and builds off a previous grant Dr. Nese received as PI, Measuring Oral Reading Fluency: Computerized Oral Reading Evaluation (CORE) (R305A140203).

 

Dr. David Purpura (IES Fellow at the University of Illinois until 2012) will lead Reading and Playing With Math: Promoting Preschoolers' Math Language Through Picture Books and Play Activities. This program will develop, refine, and evaluate a new math language intervention, Reading and Playing with Math (RP-Math). RP-Math will leverage the language instruction using storybooks and mathematics instruction.

 

Dr. Rachel Rosen (IES Fellow at the University of Michigan until 2014) will lead Choice and Information: The Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance. This project will evaluate  two widely used technology-based career advising tools for secondary school students, Naviance and YouScience, to see whether and how these tools influence student thinking about career options, career and technical education (CTE) coursework and work-based learning options, and decisions about CTE pathways and programs of study.

 

Dr. Candace Walkington (IES Fellow at the University of Wisconsin, Madison until 2013) will lead Exploring Collaborative Embodiment for Learning (EXCEL): Understanding Geometry Through Multiple Modalities. This program will explore how different types multisensory experiences and modes of collaboration affect students' geometric reasoning. The researchers will leverage augmented reality (AR) technology to see if different ways of engaging with content (such as holograms, tablet-based, or paper-based images) lead to different learning outcomes.

 


This blog was written by Shirley Liu, virtual intern and an English/Anthropology & Sociology double major at Lafayette College, and Dr. Meredith Larson, program officer for NCER postdoctoral training.

 

Calling All K-12 Students: NASA’s Artemis Program Invites You to Imagine Living on the Moon

Through its Artemis program, NASA will land the first woman and next man on the Moon by 2024, using innovative technologies to explore more of the lunar surface than ever before. NASA is collaborating with commercial and international partners to establish sustainable exploration by the end of the decade and to apply what is learned to take the next giant leap—sending astronauts to Mars.

 

K-12 Artemis Moon Pod Student Essay Contest

 

 

NASA is inviting students in kindergarten through Grade 12 to join the Artemis adventure.  Through its challenge, students can imagine “what it might be like if you were living with a pod of astronauts 250,000 miles from Earth.”

In the challenge, students write essays focusing on leading a one-week expedition at the Moon’s South Pole. Plans and details of the expedition should consider the types of skills, attributes, and personality traits of their Moon Pod crew, and one machine, robot, or technology that will be left on the lunar surface to help future astronauts explore the Moon.

Three levels of challenges are being held for students in Grades K-4, 5-8, and 9-12. Every student who submits an entry will receive a certificate from NASA and be invited to a special NASA virtual event—with an astronaut! For all entry requirements and judging criteria, please read the rules.  Students and teachers can sign up and submit their entry at the contest site. Even if you are not a student you can still participate. U.S. residents over the age of 18 can apply to be judges for the contest to help NASA make their selection.

The essay competition is being managed by a web-based platform developed by Future Engineers based in Burbank, California.  This platform was created with the support of a 2017 award from the IES Small Business Innovation Research program (ED/IES SBIR).  Future Engineers built this platform to be an online hub for classrooms and educators to access free, project-based STEM activities and to provide a portal where students submit and compete in different kinds of maker and innovation challenges across the country.  The Artemis essay contest follows the Mars 2020 “Name the Rover” contest, which was also managed by Future Engineers. (See the recap of that challenge here.)

 

Stay tuned for the winning essays in the months to come!

 


Edward Metz (Edward.Metz@ed.gov) is a research scientist at the Institute of Education Sciences at the US Department of Education and Program Manager of ED/IES SBIR.

Katherine Brown is the lead communication specialist for the Office of STEM Engagement at NASA.

 

About ED/IES SBIR

The U.S. Department of Education’s Small Business Innovation Research program, administered by the Institute of Education Sciences (IES), funds projects to develop education technology products designed to support students, teachers, or administrators in general or special education. The program emphasizes rigorous and relevant research to inform iterative development and to evaluate whether fully developed products show promise for leading to the intended outcomes. The program also focuses on commercialization once the award period ends so that products can reach students and teachers and be sustained over time. ED/IES SBIR-supported products are currently used by millions of students in thousands of schools around the country.

 

About NASA’s Office of STEM Engagement

NASA’s journeys have propelled technological breakthroughs, pushed the frontiers of scientific research, and expanded our understanding of the universe. These accomplishments, and those to come, share a common genesis: education in science, technology, engineering, and math. NASA’s Office of STEM Engagement strives to create unique opportunities for a diverse set of students to contribute to NASA’s work in exploration and discovery; build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA’s people, content and facilities; and attract diverse groups of students to STEM through learning opportunities that spark interest and provide connections to NASA’s mission and work. To achieve these goals, NASA’s Office of STEM Engagement strives to inspire the next generation to discover their way to a new era of American innovation and explore further into space than ever before.

 

Webinar Recap: EdTech Resources for Special Education Practitioners

It goes without saying the COVID19 pandemic has and continues to have a profound effect on education. Students are adjusting to hybrid or fully remote learning, and educators are continuing to make complex decisions about how best to support students in the new normal.

On October 28, 2020, InnovateEDU and the Educating All Learners Alliance hosted a webinar focused on education technology resources for special education. More than 1,100 practitioners joined the event in real-time.

 

 

The webinar featured video demonstrations of five special education technology tools that were developed through the IES Small Business Innovation Research Program and ED’s Office of Special Education Educational Technology, Media, and Materials for Individuals with Disabilities Program. The event also included conversations with special education practitioners and researchers who provided perspectives on the role of special education and technology to meet the needs of all students. The webinar involved a variety of resources and opportunities, including:

 

During the webinar, practitioners participated by adding comments in the chat box with a “wish list” of education technology they would like to have now to support teaching and learning. Participants entered dozens of responses, many calling for increased connectivity and access to hardware and software, especially in rural areas. Other responses focused on education technologies for teachers, students with or at-risk for disabilities, and parents and caregivers.

Following are just a few of the entries:

 

For Teachers

  • “More coaching tools to use with children who are learning remotely to provide instantaneous feedback”
  • “Descriptions that allow teachers to at-a-glance identify the features a program offers to match to the features that their students need”
  • “Using data to support teachers and students with decisions that move learning forward.”
  • “Resources that I can use to assist with non-compliant behaviors and keeping their attention in person and virtually.”
  • Making it possible for students to show their work for math so that we can see that rather than just their answers.”
  • “Common share place for all teachers.”
  • “I am looking for a way to deliver instructions to the home distantly”

 

For Students with Disabilities

  • “Teaching students how to be self-determined learners.”
  • “Build this skill set from kindergarten.”
  • “Develop and implement collaborative activities”
  • “My nonverbal students need hands on.”
  • “Engagement and motivation; remote resources.”
  • “Student choice and voice.”

 

For Parents

  • “Make it a family affair / Zoom with family member supporting on other side.”
  • “A resource that we can use to incorporate the parent or group home worker that have to navigate these different learning apps for the student.”
  • “Easy-to-follow videos that we can use to show parents and students how to use these resources when they aren’t in front of us.”

 

Lastly, one of the teachers provided a comment: “We need more of these events.”  From everyone involved in the October 28 webinar, thanks for attending. We are planning for more events like this one soon.

 


Edward Metz (Edward.Metz@ed.gov) is a research scientist at the Institute of Education Sciences in the US Department of Education.

Tara Courchaine (Tara.Courchaine@ed.gov) is a program officer at the Office of Special Education Programs in the US Department of Education.

Spotlight on IES Training Programs: Introduction to a Blog Series

Since 2004, IES has been preparing researchers to conduct high-quality, rigorous education and special education research through training grant programs. This roughly $281 million investment has helped change universities and departments across the nation and supported the training of over 200 students interested in beginning doctoral programs, nearly 1000 doctoral students, over 280 postdoctoral fellows, and hundreds of practicing researchers at universities, research firms, state and local agencies, and other organizations.

Over the months to come, we will be spotlighting these IES training programs and those who have participated in them. This blog series will include interviews, updates, and program descriptions as we learn more about the research, innovations, and careers of IES training program participants.

 

Join us as we celebrate the possibilities created by the following IES training programs:


For more information about the NCER training programs, contact Dr. Katina Stapleton, and for information about NCSER training programs, contact Dr. Katie Taylor.

This blog was written by Dr. Meredith Larson, program officer for NCER Postdoctoral Research Training grants, and is the first in an ongoing series: Spotlight on IES Training Programs.