Inside IES Research

Notes from NCER & NCSER

Assessing Math Understanding of Students with Disabilities During a Pandemic

For almost two decades, IES/NCSER has funded Brian Bottge and his teams at the University of Kentucky and University of Wisconsin-Madison to develop and test the efficacy of a teaching method called Enhanced Anchored Instruction (EAI), which helps low-achieving middle school students with math disabilities develop their problem-solving skills by solving meaningful problems related to a real-world problem. The research findings support the efficacy of EAI, especially for students with math disabilities. Most recently, Bottge and his team have been researching innovative forms of assessment that more adequately capture what students with disabilities know both conceptually and procedurally in solving math problems. With supplemental funding, IES/NCSER extended Dr. Bottge’s latest grant to test the use of oral assessment to measure student knowledge and compare that with the knowledge demonstrated on a pencil and paper test. The COVID-19 pandemic introduced added challenges to this work when schools closed and students shifted to online education.

Below we share a recent conversation with Dr. Bottge about the experience of conducting research during a pandemic and what he and his team were still able to learn about the value of oral assessment in mathematics for students with disabilities.

What changes did you observe in the intervention implementation by teachers due to the COVID-related shift to online learning?

Photo of Dr. Brian Bottge

The shift to online learning created changes in class size and structure. For 38 days (22 days in classroom, 16 days online through a virtual meeting platform), the middle school special education teacher first taught concepts through a widely used video-based anchored problem, the Kim’s Komet episode of the Jasper Project, in which characters compete in a “Grand Pentathlon.” The teacher then engaged the students in a hands-on application of the concepts by running a live Grand Pentathlon. In the Grand Pentathlon, students make their own cars, race them on a full-size ramp, time them at various release points on the ramp, and graph the information to estimate the speed of the cars. The purpose of both units was to help students develop their informal understanding of pre-algebraic concepts such as linear function, line of best fit, variables, rate of change (slope), reliability, and measurement error. Midway through the study, in-person instruction was suspended and moved online. Instead of working with groups of three to four students in the resource room throughout the day, the teacher provided online instruction to 14 students at one time and scheduled one-on-one sessions with students who needed extra help.

What challenges did you observe in the students interacting with the activities and their learning once they shifted to online learning?

All students had access to a computer at home and they were able to use the online platform without much confusion because they had used it in other classes. The screen share feature enabled students to interact with much of the curriculum by viewing the activities, listening to the teacher, and responding to questions, although they could not fully participate in the hands-on part of the lessons. Class attendance and student behavior were unexpectedly positive during the days when students were online. For example, one student had displayed frequent behavioral outbursts in school but became a positive and contributing member of the online class. The ability to mute mics in the platform gave the teacher the option of allowing only one student to talk at a time.

Were students still able to participate in the hands-on activities that are part of the intervention?

For the hands-on activities related to the Grand Pentathlon competition, the teacher taught online and a research staff member manipulated the cars, track, and electronic timers from campus. Students watched their computer screens waiting for their turn to time their cars over the length of the straightaway. The staff member handled each student’s cars and one by one released them from the height on the ramp as indicated by each student. After students had recorded the times, the teacher asked students to calculate and share the speeds of their cars for each time trial height.

Do you have any other observations about the impact of COVID-19 on your intervention implementation?

One of the most interesting observations was parent participation in the lessons. Several parents went beyond simply monitoring how their child was doing during the units to actively working out the problems. Some were surprised by the difficulty level of the math problems. One mother jokingly remarked: I thought the math they were going to do was as easy as 5 + 5 = 10. The next time my son might have to be the parent and I might have to be the student. You all make the kids think and I like that.

When COVID-19 shut down your participating schools, how were you able to adjust your data collection to continue with your research?

We used the same problem-solving test that we have administered in several previous studies (Figure 1 shows two of the items). On Day 1 of the study (pre-COVID), students took the math pretest in their resource rooms with pencil and paper. Due to COVID-19 school closures, we mailed the posttest and test administration instructions to student homes. On the scheduled testing day during an online class session, students removed the test from the envelope and followed directions for answering the test questions while we observed remotely. On Days 2 and 3 of the study (pre-COVID), an oral examiner (OE) pretested individual students in person. The OE asked the student questions, prompting the student to describe the overall problem, identify the information needed for solving the problem, indicate how the information related to their problem-solving plan, and provide an answer. Due to COVID-19, students took the oral posttests online. The teacher set up a breakout room in the platform where the OE conducted the oral assessments and a second member of the research team took notes.

A picture depicting two sample questions. The first shows a graph of two running paths along with the text, "3. The total distance covered by two runners is shown in the graph below. a. How much time did it take runner 1 to go 1 mile? b. About how much time after the start of the race did one runner pass the other?" The second image features a marble on top of a ramp accompanied with the question "What is the speed of a marble (feet per second) let go from the top of the ramp? (Round your answer to the nearest tenth.)"Figure 1. Sample Items from the Problem-Solving Test

During the testing sessions, the OE projected each item on the students’ computer screens. Then she asked the student to read the problem aloud and describe how to solve it. The OE used the same problem-solving prompts as was used on the pretests. For problems that involved graphs or charts, the OE used the editing tools to make notations on the screen as the students directed. One challenge is that oral testing online made it more difficult to monitor behavior and keep students on task. For example, sometimes students became distracted and talked to other people in their house.

What were the results of this study of oral assessment in mathematics for students with disabilities?

Our results suggest that allowing students to describe their understanding of problems in multiple ways yielded depth and detail to their answers. We learned from the oral assessment that most students knew how to transfer the data from the table to an approximate location on the graph; however, there was a lack of precision due to a weak understanding of decimals. For item 4 in Figure 1, the use of decimals confused students who did not have much exposure to decimals prior to or during the study. We also found that graphics that were meant to help students understand the text-based items were in some cases misleading. The representation in item 4 was different than the actual ramp and model car activity students experienced virtually. We have used this math test several times in our research and regrettably had no idea that elements of the graphics contributed to misunderstanding.

Unfortunately, our findings suggest that the changes made in response to COVID-19 may have depressed student understanding. Performances on two items (including item 4 in Figure 1) that assessed the main points of the intervention were disappointing compared to results from prior studies. The increase in class size from 3–4 to 14 after COVID and switching to online learning may have reduced the opportunity for repetition and practice. There were reduced opportunities for students to participate in the hands-on activities and participate in conversations about their thinking with other students.

We acknowledge the limitations of this small pilot study to compare knowledge of students when assessed in a pencil and paper format to an oral assessment. We are optimistic about the potential of oral assessments to reveal problem-solving insights of students with math disabilities. The information gained from oral assessment is of value if teachers use it to individualize their instruction. As we learned, oral assessment can also point to areas where graphics or other information are misleading. More research is needed to understand the value of oral assessment despite the increase in time it might add to data collection efforts for students with math disabilities. This experience highlights some of the positive experiences of students learning during COVID-19 virtually at home as well as some of the challenges and risks of reduced outcomes from these virtual learning experiences, especially for students with disabilities.

This blog was written by Sarah Brasiel, program officer for NCSER’s Science, Technology, Engineering, and Math program.

Cost Analysis in Practice: Resources for Cost Analysis Studies

IES supports rigorous research that can provide scientific evidence on how best to address our nation’s most pressing education needs. As part of the Standards for Excellence in Education Research (SEER) principles, IES-funded researchers are encouraged, and in some cases required, to conduct a cost analysis for their projects with the intended goal of supporting education agencies’ decision-making around the adoption of programs, policies, or practices. 

 

The Cost Analysis in Practice (CAP) Project is a 3-year initiative funded by IES to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. This support includes the following freely available resources.

  • Resources developed by the CAP Project
    • Introductory resources on cost analysis including Standards and Guidelines 1.1, an infographic, a video lecture, and FAQs.
    • Tools for planning your cost analysis, collecting and analyzing cost data, and reporting your results.
    • A Help Desk for you to submit inquiries about conducting a cost analysis with a response from a member of the CAP Project Team within two business days.
  • Other resources recommended by the CAP Project
    • Background materials on cost analysis
    • Guidance on carrying out a cost analysis
    • Standards for the Economic Evaluation of Educational and Social Programs
    • Cost analysis software

 

The CAP Project is also involved in longer-term collaborations with IES-funded evaluation projects to better understand their cost analysis needs. As part of this work, the CAP Project will be producing a set of three blogs to discuss practical details regarding cost studies based on its collaboration with a replication project evaluating an intervention that integrates literacy instruction into the teaching of American history. These blogs will discuss the following:

  • Common cost analysis challenges that researchers encounter and recommendations to address them
  • The development of a timeline resource for planning a cost study
  • Data collection for a cost study

 

The CAP Project is interested in your feedback on any of the CAP Project resources and welcomes suggestions for additional resources to support cost analysis. If you have any feedback, please fill out a suggestion form at the bottom of the Resources web page.

CTE Research Is Flourishing at IES!

Since its inception in 2017, the CTE portfolio in the National Center for Education Research (NCER) at IES has grown to 11 research grants and a research network! Several other CTE-related grants have been funded under other topics, such as “Postsecondary/Adult Education” and “Improving Education Systems” in the education research grants program, and in other grant programs such as “Using SLDS to Support State Policymaking.” Two CTE-related grants under the latter program were awarded in FY21—

The newest grants funded in FY21 in the CTE topic of the Education Research Grants program include—

As a causal impact study, the last project (on Virtual Enterprises) has been invited to join NCER’s CTE Research Network as its sixth and final member. Funded in 2018 to expand the evidence base for CTE, the CTE Research Network (led by PI Kathy Hughes at the American Institutes for Research) includes five other CTE impact studies (one project’s interim report by MDRC was recently reviewed by the What Works Clearinghouse and was found to meet standards without reservations). You can read more about the network’s mission and each of its member projects here.  

On AIR’s CTE Research Network website, you can find several new resources and reports, such as: 

The CTE Research Network has also been conducting training, including workshops in causal design for CTE researchers and online modules on data and research for CTE practitioners, shared widely with the field by a Network Lead partner, the Association for Career and Technical Education (ACTE). 

Last but certainly not least, if you are interested in getting your CTE project funded by IES, see the new FY22 research grant opportunities on the IES funding page. To apply to the CTE topic in the Education Research Grants program specifically, click on the PDF Request for Applications (ALN 84.305A). Contact Corinne Alfeld with any questions you might have.


Written by Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER Program Officer 

 

Two-Year Position in the Institute Of Education Sciences (NCSER) to Support Research on Accelerating Pandemic Recovery for Learners With Disabilities

A banner that states "We're Hiring" with the IES logo

The National Center for Special Education Research (NCSER) in the Institute of Education Sciences is pleased to announce a two-year position to support the work NCSER is undertaking to accelerate pandemic recovery for learners with disabilities. The position, funded through the American Rescue Plan, is for an Associate Education Research Scientist (position series AD-1730).  The incumbent will work with current NCSER staff to support research that addresses pandemic recovery for students with disabilities and manage projects funded through new pandemic recovery grant competitions and initiatives. 

Those interested in applying can submit their application through USA Jobs through an existing position posting for an Associate Education Research Scientist (AD-1730-00) at https://www.usajobs.gov/GetJob/ViewDetails/603739300.

Please note, IES can support a temporary detail to this position through the Intergovernmental Personnel Act (IPA) Mobility Program. This program provides for the temporary assignment of personnel between the Federal Government and state and local governments, colleges and universities, Indian tribal governments, federally funded research and development centers, and other eligible organizations. More information on an IPA can be found at https://www.opm.gov/policy-data-oversight/hiring-information/intergovernment-personnel-act/#url=Overview.

If you are interested in pursuing this opportunity and have additional questions, contact NCSER Commissioner Joan McLaughlin at NCSER.Commissioner@ed.gov.

NCSER is intending to fill this position as soon as possible, so please apply by August 6, 2021.

IES Announces a New Research and Development Center for Self-Directed Learning Skills in Online College Courses

In response to a call from IES for research on how to best support postsecondary teachers and students to thrive in online environment, NCER is establishing a new research and development (R&D) center. This center, led by SRI International (SRI) and the Community College Research Center (CCRC) at Teachers College (Columbia University), aims to help faculty embed support for self-directed learning skills into their online and hybrid courses.

This R&D center will support postsecondary instructors in making optimal use of technology features often available in online course tools to bolster student self-management strategies. Through its research and capacity-building programs, the center aims to strengthen teaching and learning, improve student outcomes, and ensure all students—regardless of race, ethnicity, or socioeconomic status—have equitable learning opportunities and attainment in broad-access institutions.

“Lack of self-directed learning skills can hinder a student’s success in any college course,” says SRI’s Rebecca Griffiths, a lead researcher in the new center, “but the challenge is greater in online courses, which typically place more responsibility on students to manage their own learning.”

 

Self-directed learning skills, also known as self-regulated learning skills, encompass three interrelated and mutually reinforcing areas:

  • Affect, which includes self-efficacy and the motivation to learn
  • Strategic actions, which include planning, goal setting, strategies to organize, code, and rehearse
  • Metacognition, which includes self-monitoring, self-evaluating and self-correction

 

These three areas can form a virtuous cycle. When students believe that studying helps them learn important and useful knowledge, they are more likely to study strategically and effectively. Effective study habits in turn enhance academic performance and build positive mindsets including confidence, motivation around learning, and a sense of personal growth.

SRI and CCRC will partner with Achieving the Dream and nine broad-access, public colleges, and universities across the U.S. to conduct these research program activities.

 

The research goals of the R&D center are to—

  • Generate new knowledge about how faculty can effectively use technology features and instructional practices in online STEM courses to create a positive feedback loop for students
  • Shed light on institutional policies and practices and instructional environments needed to support a coherent, intentional, and sustainable approach to helping students build self-directed learning skills across their coursework
  • Develop and pilot a technology-enabled, skills development model that will use technology features already widely available in learning management systems, adaptive courseware, and mobile apps to deliver instruction on these skills
  • Using research findings to inform the development of a rich, interactive toolkit to support institutions and faculty as they implement self-directed learning skills instruction at scale in online programs.

 

In addition to carrying out the research activities, the center will provide national leadership and capacity-building activities for postsecondary teaching and learning. Through partnership with Achieving the Dream, technology developers, researchers, education equity advocates, and others, the center will establish the critical importance of integrating self-directed learning into instruction to improve teaching and learning and improve equity in postsecondary student outcomes. They will also engage faculty, instructional designers, and educational technology developers to share knowledge and to co-develop and disseminate capacity-building resources that support teaching these skills and strategies. 


The center is led by Dr. Deborah Jonas (PI, SRI International, top photo), Dr. Nicole Edgecombe (Co-PI, Teachers College, Columbia University), Dr. Rebecca Griffiths (Co-PI, SRI International), and Dr. Neil Seftor (Co-PI, SRI International).

This blog was written by the R&D center team. For further information about the grant, contact the program officer: Dr. Meredith Larson.