Inside IES Research

Cost analysis is a critical part of education research because it communicates what resources are needed for a particular program or intervention. Just telling education leaders how promising a program or practice can be does not tell the whole story; they need to know how much it will cost so that they can prioritize limited resources. Since 2015, cost analysis has been required for IES-funded Efficacy/Impact studies (and for Development Innovation studies as of 2019) and is included in the IES Standards for Excellence in Education Research.

In this guest blog for CTE Month, two members of the CTE Research Network’s cost analysis working group, David Stern, an advisor to the network, and Eric Brunner, a co-PI of one of the research teams, discuss how costs associated with CTE programs may differ from those of standard education and how to measure those costs.

Why is cost analysis different in Career & Technical Education (CTE) research?

Due to additional, non-standard components needed in some types of career training, CTE can cost much more than the education resources needed in regular classrooms. For instance, CTE classes often use specialized equipment—for example, hydraulic lifts in automotive mechanics, stoves and refrigerators in culinary arts, or medical equipment in health sciences—which costs significantly more than equipment in the standard classroom. Having specialized equipment for student use can also constrain class size to be smaller, resulting in higher cost-per-pupil.  High schools and community colleges may also build labs within existing buildings or construct separate buildings to house CTE programs with specialized equipment. These required facility expenses will need to be recognized in cost calculations.

CTE programs can also provide co-curricular experiences for students alongside classes in career-related subjects, such as work-based learning, career exploration activities, or integrated academic coursework. Schools are usually required to provide transportation for students to workplaces, college campuses for field trips, or regional career centers, which is another expense. Finally, the budget lines for recruiting and retaining teachers from some higher paying career areas and industries (such as nursing or business) may exceed those for average teacher salaries. All of these costs add up. To provide useful guidance for the field, CTE researchers should measure and report the cost of these features separately.

How is resource cost different from reported spending? 

There are also some hidden costs to account for in research on CTE. For example, suppose a school does not have a work-based learning (WBL) coordinator, so a CTE teacher is allowed one of their 5 periods each day to organize and oversee WBL, which may include field trips to companies, job shadowing experiences, internships, or a school-based enterprise. The expenditure report would show 20% of the teacher’s salary has been allocated for that purpose. In reality, however, a teacher may devote much more than 20% of their time to this. They may in fact be donating to the program by spending unpaid time or resources (such as transportation in their own vehicle to visit employer sites to coordinate learning plans) outside the workday. It is also possible that the teacher would spend less than 20% of their time on this. To obtain an accurate estimate of the amount of this resource cost at a particular school, a researcher would have to measure how much time the teacher actually spends on WBL.  This could be done as part of an interview or questionnaire.

Similarly, high school CTE programs are increasingly being developed as pathways that allow students to move smoothly to postsecondary education, such as via dual enrollment programs or directly to the labor market. Building and sustaining these pathways takes active collaboration between secondary and postsecondary educators and employers. However, the costs of these collaborations in terms of time and resources are unlikely to be found in a school expenditure report. Thus, an incremental cost analysis for CTE pathway programs must go beyond budgets and expenditure reports to interview or survey program administrators and staff about the resources or “ingredients” that programs require to operate. A recent example of a cost study of a CTE program can be found here.

Are there any resources for calculating CTE Costs?

In this blog, we have presented some examples of how the costs associated with CTE programs may differ from those of a standard education. To help CTE researchers conduct cost analysis, the CTE Research Network has developed a guide to measuring Incremental Costs in Career and Technical Education, which explains how to account for the particular kinds of resources used in CTE. The guide was developed by the working group on cost analysis supported by the CTE Research Network.

The Career and Technical Education (CTE) Research Network has supported several cross-network working groups comprised of members of network research teams and advisors working on issues of broad interest to CTE research. Another CTE Network working group developed an equity framework for CTE researchers, which was described in a blog for CTE month in February, 2023.

This blog was produced by Corinne Alfeld, NCER program officer for the CTE research topic and the CTE Research Network. Contact: Corinne.Alfeld@ed.gov.

It is undeniable that artificial intelligence (AI) is, sooner rather than later, going to impact the work of teaching and learning in special education. Given formal adoption of AI technologies by schools and districts and informal uses of ChatGPT and similar platforms by educators and students, the field of special education research needs to take seriously how advancements in AI can complement and potentially improve our work. But there are also ways that these advancements can go astray. With these technologies advancing so quickly, and with AI models being trained on populations that may not include individuals with disabilities, there is a real risk that AI will fail to improve—or worse, harm—learning experiences for students with disabilities. Therefore, there is a pressing need to ensure that voices from within the special education community are included in the development of these new technologies.

At NCSER, we are committed to investing in research on AI technologies in a way that privileges the expertise of the special education community, including researchers, educators, and students with disabilities and their families. Below, we highlight two NCSER-funded projects that demonstrate this commitment.

Using AI to support speech-language pathologists

In 2023, NCSER partnered with the National Science foundation to fund AI4ExceptionalEd, a new AI Institute that focuses on transforming education for children with speech and language disorders. Currently, there is a drastic shortage of speech-language pathologists (SLPs) to identify and instruct students with speech and language needs. AI4ExceptionalEd brings together researchers from multiple disciplines including special education, communication disorders, learning sciences, linguistics, computer science, and AI from nine different universities across the United States to tackle pressing educational issues around the identification of students and the creation of specially designed, individualized instruction for students with speech and language disorders.

By bringing together AI researchers and education researchers, this interdisciplinary research partnership is setting the foundation for cutting-edge AI technologies to be created that solve real-world problems in our schools. A recent example of this is in the creation of flash cards for targeted intervention. It is common practice for an SLP to use flash cards that depict a noun or a verb in their interventions, but finding or creating the exact set of flash cards to target a specific learning objective for each child is very time consuming. Here is where AI comes into play. The Institute’s team of researchers is leveraging the power of AI to help SLPs identify optimal sets of flash cards to target the learning objectives of each learner while also creating the flash cards in real time. To do this effectively, the AI researchers are working hand-in-hand with speech and language researchers and SLPs in the iterative development process, ensuring that the final product is aligned with sound educational practices. This one AI solution can help SLPs optimize their practice and reduce time wasted in creating materials.

Adapting a popular math curriculum to support students with reading disabilities

Another example of how partnerships can strengthen cutting-edge research using AI to improve outcomes for students with disabilities is a 2021 grant to CAST to partner with Carnegie Learning to improve their widely used digital math curriculum, MATHia. The goal of this project is to develop and evaluate reading supports that can be embedded into the adaptive program to improve the math performance, particularly with word problems, of students with reading disabilities. CAST is known for its research and development in the area of universal design for learning (UDL) and technology supports for students with disabilities. Carnegie Learning is well known for their suite of curriculum products that apply cognitive science to instruction and learning. The researchers in this partnership also rely on a diverse team of special education researchers who have expertise in math and reading disabilities and an educator advisory council of teachers, special educators, and math/reading specialists.

It has taken this kind of partnership—and the inclusion of relevant stakeholders and experts—to conduct complex research applying generative AI (ChatGPT) and humans to revise word problems within MATHia to decrease reading challenges and support students in understanding the semantic and conceptual structure of a word problem. Rapid randomized control trials are being used to test these revised versions with over 116,000 students participating in the study. In 2022-2023 the research team demonstrated that humans can successfully revise word problems in ways that lead to improvements in student performance, including students with disabilities. The challenge is in trying to train generative AI to reproduce the kinds of revisions humans make. While generative AI has so far been unevenly successful in making revisions that similarly lead to improvements in student outcomes, the researchers are not ruling out the use of generative AI in revising word problems in MATHia.

The research team is now working with their expert consultants on a systematic reading and problem-solving approach as an alternative to revising word problems. Instead of text simplification, they will be testing the effect of adding instructional support within MATHia for some word problems.

The promise of AI

AI technologies may provide an opportunity to optimize education for all learners. With educators spending large amounts of their day planning and doing paperwork, AI technologies can be leveraged to drastically decrease the amount of time teachers need to spend on this administrative work, allowing more time for them to do what only they can—teach children. Developers and data scientists are invariably going to continue developing AI technologies, many with a specific focus on solutions to support students with disabilities. We would like to encourage special education researchers to exert their expertise in this development work, to partner with developers and interdisciplinary teams to apply AI to create innovative and novel solutions to improve outcomes for students with disabilities. For AI to lead to lasting advances in education spaces, it will be imperative that this development is inclusive of the special education field.

This blog was written by NCSER Commissioner, Nate Jones (Nathan.Jones@ed.gov) and NCSER program officers Britta Bresina (Britta.Bresina@ed.gov) and Sarah Brasiel (Sarah.Brasiel@ed.gov).

Through the Transformative Research in the Education Sciences Grants program (ALN 84.305T), IES  invests in innovative research that has the potential to make dramatic advances towards solving seemingly intractable problems and challenges in the education field, as well as to accelerate the pace of conducting education research to facilitate major breakthroughs. In the most recent FY 2024 competition for this program, IES invited applications from partnerships between researchers, product developers, and education agencies to propose transformative solutions to major education problems that leverage advances in technology combined with research insights from the learning sciences.

IES is thrilled to announce that three grants have been awarded in the FY 2024 competition. Building on 20 years of IES research funding to lay the groundwork for advances, these three projects focus on exploring potentially transformative uses of generative artificial intelligence (AI) to deliver solutions that can scale in the education marketplace if they demonstrate positive impacts on education outcomes. The three grants are:

Active Learning at Scale (Active L@S): Transforming Teaching and Learning via Large-Scale Learning Science and Generative AI

Awardee: Arizona State University (ASU; PI: Danielle McNamara)

The project team aims to solve the challenge that postsecondary learners need access to course materials and high-quality just-in-time generative learning activities flexibly and on-the-go.  The solution will be a mobile technology that uses interactive, research-informed, and engaging learning activities created on the fly, customized to any course content with large language models (LLMs). The project team will leverage two digital learning platforms from the SEERNet network – Terracotta and ASU Learning@Scale – to conduct research and will include over 100,000 diverse students at ASU, with replication studies taking place at Indiana University (IU). IES funding has supported a large portion of the research used to identify the generative learning activities the team will integrate into the system—note-taking, self-explanation, summarization, and question answering (also known as retrieval practice). The ASU team includes in-house technology developers and researchers, and they are partnering with researchers at IU and developers at INFLO and Clevent AI Technology LLC. The ASU and IU teams will have the educator perspective represented on their teams, as these universities provide postsecondary education to large and diverse student populations.

Talking Math: Improving Math Performance and Engagement Through AI-Enabled Conversational Tutoring

Awardee: Worcester Polytechnic Institute (PI: Neil Heffernan)

The project team aims to provide a comprehensive strategy to address persistent achievement gaps in math by supporting students during their out-of-school time. The team will combine an evidence-based learning system with advances in generative AI to develop a conversational AI tutor (CAIT– pronounced as “Kate”) to support independent math practice for middle school students who struggle with math, and otherwise, may not have access to after-school tutoring. CAIT will be integrated into ASSISTments, a freely available, evidence-based online math platform with widely used homework assignments from open education resources (OER). This solution aims to dramatically improve engagement and math learning during independent math problem-solving time. The team will conduct research throughout the product development process to ensure that CAIT is effective in supporting math problem solving and is engaging and supportive for all students. ASSISTments has been used by over 1 million students and 30,000 teachers, and IES has supported its development and efficacy since 2003. The project team includes researchers and developers at Worcester Polytechnic Institute and the ASSISTments Foundation, researchers from WestEd, educator representation from Greater Commonwealth Virtual School, and a teacher design team.

Scenario-Based Assessment in the age of generative AI: Making space in the education market for alternative assessment paradigm

Awardee: University of Memphis (PI: John Sabatini)

Educators face many challenges building high-quality assessments aligned to course content, and traditional assessment practices often lack applicability to real world scenarios. To transform postsecondary education, there needs to be a shift in how knowledge and skills are assessed to better emphasize critical thinking, complex reasoning, and problem solving in practical contexts. Supported in large part by numerous IES-funded projects, including as part of the Reading for Understanding Initiative, the project team has developed a framework for scenario-based assessments (SBAs). SBAs place knowledge and skills into a practical context and provide students with the opportunity to apply their content knowledge and critical thinking skills. The project team will leverage generative AI along with their framework for SBAs to create a system for postsecondary educators to design and administer discipline-specific SBAs with personalized feedback to students, high levels of adaptivity, and rich diagnostic information with little additional instructor effort. The project team includes researchers, developers, and educators at University of Memphis and Georgia State University, researchers and developers at Educational Testing Service (ETS), and developers from multiple small businesses including Capti/Charmtech, MindTrust, Caimber/AMI, and Workbay who will participate as part of a technical advisory group.

We are excited by the transformative potential of these projects and look forward to seeing what these interdisciplinary teams can accomplish together. While we are hopeful the solutions they create will make a big impact on learners across the nation, we will also share lessons learned with the field about how to build interdisciplinary partnerships to conduct transformative research and development.

For questions or to learn more about the Transformative Research in the Education Sciences grant program, please contact Erin Higgins (Erin.Higgins@ed.gov), Program Lead for the Accelerate, Transform, Scale Initiative.

For students with learning disabilities, many of whom may take more time to read and process information than non-disabled peers, an extended time accommodation (ETA) is often used on standardized assessments. In 2021, IES awarded a grant for researchers to explore the test-taking behavior, including use of accommodations such as ETA, of students with disabilities in middle school using response process data from the NAEP mathematics assessment. In this blog, we interview Dr. Xin Wei from Digital Promise to see what she and Dr. Susu Zhang from University of Illinois at Urbana-Champaign are learning from their study.

The researchers have delved into the performance, process, and survey data of the eighth graders who took the digital NAEP mathematics test in 2017. Their recent article presents a quasi-experimental study examining the differences in these data across three distinct profiles of students with learning disabilities (LDs)—students with LD who received and utilized ETAs, students with LD who were granted ETAs but did not use them, and students with LD who did not receive ETAs.

The key findings from their study are as follows:

• Students with LDs who used their ETAs performed statistically significantly better than their peers with LDs who were not granted ETA and those who received ETA but did not use it. They also engaged more with the test, as demonstrated by more frequent actions, revisits to items, and greater use of universal design features like drawing tool and text-to-speech functionalities on most of the math items compared to students who were not granted extended time.
• Students with LDs who had ETAs but chose not to use them performed significantly worse than their peers with LDs who were not granted extended time.
• Students with LDs who were granted ETAs saw the best performance with an additional 50% time (45 minutes compared to the usual 30 minutes provided to students without ETA).
• Students who were given extra time, regardless of whether they used it, reported feeling less time pressure, higher math interest, and enjoying math more.
• There were certain item types for which students who used ETAs performed more favorably.

We recently discussed the results of the study with Dr. Wei to learn more.

Which types of items on the test favored students who used extended time and why do you think they benefited?

The assessment items that particularly benefited from ETAs were not only complex but also inherently time-consuming. For example, students need to complete four sub-questions for item 5, drag six numbers to the correct places for item 6, type answers into four places to complete an equation for item 9, type in a constructive response answer for item 11, and complete a multiple-choice question and type answers in eight places to complete item 13.

For students with LDs, who often have slower processing speeds, these tasks become even more time-intensive. The additional time allows students to engage with each element of the question thoroughly, ensuring they have the opportunity to fully understand and respond to each part. This extended time is not just about accommodating different processing speeds; it's about providing the necessary space for these students to engage with and complete tasks that are intricate and time-consuming by design.

Why did you decide to look at the additional survey data NAEP collects on math interest and enjoyment in your study of extended time?

These affective factors are pivotal to academic success, particularly in STEM fields. Students who enjoy the subject matter tend to perform better, pursue related fields, and continue learning throughout their lives. This is especially relevant for students with LDs, who often face heightened test anxiety and lower interest in math, which can be exacerbated by the pressure of timed assessments. Our study's focus on these affective components revealed that students granted extra time reported a higher level of math interest and enjoyment even if they did not use the extra time. ETAs appear to alleviate the stress tied to time limits, offering dual advantages by not only aiding in academic achievement but also by improving attitudes toward math. ETAs could be a low-cost, high-impact accommodation that not only addresses academic needs but also contributes to emotional health.

What recommendations do you have based on your findings for classroom instruction?

First, it is crucial to prioritize extra time for students with LDs to enhance their academic performance and engagement. This involves offering flexible timing for assignments and assessments to reduce anxiety and foster a greater interest in learning. Teachers should be encouraged to integrate Universal Design for Learning principles into their instructional methods, emphasizing the effective use of technology, such as text-to-speech tools and embedded digital highlighters and pencils for doing scratchwork. Professional development for educators is essential to deepen their proficiency in using digital learning tools. Additionally, teachers should motivate students to use the extra time for thorough problem-solving and to revisit math tasks for accuracy. Regularly adjusting accommodations to meet the evolving needs of students with LDs is vital in creating an inclusive learning environment where every student can achieve success.

What is the implication of the study findings on education equity? 

Our study demonstrates that ETAs offer more than just a performance boost: they provide psychological benefits, reducing stress and enhancing interest and enjoyment with the subject matter. This is vital for students with LDs, who often face heightened anxiety and performance pressure. To make the system more equitable, we need a standardized policy for accommodations that ensures all students who require ETAs receive them. We must consider the variable needs of all students and question the current practices and policies that create inconsistencies in granting accommodations. If the true aim of assessments is to gauge student abilities, time is a factor that should not become a barrier.

U.S. Department of Education Resources

Learn more about the Department’s resources to support schools, educators, and families in making curriculum, instruction, and assessment accessible for students with disabilities.

• The Center on Inclusive Technology and Education Systems (CITES) funded by the Office of Special Education Programs (OSEP)
• The National Center on Accessible Educational Materials (AEM) funded by OSEP
• Myths and Facts Surrounding Assistive Technology Devices and Services from OSEP
• National Education Technology Plan from the Office of Education Technology

Learn more about conducting research using response process data from the 2017 NAEP Mathematics Assessment.

This  interview blog was produced by Sarah Brasiel (Sarah.Brasiel@ed.gov), a program officer in the National Center for Special Education Research.

A key challenge for children with autism is the need to strengthen social communication, something that can be supported early in a child’s development. Dr. Hannah Schertz, professor at Indiana University Bloomington’s School of Education, has conducted a series of IES-funded projects to develop and evaluate the impact of early intervention, mediated through parents, for improving social communication in toddlers with or at risk for autism. We recently interviewed Dr. Schertz to learn more about the importance of guiding parents in the use of mediated learning practices to promote social communication, how her current research connects with her prior research, and what she hopes to accomplish.

Why is parental mediation in early intervention important for very young children with autism? How does it work and why do you focus this approach on improving children’s social communication development?

The intervention targets social communication because it is the core autism challenge and it’s important to address concerns early, as signs of autism emerge. Research has found that preverbal social communication is related to later language competency. Our premise is that this foundation will give toddlers a reason to communicate and set the stage for verbal communication. More specifically, joint attention—one preverbal form of social communication—is the key intervention target in our research. It is distinct from requesting/directing or following requests, which are instrumental communications used to accomplish one’s own ends. Joint attention, which takes the partner’s interests and perspectives into account, is an autism-specific challenge whereas more instrumental communication skills are not.

Our research team incorporates a mediated learning approach at two levels—early intervention providers supporting parents and then parents supporting their toddlers. The approach is designed to promote active engagement in the learning process and leverage the parent’s privileged relationship with the child as the venue for social learning. Early intervention providers help parents understand both the targeted social communication outcomes for their children (intervention content) and the mediated learning practices (intervention process) used to promote these child outcomes. As parents master these concepts, they can translate them flexibly into a variety of daily parent-child interactions. This understanding allows parents to naturally integrate learning opportunities with child interests and family cultural/language priorities and preferences. Over time, their accrued knowledge, experience, and increased self-efficacy should prepare them to continually support the child’s social learning even after their participation in the project ends.

How does your more recent work, developing and testing Building Interactive Social Communication (BISC), extend your prior research examining Joint Attention Mediated Learning (JAML)?

Both JAML and BISC address the same goal—supporting social communication as early signs of autism emerge. In JAML, researchers guided parent learning directly while parents incorporated social communication into interaction with their toddlers. BISC extends the intervention by supporting community-based practitioners in facilitating parent learning rather than parents learning directly from the research team. BISC also added a component to address cases in which parents identify child behaviors that substantially interfere with the child’s social engagement.

You recently completed a pilot study to test a new professional development framework for supporting early intervention providers in implementing BISC. Please tell us about the findings of this study. What were the impacts on the early intervention providers, parents, and toddlers?

We tested an early version of BISC to study its preliminary effects on early intervention provider, parent, and child outcomes for 12 provider/parent/child triads. In effect size estimates derived from single-case design data, we found large effects for early intervention provider fidelity (for example, mediating parent learning, guiding parents’ reflection on video-recorded interaction with their toddlers, and supporting active parent engagement) and parent application of mediated learning practices to promote toddler social communication. We also found large effects on child outcomes (social reciprocity, child behavior, and social play) and a small effect on joint attention.

As you begin your larger-scale trial to examine the efficacy of BISC on provider, parent, and child outcomes, what impact do you hope your work will have on the field of early intervention generally and the development of social communication in children with autism more specifically? 

Approximately 165 community-based early intervention practitioners will have learned to support parent learning through direct participation or as control group participants who receive self-study materials. These providers will be equipped to bring this knowledge to their future work. We anticipate that practitioners will experience their implementation role as feasible and effective. Ultimately, toddlers with early signs of autism will have greater access to early, developmentally appropriate, and family-empowering early intervention that directly addresses the core social difficulty of autism. Forthcoming published materials will extend access to other providers, offering an intervention that is more specifically tailored to the needs of very young children with social communication challenges than other approaches.

Is there anything else you would like to share/add regarding your projects? 

I would like to thank my colleagues and project co-principal investigators (Co-PIs) for their expertise and contributions to this work. For our current BISC efficacy project, Co-PI Dr. Patricia Muller (Director of the Center for Evaluation, Policy, and Research) is leading the randomized controlled trial and cost-effectiveness study, and Co-PI Dr. Jessica Lester (professor of Counseling and Educational Psychology) is overseeing the qualitative investigation of parent-child interactions using conversation analysis to explore potential influences on child outcomes. Kathryn Horn coordinates intervention activities, Lucia Zook oversees operational and assessment activities, and Addison McGeary supports recruitment and logistical activities.

This blog was authored by Skyler Fesagaiga, a Virtual Student Federal Service intern for NCSER and graduate student at the University of California, San Diego. The grants in this connected line of research have been managed by Amy Sussman (PO for NCSER’s early intervention portfolio) and Emily Weaver (PO for NCSER’s autism research portfolio).