IES Blog

Cost analysis is a critical part of education research because it communicates what resources are needed for a particular program or intervention. Just telling education leaders how promising a program or practice can be does not tell the whole story; they need to know how much it will cost so that they can prioritize limited resources. Since 2015, cost analysis has been required for IES-funded Efficacy/Impact studies (and for Development Innovation studies as of 2019) and is included in the IES Standards for Excellence in Education Research.

In this guest blog for CTE Month, two members of the CTE Research Network’s cost analysis working group, David Stern, an advisor to the network, and Eric Brunner, a co-PI of one of the research teams, discuss how costs associated with CTE programs may differ from those of standard education and how to measure those costs.

Why is cost analysis different in Career & Technical Education (CTE) research?

Due to additional, non-standard components needed in some types of career training, CTE can cost much more than the education resources needed in regular classrooms. For instance, CTE classes often use specialized equipment—for example, hydraulic lifts in automotive mechanics, stoves and refrigerators in culinary arts, or medical equipment in health sciences—which costs significantly more than equipment in the standard classroom. Having specialized equipment for student use can also constrain class size to be smaller, resulting in higher cost-per-pupil.  High schools and community colleges may also build labs within existing buildings or construct separate buildings to house CTE programs with specialized equipment. These required facility expenses will need to be recognized in cost calculations.

CTE programs can also provide co-curricular experiences for students alongside classes in career-related subjects, such as work-based learning, career exploration activities, or integrated academic coursework. Schools are usually required to provide transportation for students to workplaces, college campuses for field trips, or regional career centers, which is another expense. Finally, the budget lines for recruiting and retaining teachers from some higher paying career areas and industries (such as nursing or business) may exceed those for average teacher salaries. All of these costs add up. To provide useful guidance for the field, CTE researchers should measure and report the cost of these features separately.

How is resource cost different from reported spending? 

There are also some hidden costs to account for in research on CTE. For example, suppose a school does not have a work-based learning (WBL) coordinator, so a CTE teacher is allowed one of their 5 periods each day to organize and oversee WBL, which may include field trips to companies, job shadowing experiences, internships, or a school-based enterprise. The expenditure report would show 20% of the teacher’s salary has been allocated for that purpose. In reality, however, a teacher may devote much more than 20% of their time to this. They may in fact be donating to the program by spending unpaid time or resources (such as transportation in their own vehicle to visit employer sites to coordinate learning plans) outside the workday. It is also possible that the teacher would spend less than 20% of their time on this. To obtain an accurate estimate of the amount of this resource cost at a particular school, a researcher would have to measure how much time the teacher actually spends on WBL.  This could be done as part of an interview or questionnaire.

Similarly, high school CTE programs are increasingly being developed as pathways that allow students to move smoothly to postsecondary education, such as via dual enrollment programs or directly to the labor market. Building and sustaining these pathways takes active collaboration between secondary and postsecondary educators and employers. However, the costs of these collaborations in terms of time and resources are unlikely to be found in a school expenditure report. Thus, an incremental cost analysis for CTE pathway programs must go beyond budgets and expenditure reports to interview or survey program administrators and staff about the resources or “ingredients” that programs require to operate. A recent example of a cost study of a CTE program can be found here.

Are there any resources for calculating CTE Costs?

In this blog, we have presented some examples of how the costs associated with CTE programs may differ from those of a standard education. To help CTE researchers conduct cost analysis, the CTE Research Network has developed a guide to measuring Incremental Costs in Career and Technical Education, which explains how to account for the particular kinds of resources used in CTE. The guide was developed by the working group on cost analysis supported by the CTE Research Network.

The Career and Technical Education (CTE) Research Network has supported several cross-network working groups comprised of members of network research teams and advisors working on issues of broad interest to CTE research. Another CTE Network working group developed an equity framework for CTE researchers, which was described in a blog for CTE month in February, 2023.

This blog was produced by Corinne Alfeld, NCER program officer for the CTE research topic and the CTE Research Network. Contact: Corinne.Alfeld@ed.gov.

Through the Transformative Research in the Education Sciences Grants program (ALN 84.305T), IES  invests in innovative research that has the potential to make dramatic advances towards solving seemingly intractable problems and challenges in the education field, as well as to accelerate the pace of conducting education research to facilitate major breakthroughs. In the most recent FY 2024 competition for this program, IES invited applications from partnerships between researchers, product developers, and education agencies to propose transformative solutions to major education problems that leverage advances in technology combined with research insights from the learning sciences.

IES is thrilled to announce that three grants have been awarded in the FY 2024 competition. Building on 20 years of IES research funding to lay the groundwork for advances, these three projects focus on exploring potentially transformative uses of generative artificial intelligence (AI) to deliver solutions that can scale in the education marketplace if they demonstrate positive impacts on education outcomes. The three grants are:

Active Learning at Scale (Active L@S): Transforming Teaching and Learning via Large-Scale Learning Science and Generative AI

Awardee: Arizona State University (ASU; PI: Danielle McNamara)

The project team aims to solve the challenge that postsecondary learners need access to course materials and high-quality just-in-time generative learning activities flexibly and on-the-go.  The solution will be a mobile technology that uses interactive, research-informed, and engaging learning activities created on the fly, customized to any course content with large language models (LLMs). The project team will leverage two digital learning platforms from the SEERNet network – Terracotta and ASU Learning@Scale – to conduct research and will include over 100,000 diverse students at ASU, with replication studies taking place at Indiana University (IU). IES funding has supported a large portion of the research used to identify the generative learning activities the team will integrate into the system—note-taking, self-explanation, summarization, and question answering (also known as retrieval practice). The ASU team includes in-house technology developers and researchers, and they are partnering with researchers at IU and developers at INFLO and Clevent AI Technology LLC. The ASU and IU teams will have the educator perspective represented on their teams, as these universities provide postsecondary education to large and diverse student populations.

Talking Math: Improving Math Performance and Engagement Through AI-Enabled Conversational Tutoring

Awardee: Worcester Polytechnic Institute (PI: Neil Heffernan)

The project team aims to provide a comprehensive strategy to address persistent achievement gaps in math by supporting students during their out-of-school time. The team will combine an evidence-based learning system with advances in generative AI to develop a conversational AI tutor (CAIT– pronounced as “Kate”) to support independent math practice for middle school students who struggle with math, and otherwise, may not have access to after-school tutoring. CAIT will be integrated into ASSISTments, a freely available, evidence-based online math platform with widely used homework assignments from open education resources (OER). This solution aims to dramatically improve engagement and math learning during independent math problem-solving time. The team will conduct research throughout the product development process to ensure that CAIT is effective in supporting math problem solving and is engaging and supportive for all students. ASSISTments has been used by over 1 million students and 30,000 teachers, and IES has supported its development and efficacy since 2003. The project team includes researchers and developers at Worcester Polytechnic Institute and the ASSISTments Foundation, researchers from WestEd, educator representation from Greater Commonwealth Virtual School, and a teacher design team.

Scenario-Based Assessment in the age of generative AI: Making space in the education market for alternative assessment paradigm

Awardee: University of Memphis (PI: John Sabatini)

Educators face many challenges building high-quality assessments aligned to course content, and traditional assessment practices often lack applicability to real world scenarios. To transform postsecondary education, there needs to be a shift in how knowledge and skills are assessed to better emphasize critical thinking, complex reasoning, and problem solving in practical contexts. Supported in large part by numerous IES-funded projects, including as part of the Reading for Understanding Initiative, the project team has developed a framework for scenario-based assessments (SBAs). SBAs place knowledge and skills into a practical context and provide students with the opportunity to apply their content knowledge and critical thinking skills. The project team will leverage generative AI along with their framework for SBAs to create a system for postsecondary educators to design and administer discipline-specific SBAs with personalized feedback to students, high levels of adaptivity, and rich diagnostic information with little additional instructor effort. The project team includes researchers, developers, and educators at University of Memphis and Georgia State University, researchers and developers at Educational Testing Service (ETS), and developers from multiple small businesses including Capti/Charmtech, MindTrust, Caimber/AMI, and Workbay who will participate as part of a technical advisory group.

We are excited by the transformative potential of these projects and look forward to seeing what these interdisciplinary teams can accomplish together. While we are hopeful the solutions they create will make a big impact on learners across the nation, we will also share lessons learned with the field about how to build interdisciplinary partnerships to conduct transformative research and development.

For questions or to learn more about the Transformative Research in the Education Sciences grant program, please contact Erin Higgins (Erin.Higgins@ed.gov), Program Lead for the Accelerate, Transform, Scale Initiative.

On Tuesday, November 7, 2023, the White House’s Office of Science and Technology Policy hosted a Demo Day of American Possibilities at the Showroom in Washington, DC.  The event featured 45 emerging technologies created by innovators through federal research and development programs across areas such as health, national security, AI, robotics, climate, microelectronics, and education. President Biden attended the event and met with several developers to learn about and see demonstrations of the innovations.

An IES-supported project by a Michigan-based Alchemie, the KASI Learning System (KASI), was invited to represent the U.S. Department of Education and its Small Business Innovation Research program, which IES administers.

KASI is an inclusive assistive technology that employs computer vision and multi-sensory augmented reality to support blind and low vision learners in using hand-held physical manipulatives to practice chemistry. A machine learning engine in KASI generates audio feedback and prompts to personalize the experience as learners progress. At the event, the project’s principal investigator and former high school chemistry educator, Julia Winter, demonstrated KASI to leaders in government and to attendees from the assistive technology field.

ED/IES SBIR supported the initial development for KASI through three awards. Based on these awards, Alchemie received funding from angel investors in Michigan, won a commercialization grant from the Michigan Emerging Technology Fund, and is establishing partnerships with publishers in K-12 and higher education. To extend KASI to more topics, Alchemie has won additional SBIR awards from the National Science Foundation, the National Institutes of Health, and the National Institute of Disability, Independent Living, and Rehabilitation Research, and is currently a finalist in the 2024 Vital Prize Challenge competition. KASI has also recently been highlighted in Forbes and Crain’s Detroit Business.

Stay tuned for updates on KASI and other education technology projects through the ED/IES SBIR program on Twitter, Facebook, and LinkedIn.

About ED/IES SBIR: The Department of Education’s (ED) Small Business Innovation Research (SBIR) program, administered by the Institute of Education Sciences (IES), funds entrepreneurial developers to create the next generation of technology products for learners, educators, and administrators. The program, known as ED/IES SBIR, emphasizes an iterative design and development process and pilot research to test the feasibility, usability, and promise of new products to improve outcomes. The program also focuses on planning for commercialization so that the products can reach schools and end-users and be sustained over time. Millions of students in thousands of schools around the country use technologies developed through ED/IES SBIR.

Edward Metz (Edward.Metz@ed.gov) is the Program Manager of the ED/IES SBIR program.

Laurie Hobbs (Laurie.Hobbs@ed.gov) is the Program Analyst of the ED/IES SBIR program.

Dr. Meredith Larson, program officer for adult education at NCER, interviewed Dr. Javier Suárez-Álvarez, associate professor and associate director at the Center for Educational Assessment, University of Massachusetts Amherst. Dr. Suárez-Álvarez has served as the project director for the Adult Skills Assessment Project: Actionable Assessments for Adult Learners (ASAP) grant and was previously an education policy analyst in France for the Organisation for Economic Co-operation and Development (OECD), where he was the lead author of the PISA report 21st-Century Readers: Developing Literacy Skills in a Digital World. He and the ASAP team are working on an assessment system to meet the needs of adult education learners, educators, and employers that leverages online validated and culturally responsive banks of literacy and numeracy tasks. In this interview, Dr. Suárez-Álvarez discusses the importance of attending to learners’ goals and cultural diversity in assessment.

How would you describe the current context of assessment for adult education, and how does ASAP fit in it?

In general, the adult education field lacks assessments that meet the—sometimes competing—needs and goals of educators and employers and that attend to and embrace learner characteristics, goals, and cultural diversity. There is often a disconnect where different stakeholders want different things from the same assessments. Educators ask for curriculum-aligned assessments, learners want assessments to help them determine whether they have job-related skills for employment or promotion, and employers want to determine whether job candidates are trained in high-demand skills within their industries.

Despite these differing needs and interests, everyone involved needs assessment resources for lower skilled and culturally diverse learners that are easy to use, affordable or free, and provide actionable information for progress toward personal or occupational goals. ASAP is one of the first attempts to respond to these needs by developing an assessment system that delivers real-time customizable assessments to measure and improve literacy and numeracy skills. ASAP incorporates socioculturally responsive assessment principles to serve the needs of all learners by embracing the uniqueness of their characteristics. These principles involve ensuring that stakeholders from diverse socioeconomic, cultural, linguistic, racial, and ethnic groups are represented in our test design and development activities.

Why is attending to cultural diversity important to ASAP and assessment, and how are you incorporating this into your work?

U.S. Census projections for 2045 predict a shift in the demographic composition of the population from a White majority to a racially mixed majority. This suggests that we should prepare for cultural shifts and ensure our assessments fully embrace socioculturally responsive assessment practices. Without these practices, assessments limit the ability of adults from varied demographic backgrounds to demonstrate their capabilities adequately. Socioculturally responsive assessments are pivotal for representing the growing diversity in the learner population and for uncovering undetected workforce potential.

In ASAP, we are conducting focus groups, interviews, and listening sessions with learners, educators, and employers to understand their needs. We are also co-designing items in collaboration with key stakeholders and building consensus across adult education, workforce, and policy experts. We are developing use cases to understand hypothetical product users and conducting case studies to establish linkages between instruction and assessment as well as across classroom and workplace settings.

How has your background informed your interest in and contributions to ASAP?

As a teenager growing up in Spain, I saw first-hand the possible negative impact assessments could have when they don’t attend to learner goals and circumstances. When I was 15, my English teacher, based on narrow assessments, told my parents I was incapable of learning English, doubted my academic potential, and suggested I forego higher education for immediate employment. Defying this with the support of other teachers and my family, I pursued my passion. I became proficient in English at the age of 25 when I needed it to be a researcher, and I completed my PhD in psychology (psychometrics) at the age of 28.

Many adult students may have heard similar messages from prior teachers based on assessment results. And even now, many of the assessments the adult education field currently uses for these learners are designed by and for a population that no longer represents most learners. These adult learners may be getting advice or feedback that does not actually reflect their abilities or doesn’t provide useful guidance. Unfortunately, not all students are as lucky as I was. They may not have the support of others to counterbalance narrow assessments, and that shouldn’t be the expectation.

What are your hopes for the future of assessments for this adult population and the programs and employers that support them?

I hope we switch from measuring what we know generally how to measure (such as math and reading knowledge on a multiple-choice test) to measuring what matters to test takers and those using assessment results so that they can all accomplish goals in ways that honor individuals’ circumstances. Knowledge and skills—like the real world—are much more than right and wrong responses on a multiple-choice item. I also hope that as we embrace the latest developments in technology, such as AI, we can use them to deliver more flexible and personalized assessments.

In addition, I hope we stop assuming every learner has the same opportunities to learn or the same goals for their learning and that we start using assessments to empower learners rather than just as a measure of learning. In ASAP, for example, the adult learner will decide the type of test they want to take when to take it, the context within which the assessment will be framed, and when, where, and to whom the assessment result will be delivered.

This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), program officer for adult education at NCER.

The Predoctoral Interdisciplinary Research Training Program in the Education Sciences was established by IES to increase the number of well-trained PhD students who are prepared to conduct rigorous and relevant education research. IES encourages our predoctoral fellows to develop strong writing skills in addition to subject-matter and methodological expertise. In this guest blog, we asked IES predoctoral fellow, Todd Hall, co-chair of the Black Scholars in Education and Human Development Writing Group at the University of Virginia, to discuss how participating in this writing group has helped his development as an education researcher. Todd, is part of the IES-funded Virginia Education Science Training (VEST) program and studies early childhood education policy as well as school discipline in both early childhood and K-12 settings.

How did you become involved in the Black Scholars in Education and Human Development Writing Group?

I started my PhD in education policy in August 2020. The COVID-19 pandemic made networking and simply making friends awkward. During my first week in Charlottesville, VA, I watched wistfully from my window as a Black person jogged past my house. For me, the jogger represented communities of color at UVA that I did not know how to connect with.

Enter Dr. Edward Scott and Dr. Miray Seward, then students and co-chairs of the Black Scholars in Education and Human Development Writing Group. They sent me a personal email invitation to join the group’s first virtual writing retreat. When I joined the Zoom room, I found the affinity space I was looking for. I connected with graduate students whom I later turned to for informal mentorship, course recommendations, tips on navigating the hidden curriculum of grad school, insights from job market experiences, and examples of successful written proposals. The laughs shared virtually during check-ins between writing blocks helped ward off the pandemic blues.

I resolved to pay it forward, so I began shadowing Edward and Miray. When they graduated, I stepped into a leadership role alongside my co-chair, Sasha Miller-Marshall.

How has participating in the writing group helped you develop as a scholar?

The writing group has reminded me that I am not the only one who experiences writer’s block and has provided me with writing process role models. The professional development sessions we host have been one of the few opportunities that I have found to see faculty expose and reflect on their own writing challenges, from protecting their time for writing to incorporating critical feedback. This provides a unique perspective on the writing process—I often see faculty discuss works in progress, but the format is usually an oral presentation with slides rather than something written.

In the Black Scholars Writing Group sessions, speakers often share candidly about their own process, including writer’s block and how they overcome it. For example, a senior faculty member shared that they used voice memos to process their thoughts when they feel stuck. That disclosure normalized my experience of writer’s block and made me feel comfortable sharing that I write memos on my phone when I feel stuck. Moments like these have provided tools to overcome resistance in my writing process and normalized the experience of strategizing about writing rather than expecting words to flow effortlessly.

The presenters who lead sessions with our group have diverse racial/ethnic backgrounds, but the focus of the group on creating affinity space for Black doctoral and PhD students allows me to be less concerned about stereotype threat. Whereas I am often the only Black person in other rooms, I am never the only Black person in this writing group. That alleviates any concern about being perceived as a token representative of Black people, or worse, as less capable if I choose to share my difficulties. In one session, I was able to unpack with the faculty speaker that a particular piece of writing was difficult because I had not yet answered the simple question of why the work was important. I got to that realization because the speaker modeled vulnerability about their own writing process, and I felt at ease to discuss my own.

How can the broader education research community help graduate student researchers develop as writers?

Where appropriate and feasible, education researchers can share their successful conference proposals, grant applications, budgets, reviewer response letters, and perhaps even dissertation chapters. If it does not make sense to post them publicly, researchers could offer to share materials with graduate students that they meet at speaking engagements, conferences, etc.

Successful models have given me helpful guidance, especially when tackling a new format. Beyond the writing group, I am immensely grateful to the alumni of my IES pre-doctoral fellowship who have provided many of their materials for current students to reference.

What advice can you give other student researchers who wish to further develop their writing skills?

Cultivate authentic relationships with a network of mentors who are willing to share examples of their successful writing and review your work. My advisor is amazing and thorough with her feedback. That said, it has been useful to strategically ask others who bring in complementary perspectives to review my work. For example, my advisor is a quantitative researcher, and I recently proposed a mixed methods study. Researchers who do qualitative and mixed methods work were able to challenge and strengthen the qualitative aspects of my proposal based on their expertise. You might also be applying for opportunities or submitting to journals that other mentors have succeeded with or reviewed for. They may help you anticipate what that audience might be looking for.

In addition, when you receive feedback, do so graciously, weigh it seriously, and ask yourself if there’s a broader piece of constructive criticism to apply to your other writing.

This blog was produced training program officer Katina Stapleton (Katina.Stapleton@ed.gov) and is part of a larger series on the IES research training programs.