IES Blog

Institute of Education Sciences

Cost Analysis in Practice: Resources for Cost Analysis Studies

IES supports rigorous research that can provide scientific evidence on how best to address our nation’s most pressing education needs. As part of the Standards for Excellence in Education Research (SEER) principles, IES-funded researchers are encouraged, and in some cases required, to conduct a cost analysis for their projects with the intended goal of supporting education agencies’ decision-making around the adoption of programs, policies, or practices. 

 

The Cost Analysis in Practice (CAP) Project is a 3-year initiative funded by IES to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. This support includes the following freely available resources.

  • Resources developed by the CAP Project
    • Introductory resources on cost analysis including Standards and Guidelines 1.1, an infographic, a video lecture, and FAQs.
    • Tools for planning your cost analysis, collecting and analyzing cost data, and reporting your results.
    • A Help Desk for you to submit inquiries about conducting a cost analysis with a response from a member of the CAP Project Team within two business days.
  • Other resources recommended by the CAP Project
    • Background materials on cost analysis
    • Guidance on carrying out a cost analysis
    • Standards for the Economic Evaluation of Educational and Social Programs
    • Cost analysis software

 

The CAP Project is also involved in longer-term collaborations with IES-funded evaluation projects to better understand their cost analysis needs. As part of this work, the CAP Project will be producing a set of three blogs to discuss practical details regarding cost studies based on its collaboration with a replication project evaluating an intervention that integrates literacy instruction into the teaching of American history. These blogs will discuss the following:

  • Common cost analysis challenges that researchers encounter and recommendations to address them
  • The development of a timeline resource for planning a cost study
  • Data collection for a cost study

 

The CAP Project is interested in your feedback on any of the CAP Project resources and welcomes suggestions for additional resources to support cost analysis. If you have any feedback, please fill out a suggestion form at the bottom of the Resources web page.

Partnering with Practitioners to Address Mental Health in Rural Communities

IES values and encourage collaborations between researchers and practitioners to ensure that research findings are relevant, accessible, feasible, and useful. In 2017, Dr. Wendy Reinke, University of Missouri, received IES funding to formalize the Boone County School’s Mental Health Coalition by strengthening their partnership and validating the Early Identification System (EIS) to screen for social, emotional, behavioral, and academic risk among K-12 students in rural schools. Building on these successes, Dr. Reinke now leads the National Center for Rural School Mental Health (NCRSMH), a consortium of researchers leading efforts to advance mental health screening and support in rural communities.

Bennett Lunn, a Truman-Albright Fellow at IES, asked Dr. Reinke about the work of the original partnership and how it has informed her efforts to build new partnerships with other rural schools around the country. Below are her responses.

 

What was the purpose of the Boone County Schools Mental Health Coalition and what inspired you to do this work?

In 2015, our county passed an ordinance in which a small percent of our sales tax is set aside to support youth mental health in our community. As a result, the schools had visits from many of the local mental health agencies to set up services in school buildings. The superintendents quickly realized that it would be wise to have a more coordinated effort across school districts. They formed a coalition and partnered with researchers at the University of Missouri to develop a comprehensive model to prevent and intervene in youth mental health problems. The enthusiasm of our school partners and their willingness to consider research evidence to inform the model was so energizing! We were able to build a multi-tiered prevention and intervention framework that uses universal screening data to inform supports. In addition, we were awarded an IES partnership grant to help validate the screener, conduct focus groups and surveys of stakeholders to understand the feasibility and social validity of the model, and determine how fidelity to the model is related to student outcomes. The EIS is now being used in 54 school buildings across six school districts as part of their daily practice. 

 

Were there advantages to operating in a partnership to validate the screener?  

The main benefit of working in partnership with school personnel is that you learn what works under what circumstances from those directly involved in supporting students. We meet every month with the superintendents and other school personnel to ensure that if things are not working, we can find solutions before the problems become too big. We vote on any processes or procedure that were seen as needing to change. The meeting includes school personnel sharing the types of activities they are doing in their buildings so that others can replicate those best practices, and we meet with students to get their perspectives on what is working. In addition, the university faculty bring calls for external funding of research to the group to get ideas for what types of research would be appropriate and beneficial to the group. Schools are constantly changing and encountering new challenges. Being close to those who are working in the buildings allows for us to work together in forming and implementing feasible solutions over time.

 

What advice do you have for researchers trying to make research useful and accessible to practitioners? 

Be collaborative and authentic. Demonstrate that you are truly there to create meaningful and important changes that will benefit students. Show that your priority is improving outcomes for schools and students and not simply collecting data for a study. These actions are vital to building trust in a partnership. By sharing the process of reviewing data, researchers can show how the research is directly impacting schools, and practitioners have an opportunity to share how their experience relates to the data. A good way to do this is by presenting with practitioners at conferences or collaboratively writing manuscripts for peer reviewed journals. For example, we wrote a manuscript (currently under review) with one of our school counselor partners describing how he used EIS data in practice. Through collaboration like this, we find that the purpose and process of research becomes less mysterious, and schools can more easily identify and use practices that are shown to work. In this way, long-term collaboration between partners can ultimately benefit students!

 

How does the work of the original partnership inform your current work with the National Center for Rural School Mental Health? 

We are bringing what we have learned both in how to be effective partners and to improve the model to the National Center for Rural School Mental Health. For instance, we are developing an intervention hub on our Rural Center website that will allow schools to directly link evidence-based interventions to the data. We learned that having readily available ideas for intervening using the data is an important aspect of success. We have also learned that schools with problem solving teams can implement the model with higher fidelity, so we are developing training modules for schools to learn how to use the data in problem solving teams. We will be taking the comprehensive model to prevent and intervene with youth mental health and using it in rural schools. We will continue to partner with our rural schools to continuously improve the work so that it is feasible, socially valid, and important to rural schools and youth in those schools.


 

Dr. Wendy Reinke is an Associate Vice Chancellor for Research at the University of Missouri College of Education. Her research focuses on school-based prevention interventions for children and youth with social emotional and behavioral challenges.

Written by Bennett Lunn (Bennett.lunn@ed.gov), Truman-Albright Fellow, National Center for Education Research and National Center for Special Education Research

Highlighting the Science of Learning at the 2021 ED Games Expo

Research on how people learn is critical for informing the design of effective education technology products. To design products that improve student learning, we need to understand how students approach solving problems, the information they need to adopt optimal solution strategies, the skills that underlie success in particular academic domains, the best ways to arrange information on a screen to guide student attention to relevant information, and the best study strategies for optimizing learning and retention. Through its research grants programs, IES has invested in research projects to develop and test education technology products based in the science of learning.

 

The 2021 ED Games Expo, which takes place virtually from June 1-5, features a number of these products, and you can learn more about them through a mix of live and pre-recorded sessions and videos. Most are ready to demo now. Students and educators can send questions about the research, game-play, or ed tech experience directly to the participating researchers. Here are just a few of the many ways you can interact with IES-funded researchers at the Expo:

 

  1. The Virtual Learning Lab (VLL) will be hosting a live, virtual session on Friday, June 4th from 4:00-5:30pm Eastern Time to celebrate their 5-year research collaboration to explore precision education in the context of algebra instruction. The VLL developed an AI-powered video recommendation system that personalizes math instruction within Math Nation. The researchers measured student ability and engagement, detected effects of virtual learning environment usage on achievement, and identified characteristics of effective online tutoring. The session will feature short talks and opportunities for Q&A: 
  • A Video Recommendation System for Algebra (Walter Leite, University of Florida)
  • Reinforcement Learning for Enhancing Collaborative Problem Solving (Guojing Zhou, University of Colorado Boulder)
  • Natural Language Processing for VLE Research (Danielle McNamara, Arizona State University)
  • Identifying Pedagogical Conversational Patterns in Online Algebra Learning (Jinnie Shin, University of Florida)
  • Scaling Items and Persons for Obtaining Ability Estimates in VLEs (A. Corinne Huggins-Manley, University of Florida)
  • Measuring Student Ability from Single- and Multiple-Attempt Practice Assessments in VLEs (Ziying Li, University of Florida)
  • Detecting Careless Responding to Assessment Items in VLEs (Sanaz Nazari, University of Florida)
  • Personalization, Content Exposure, and Fairness in Assessment (Daniel Katz, University of California, Santa Barbara)
  • Fair AI in VLEs (Chenglu Li, University of Florida)

 

  1. The ED Games Expo YouTube Playlist, which will be posted on June 1st on the event page, features 26 products developed with IES grant funding. For ed tech products informed by research on how people learn, check out the products funded through the Cognition and Student Learning topic:
  • Graspable Math allows math teachers to assign interactive algebra tasks and turns equations into tangible objects that middle school and high school students can manipulate to practice and explore. Teachers can follow live, step-by-step, student work.
  • eBravo Boulder Reading Intervention is a self-paced personalized reading comprehension curriculum that teaches secondary students the problem-solving skills good readers use to learn from challenging texts, in this case in the science discipline of ecology. Reading strategies and exercises are guided by well-researched models of reading comprehension, helping students build deep, durable, and reusable knowledge from text. 
  • iSTART and Writing Pal are interventions designed for middle school students, high school students, and young adults improve their reading and writing skills. Within these interventions, students play games to practice reading comprehension and writing strategies.
  • All You Can Eat, Gwakkamole, and CrushStations are part of a suite of Executive Function skill-building games, designed to improve student shifting, inhibitory control, and working memory respectively.

 

We hope you can join us for this exciting event in June to learn more about and try out all the research-based products ready to be used in our nation’s schools. For more information on the featured resources and online events, please see this blog.


Written by Erin Higgins (Erin.Higgins@ed.gov), Program Officer for the Cognition and Student Learning program, National Center for Education Research

Towards a Better Understanding of Middle-Schoolers’ Argumentation Skills

What is the difference between fact and opinion? How do you find relevant evidence and use it to support a position? Every day, teachers help students practice these skills by fostering critical discussions, a form of argumentation that encourages students to use reasoning to resolve differences of opinion.

In their IES-funded study, Exploring and Assessing the Development of Students' Argumentation Skills, Yi Song and her colleagues are uncovering activities (both teacher led and technology supported) that can improve middle-school students’ ability to generate better oral and written arguments.

This project began in 2019 and is working in classrooms and with teachers and students. The researchers have created a series of videos that describe their work. In this series, Dr. Song and her co-PIs, Dr. Ralph Ferretti and Dr. John Sabatini, discuss why the project is important to education, how they will conduct the research plan, and how educators can apply what they are learning in classrooms.

 

 


For questions and more information, contact Meredith Larson (Meredith.Larson@ed.gov), Program Officer, NCER

"Boys Have It; Girls Have to Work for It": Examining Gender Stereotypes in Mathematics Achievement

In 2020, Andrei Cimpian, along with co-PIs Sapna Cheryan, Joseph Cimpian, and Sarah Lubienski, were awarded a grant for “Boys Have It; Girls Have to Work for It”: The Development and Consequences of Gender Stereotypes About Natural Talent vs. Effort in Mathematics. The goal of this project is threefold: 1) to explore the origins of the gender stereotype that girls achieve in math due to effort and boys achieve in math due to natural talent, 2) to investigate the consequences of these stereotypes, and 3) to identify ways of reducing the negative effects of these stereotypes on mathematics outcomes. In this blog, we interviewed Dr. Andrei Cimpian on his inspiration and insights on this research, as well as his plans to disseminate the findings to education practitioners.

 

Dr. Andrew CimpianWhat spurred your research, and what prior research was foundational for this current study? 

The co-PIs and I were inspired to do this research because we were struck by the contrast between two sets of facts. On the one hand, girls do better in school than boys from kindergarten to grade 12. Women also obtain more bachelor’s and graduate degrees than men. On the other hand, we as a society still think of men as more brilliant and genius-like than women. For example, participants in a 2018 study referred more male than female acquaintances for a job that they were told requires natural smarts1. When the same job was said to require a strong work ethic instead, participants referred equal numbers of women and men.

Of course, societal views of women and men have changed quite a bit over the last century. With respect to general competence, women are now equal with men in the eyes of the American public. But the stereotype that associates “raw,” high-level intellectual talent with men more than women seems to have resisted change. Why?

Our research is testing a promising hypothesis: It is possible that people give different explanations for women’s and men’s intellectual successes, explaining men’s competence as being due primarily to their inborn intellectual talent and women’s as being due to their efforts. This effort-vs.-talent stereotype “explains away” women’s achievements by attributing them to a quality—perseverance—that is less valued in American culture than natural ability.

Versions of this explanatory stereotype have been documented in adults, but our project will provide its first systematic investigation among children. In particular, we will investigate the effort-vs.-talent stereotype in the domain of mathematics because innate ability is particularly valued in this domain1, which might make this stereotype especially consequential.

 

What are some examples of language or behavior that might suggest an individual holds a particular stereotype? Are there potential ways of mitigating the negative effects of stereotypes?

The best example of this stereotype that I can think of—and this is in fact the anecdote that crystallized our team’s interest in this topic—was recounted by co-PI Joseph Cimpian in a recent piece for The Brookings Institution (emphasis is mine):

About five years ago, while Sarah [Lubienski] and I were faculty at the University of Illinois, we gathered a small group of elementary teachers together to help us think through […] how we could intervene on the notion that girls were innately less capable than boys. One of the teachers pulled a stack of papers out of her tote bag, and spreading them on the conference table, said, “Now, I don’t even understand why you’re looking at girls’ math achievement. These are my students’ standardized test scores, and there are absolutely no gender differences. See, the girls can do just as well as the boys if they work hard enough.” Then, without anyone reacting, it was as if a light bulb went on. She gasped and continued, “Oh my gosh, I just did exactly what you said teachers are doing,” which is attributing girls’ success in math to hard work while attributing boys’ success to innate ability. She concluded, “I see now why you’re studying this.”

In terms of what can be done to mitigate the effects of this stereotype, our project will investigate a potential strategy: normalizing effort by making it clear to students that everyone (not just particular groups) needs to work hard to learn math. This message reframes what is viewed as necessary for success in math away from the belief that natural talent is key, thereby undercutting the power of effort-vs.-talent stereotypes.

 

The current study focuses on elementary school students in grades 1 through 4. What was the motivation for choosing this specific age group?

In general, gender stereotypes about intellectual ability seem to emerge quite early. For instance, girls as young as 6 and 7 are less likely than boys to associate being “really, really smart” with members of their own gender. For this reason, we think it is really important to focus on young children—we need to understand when effort-vs.-talent stereotypes first take root!

“Catching” these stereotypes when they first arise is also important for intervention purposes. If left unchecked, the effects of effort-vs.-talent stereotypes may snowball over time (for example, differences in the types of careers that young women and men are motivated to pursue).

 

What plans do you have to disseminate the findings of this research in ways that will be useful for education practitioners? 

We are mindful of the importance of getting this research into the hands of teachers so that they can use it in practice. We hope to write articles on this work for media outlets that draw educationally oriented audiences. To reach parents as well, we will coordinate with popular media outlets to disseminate the results of this work to general audiences. More generally, we will make every effort to ensure that the findings have maximal societal impact, raising awareness of effort-vs.-talent stereotypes among parents, educators, and the general public.

 


Andrei Cimpian, PhD (@AndreiCimpian), Professor of Psychology at New York University, has conducted extensive research on children’s conceptual development, explanations, and motivation in school.

Written by Yuri Lin (ylin010101@g.ucla.edu), intern for the Institute of Education Sciences.

Photo credit: Brian Stauffer


1The full PDF and resources are available at https://www.cimpianlab.com/motivation.