IES Blog

Institute of Education Sciences

Cost Analysis in Practice (CAP) Project Provides Guidance and Assistance

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

 

STAGES FOR CONDUCTING A COST ANALYSIS

 

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

 

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

 

*More resources available here.


The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

IES Grantees Receive SPR Awards

Three IES-funded investigators were presented with awards from the Society for Prevention Research (SPR) last week. We recognize and applaud these investigators (pictured above from left to right) Elizabeth Stormshak, Dorothy Espelage, and Patrick Tolan.

Dr. Elizabeth Stormshak, Philip H. Knight Chair and Department Head for Counseling Psychology and Human Services at University of Oregon, received the Translational Science Award from SPR. This award is given to an individual or a team of individuals in recognition for contributions to the field of prevention science through translational research. Dr. Stormshak’s research focuses on understanding risk factors in early and middle childhood associated with the development of problem behavior in late adolescence, including substance use and delinquency. She also studies the process of disseminating evidence-based interventions into real world community settings. She has been the Principal Investigator on multiple IES grants, including a 2018 NCSER-funded project to examine the long-term efficacy of the Kindergarten Family Check-Up (FCU), a school-based, family-centered intervention intended to prevent student social and behavioral problems. This grant is a follow-up and extension to her recently completed randomized controlled trial of Kindergarten FCU, funded by NCER, which found positive impacts on student behavior and academic outcomes during and up to 3 years after the transition to kindergarten. The primary aims of the newer grant are to determine the long-term impact of receiving the original kindergarten intervention and the effects of a middle school booster session of FCU on students' behavior and academic outcomes. Dr. Stormshak is also the director of a NCSER-funded postdoctoral training program focused on the prevention of school-based social and behavioral problems.

Dr. Dorothy Espelage, William C. Friday Distinguished Professor of Education at the University of North Carolina, Chapel Hill, received SPR’s Prevention Science Award. This award is given for the application of scientific methods to develop and test prevention strategies. Dr. Espelage is a leading expert on school safety and has led multiple research studies on school-based violence and bullying. She has been involved in several IES grants. Most recently, she is serving as the Principal Investigator of a 2019 NCSER-funded project to develop and test a professional development program aimed at enhancing elementary school teachers' knowledge and skills for identifying, mitigating, and preventing bullying among students with and without disabilities. Dr. Espelage also led an exploratory study funded by NCER to better understand how teacher practices influence elementary school students’ interpersonal relationships in the classroom and related behavioral outcomes. 

Dr. Patrick H. Tolan, Charles S. Robb Professor of Education at the University of Virginia in the Curry School of Education and in the Department of Psychiatry and Neurobehavioral Sciences in the School of Medicine, and Director Emeritus of the YouthNex Center in the UVA Center to Promote Effective Youth Development, received the Advances in Culture and Diversity in Prevention Science Award as one of three founding members of the Boys of Color Collaborative. This award is given for contributions to the field of prevention science in the area of community and culture. Dr. Tolan’s research career spans 34 years with a focus on program evaluation for promoting positive youth development and preventing youth violence. Dr. Tolan is currently the Principal Investigator of a 2019 Follow-Up study of an IES efficacy study of the integration of two prevention programs, Good Behavior Game and My Teaching Partner (GBG+MTP), for teachers who have recently entered the teaching profession.

Congratulations to the award recipients!

This blog was co-authored by Jackie Buckley (NCSER), Katie Taylor (NCSER), Emily Doolittle (NCER), and Amy Sussman (NCSER).

 

New Report on Crime and Safety in Schools and on College Campuses

Crime in our nation’s schools and college campuses has generally declined over the past two decades, according to Indicators of School Crime and Safety 2019, a recently released NCES report. This report highlights new analyses of mental health services provided by public schools and the prevalence of school and school neighborhood problems. The report also covers topics such as victimization, school conditions, safety and security measures at school, and criminal incidents at postsecondary institutions.

In 2018, students ages 12–18 experienced 836,100 total victimizations (i.e., thefts and nonfatal violent victimizations) at school and 410,200 total victimizations away from school. These figures represent a rate of 33 victimizations per 1,000 students at school and 16 victimizations per 1,000 students away from school. From 1992 to 2018, the total victimization rate and the rates of specific crimes—thefts and violent victimizations—declined for students ages 12–18, both at school and away from school.

This edition of Indicators of School Crime and Safety examines new data on school shootings. While such events represent a small subset of the violent incidents that occur at schools, they are of high concern to those interested in the safety of our nation’s students. In school year 2018–19, there were 66 reported school shootings with casualties at public and private elementary and secondary schools (29 school shootings with deaths and 37 school shootings with injuries only). Between 2000–01 and 2018–19, the number of school shootings with casualties per year ranged from 11 to 66.

Student bullying was the most commonly reported discipline problem among public schools over the past two decades. In school year 2017–18, about 14 percent of public schools reported that bullying occurred among students at least once a week, representing a decrease from the 29 percent of schools that reported student bullying in 1999–2000. In 2017–18, about 15 percent of public schools reported that cyberbullying had occurred among students at least once a week either at school or away from school.

This edition of the report also contains an analysis of new survey items that asked administrators at schools serving fifth-graders about issues in neighborhoods around their schools. In spring 2016, “crime in the neighborhood” and “selling or using drugs or excessive drinking in public” were the two most commonly reported school neighborhood problems. Thirty-four percent of fifth-graders attended schools where crime in the neighborhood was a problem, and 31 percent attended schools where selling or using drugs or excessive drinking in public was a problem. For the five school neighborhood problems examined in the report, fifth-graders attending schools where these were a big problem or somewhat of a problem consistently had lower scores in reading, mathematics, and science than did those attending schools where these were not a problem.



In addition to reporting data on student victimizations and school safety conditions, Indicators of School Crime and Safety 2019 also includes information on the programs and practices that schools had in place to promote a safe school. The new report includes a special analysis of mental health services provided by public schools. During the 2017–18 school year, 51 percent of public schools reported providing diagnostic mental health assessments to evaluate students for mental health disorders. Thirty-eight percent of public schools reported providing treatment to students for mental health disorders. When asked about whether certain factors limited their efforts to provide mental health services in a major way, 52 percent of public schools reported that inadequate funding was a major limitation, and 41 percent reported that inadequate access to licensed mental health professionals was a major limitation.



The report also looked at safety and security practices. In school year 2017–18, about 92 percent of public schools had a written plan in place for procedures to be performed in the event of an active shooter. Forty-six percent had a plan for procedures in the event of a pandemic disease. Between 2005–06 and 2017–18, the percentage of public schools that reported having one or more security staff present at school at least once a week increased from 42 to 61 percent.

Shedding light on postsecondary campus safety and security, the report shows that the number of reported forcible sex offenses on college campuses increased greatly while the overall number of reported criminal incidents at postsecondary institutions fell. Between 2001 and 2017, the number of reported forcible sex offenses on college campus increased 372 percent (from 2,200 to 10,400 offenses) while the overall number of criminal incidents reported on postsecondary campuses decreased by 31 percent (from 41,600 to 28,900 incidents). However, in the most recent data (between 2016 and 2017), the overall number of criminal incidents reported on postsecondary campuses increased by 2 percent. In 2017, a total of 958 hate crimes were reported on college campuses, of which the most common types were destruction, damage, and vandalism (437 incidents) and intimidation (385 incidents). Race, religion, and sexual orientation were the categories of motivating bias most frequently associated with these hate crimes.

To view the full Indicators of School Crime and Safety 2019 report, please visit https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2020063.

 

By Ke Wang, AIR

Using Mistakes as a Vehicle for Learning in Mathematics: From Research to Practice at Scale in Education

Every student makes mistakes. But not every student is given the opportunity to learn from mistakes. Left unaddressed, the mathematical misconceptions that underlie many mistakes can keep students from progressing in mathematics.

 

At the request of districts in the Minority Student Achievement Network (MSAN), a Strategic Education Research Partnership (SERP) team was convened in 2007 to address a widening achievement gap in Algebra I. The team was charged with identifying an intervention strategy, subject to several district constraints:

  1. The solution would need to be applied to all students in the regular classroom to avoid the stereotype threat associated with separating students based on performance and to protect the intervention from budget cuts that target supplemental, after-school, and summer programs first.
  2. A new curriculum was off the table because it would create upheaval for a time and would be followed by a decline in student performance during the period of adjustment.
  3. Extensive teacher training was considered undesirable because it would be costly and because algebra teachers consider themselves more expert in mathematics teaching than central office staff who would be requiring the training.

 

Julie Booth joined the partnership, and with funding from IES, led the iterative development and testing of worked example assignments that, with the input of teachers and administrators, fit within the routines of the classroom. The result—AlgebraByExample—consists of 42 uniquely designed assignments that address misconceptions, harness the power of explanation, and use mistakes as a vehicle for learning.

Typical math assignments require students to solve problems on their own. If a student’s work is incorrect, the student may never focus on what went wrong. ByExample assignments also give students problems to solve, but they first provide a solution to a similar problem that is marked right or wrong. Students are prompted with questions that target common misconceptions and errors before solving a similar problem on their own. Each assignment contains several strategically designed item pairs:

 

 

Designed in collaboration with teachers from districts in several states, the assignments can be easily incorporated into any Algebra I curriculum and teachers can choose in what way and in what order to use them. The assignments were tested in randomized trials in classrooms in eight districts with more than 6,000 students. Not only did students using AlgebraByExample improve an average of 7 percentage points on an assessment of standardized test items, students at the lower end of the distribution improved the most. The PDF downloads of the assignments are freely available for anyone to use.

The success of AlgebraByExample  led to  further IES funding of MathByExample for Grades 4 and 5 and GeometryByExample for high school geometry .

 

Resources:

AlgebraByExample website

MathByExample website

Booth et al, 2015

NSF Stem for All Video Submission 2019

 

Interview with Dr. Suzanne Donovan (SERP), Dr. Julie Booth (Temple University), and Allie Huyghe (SERP), the developers of the ByExample interventions.

 

 

Was it part of the original plan to develop an intervention that could one day be used at scale in schools?

Yes. SERP partnerships begin with problems of practice nominated by district partners, but the partnership agreement distinguishes SERP from a consultant. The intention from the start is to frame the problem and design a solution that can be used at scale. SERP has developed in-house, user-centered design expertise so that resources (such as the ByExample products) developed through partnerships meet the needs of teachers and students. Products scale when they improve the experience of teachers and students. Both the model and the internal design capacity allow SERP to move from problem framing through research, development, and dissemination of a product with IES grant funding.

 

Describe the initial research and development that occurred.

Dr. Julie Booth drafted initial assignments drawing on the mathematics misconceptions literature. SERP held regular partnership meetings with teachers and administrators at which assignments were reviewed and additional misconceptions were nominated for attention in the assignments. Administrators agreed to randomization of the assignments across classrooms and within-teacher. Assignments were first tested in individual topic blocks and revised in accordance with student performance data, observations, and teacher feedback. A year-long pilot study was then conducted using the full set of assignments.

 

Beyond IES or ED grants, what additional funding was needed to develop the intervention?

For the ByExample work, additional funding was provided by the Goldman Sachs Foundation in the initial phase to support partnership formation, problem framing, and the solution generation. IES grants funded the research and development, along with initial dissemination activities to make the materials available to the public. We were also able to develop an online platform to allow for digital use with the IES grant funds.

 

What model was used for dissemination and sustainability?

The assignments are available as free downloads on SERP’s website, and as printed workbooks through SERP’s partner print-on-demand company. They have been publicized through online communications, journal articles, presentations at conferences of various types, social media, and word of mouth. There will be a small fee for use of the digital platform to support its maintenance, but the PDFs will remain as free downloads. We have been able to sustain the collaboration of the partnership team by responding to requests from educators to expand the approach to other grade levels and submitting additional proposals to IES that have been awarded.

 

What advice would you provide to researchers who are looking to move their research from the lab to market? What steps should they take? What resources should they look for?

First, I would note that it is difficult to persuade educators to use a product that solves a problem they don’t believe they have. Listen to educators and apply research expertise to address the challenges that they experience on a day-to-day basis. Design for ease of use by teachers. No matter how good your strategy or marketing is, if it’s too much work for an already busy teacher to use, you may get uptake by a few committed teachers, but not at scale. Finally, pay attention to where teachers get their information. For AlgebraByExample, we got a big boost from the Marshall Report, produced by a teacher for other teachers to call attention to usable research.  

 

In one sentence, what would you say is most needed for gaining traction and wide scale use by educators?

Design for the routines of the classroom.

 


Suzanne Donovan, PhD, is the founding Executive Director of the SERP Institute, an education research, development, and implementation organization incubated at the National Academies. SERP leads collaborations of educators, researchers, and designers to generate research-based, scalable, and sustainable solutions to critical problems of practice. 

Julie Booth, PhD, is a Professor of STEM Education and Psychology and the Deputy Dean of Academic and Faculty Affairs at Temple University’s College of Education and Human Development. Her work focuses on translating between cognitive science and education to better understand students’ learning and improve instruction, primarily in mathematics education. She is currently an Executive Editor for the Journal of Experimental Education.

Allie Huyghe is the Assistant Director of the SERP Institute, where she manages several projects, including the IES-funded MathbyExample and GeometryByExample projects. She is also intricately involved with other SERP areas of work, participating in the design of materials from early development through release to the public.

 

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov) and Edward Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This is the fifth in an ongoing series of blog posts examining moving from university research to practice at scale in education.​