This blog is part of a guest series by the Cost Analysis in Practice (CAP) project team to discuss practical details regarding cost studies.
A few months ago, a team of researchers conducting a large, IES-funded randomized controlled trial (RCT) on the intervention Promoting Accelerated Reading Comprehension of Text-Local (PACT-L) met with the Cost Analysis in Practice (CAP) Project team in search of planning support. The PACT-L team had just received funding for a 5-year systematic replication evaluation and were consumed with planning its execution. During an initial call, Iliana Brodziak, who is leading the cost analysis for the evaluation study, shared, “This is a large RCT with 150 schools across multiple districts each year. There is a lot to consider when thinking about all of the moving pieces and when they need to happen. I think I know what needs to happen, but it would help to have the key events on a timeline.”
The comments and feeling of overload are very common even for experienced cost analysts like Iliana because conducting a large RCT requires extensive thought and planning. Ideally, planning for a cost analysis at this scale is integrated with the overall evaluation planning at the outset of the study. For example, the PACT-L research team developed a design plan that specified the overall evaluation approach along with the cost analysis. Those who save the cost analysis for the end, or even for the last year of the evaluation, may find they have incomplete data, insufficient time or budget for analysis, and other avoidable challenges. Iliana understood this and her remark set off a spark for the CAP Project team—developing a timeline that aligns the steps for planning a cost analysis with RCT planning.
As the PACT-L and CAP Project teams continued to collaborate, it became clear that the PACT-L evaluation would be a great case study for crafting a full cost analysis timeline for rigorous evaluations. The CAP Project team, with input from the PACT-L evaluation team, created a detailed timeline for each year of the evaluation. It captures the key steps of a cost analysis and integrates the challenges and considerations that Iliana and her team anticipated for the PACT-L evaluation and similar large RCTs.
In addition, the timeline provides guidance on the data collection process for each year of the evaluation.
- Year 1: The team designs the cost analysis data collection instruments. This process includes collaborating with the broader evaluation team to ensure the cost analysis is integrated in the IRB application, setting up regular meetings with the team, and creating and populating spreadsheets or some other data entry tool.
- Year 2: Researchers plan to document the ingredients or resources needed to implement the intervention on an ongoing basis. The timeline recommends collecting data, reviewing the data, and revising the data collection instruments in Year 2.
- Year 3 (and maybe Year 4): The iteration of collecting data and revising instruments continue in Year 3 and, if needed, in Year 4.
- Year 5: Data collection should be complete, allowing for the majority of the analysis.
This is just one example of the year-by-year guidance included in the timeline. The latest version of the Timeline of Activities for Cost Analysis is available to help provide guidance to other researchers as they plan and execute their economic evaluations. As a planning tool, the timeline gathers all the moving pieces in one place. It includes detailed descriptions and notes for consideration for each year of the study and provides tips to help researchers.
The PACT-L evaluation team is still in the first year of the evaluation, leaving time for additional meetings and collective brainstorming. The CAP Project and PACT-L teams hope to continue collaborating over the next few years, using the shared expertise among the teams and PACT-L’s experience carrying out the cost analysis to refine the timeline.
Visit the CAP Project website to find other free cost analysis resources or to submit a help request for customized technical assistance on your own project.
Jaunelle Pratt-Williams is an Education Researcher at SRI International.
Iliana Brodziak is a senior research analyst at the American Institutes for Research.
Katie Drummond, a Senior Research Scientist at WestEd.
Lauren Artzi is a senior researcher at the American Institutes for Research.