IES Blog

Institute of Education Sciences

Cost Analysis in Practice (CAP) Project Provides Guidance and Assistance

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

 

STAGES FOR CONDUCTING A COST ANALYSIS

 

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

 

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

 

*More resources available here.


The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

Equity: Alignment of Mission and Methods

Editor's Note: The following post was originally posted on the IES-funded CTE Research Network. The grantee has given us permission to post it on the IES blog.

 

 

Funded in 2018 by the Institute of Education Sciences (IES), the Career and Technical Education (CTE) Research Network aims to conduct and promote high-quality casual studies examining the impact of career and technical education. Aligned with the theme of the January 2020 IES Principal Investigators Meeting – Closing the Gaps for All Learners – the Network’s activities include working to deepen the field’s understanding of issues of equity and inequity in CTE research and evaluation.

The importance of understanding equity in CTE research

The Wisconsin Department of Public Instruction defines equity in the following way:

“Every student has access to the educational resources and rigor they need at the right moment in their education across race, gender, ethnicity, language, disability, sexual orientation, family background, and/or family income.”

An explicit focus on equity in CTE is particularly important considering that in the not so distant past, vocational education (a precursor to the term career and technical education, or CTE) often served as the track for youth deemed “unable to learn” or “not college material.” In many cases, vocational education was used to systematically relegate students—many of whom were low-income, Black or African American, Latinx, or American Indian—into low-wage jobs that offered limited opportunities for growth.

Today, the focus of CTE has expanded to include fields in science, technology, engineering and mathematics (STEM) and represents for many young people an opportunity to graduate high school and enter postsecondary education or the labor market with highly valued skills and certifications in numerous fields. As CTE has evolved, participation has become associated with a variety of positive outcomes. For example, researchers have found that CTE course taking is associated with higher high school graduation and postsecondary enrollment rates, higher labor market earnings, and better overall student outcomes.

While these positive CTE outcomes are promising, there is more to understand about the causal outcomes associated with CTE participation, especially among subgroups of students based on race, gender, socioeconomic level, and ability status. IES and the CTE Research Network are committed to deepening the field’s understanding of equity and inequity in CTE studies. Along with acknowledging the pernicious ways in which vocational education has historically been used to discriminate against some students and disaggregating outcome data by student subpopulation (an emphasis in recent Perkins V legislation), the network concludes that at a minimum, engaging in equity-minded research and evaluation requires:

  • Establishing diverse research teams: Research has shown that diversity on teams yields greater innovation, more productivity, and better financial results (Levine, 2020). With these benefits in mind, it is important to be intentional in creating diverse research teams that can bring new perspectives, voices, and approaches to studies that aim to identify, analyze and interpret equity data.
  • Adopting an equity mindset in research and evaluation: To inform the field’s understanding of how CTE may promote or inhibit equitable student outcomes, researchers must commit to recognizing their own biases and examining how those biases may influence their research designs and analyses. An equity mindset also requires capturing and analyzing patterns of inequities that appear in administrative and implementation data.
  • Exploring intersectionality: Adopting an equity mindset—as important for research as is using valid and reliable measures—also requires conducting analyses of CTE outcomes that go beyond merely examining differences between subpopulations. Rather, analyses should also examine intersectionality within subpopulations (for example, by gender and race), which affords the field a more nuanced understanding of how outcomes for members of the same subpopulation may vary by other dimensions of identity (such as gender or ability status). Such analyses can help the field understand what works and for whom—information that can help drive policy and practice.
  • Addressing the systems, policies, and procedures that promote inequities: Inequities do not exist in a vacuum. Thus, it is important to contextualize causal CTE studies, acknowledging how systems, policies, and procedures may create barriers to success for some students. Analyses that take an ecosystems approach—focusing on how the social, economic, and geographic environment shapes outcomes—provide valuable insight into the nature of inequities that exist and how these inequities might be overcome. Equally important is to identify the possible or probable causes of inequities to understand how race, gender, and other variables influence students’ experiences in CTE. Analyses must also extend beyond merely identifying average effect sizes to investigating variation in treatment experiences by subpopulations, an approach that provides valuable insights into how young people in different subpopulations fare relative to their peers in specific contexts. Using data and analysis in this way can provide the evidence needed to support policy recommendations aimed at closing equity gaps and creating the conditions that all students need to transition successfully into adulthood.
  • Engaging the communities that participate in our studies: Because evidence is critical for making data-driven decisions, it is important when designing causal studies to include the participating communities and other stakeholders in the knowledge generation and interpretation processes. These communities and stakeholders can also play an important role in informing researchers’ understanding of the specific causes of inequities identified in study findings. Research should be an inclusive process—the communities being studied and those directly affected by research findings should be included in the planning, implementation, and interpretation of research.
  • Asking what more is needed to promote equity: Embracing equity as a measure of success in education research will take time and will require a significant shift in the way research is conceptualized, designed, and conducted. However, to promote a more just society, it is imperative that researchers keep equity at the center of their work.

Although the CTE Research Network is funded to conduct causal studies, which can play a role in identifying inequities, we realize that other research methods also play a role in deepening the field’s understanding of such inequities. For example, qualitative and implementation research can be used to gain important insight into the contextual factors that shape or reinforce inequities and can also be used to engage stakeholders as informants on the topic. Therefore, building the field’s knowledge of these issues will require employing a range of data collection efforts.

In the meantime, the CTE Research Network is taking the following action steps to continue to advance our equity-minded approach to CTE research:

  • Developing a set of equity questions to consistently consider during network convenings
  • Elevating issues of equity in all network presentations
  • Sharing resources on equity to help network members think critically about how best to bring an equity lens to bear on research and evaluation studies
  • Creating and promoting opportunities to help diversify researchers engaged in causal CTE research

As a network, we believe these research practices will shine a light on (in)equity in CTE. Where inequities exist, we hope our work will inform education policymaking that aims not only to close existing equity gaps but also to prevent the perpetuation of inequities in CTE. We invite other researchers to join us in this effort by taking similar action steps as part of their own research and evaluation endeavors. The following resources can inform researchers’ understanding of equity issues in general and in CTE studies in particular:

 

References

Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Washington, DC: Child Trends.

Dougherty, S. M. (2016). Career and technical education in high school: Does it improve student outcomes? Washington, DC: Thomas B. Fordham Institute. https://eric.ed.gov/?id=ED570132

Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2017). Building better bridges to life after high school: Experimental evidence on contemporary career academies. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. https://eric.ed.gov/?id=ED572934

Hodge, E., Dougherty, S., & Burris, C. (2020). Tracking and the future of career and technical education: How efforts to connect school and work can avoid the past mistakes of vocational education. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/cte

Kemple, J. (2008). Career academies: Long-term impacts on work, education, and transitions to adulthood. New York: MDRC. Retrieved from https://www.mdrc.org/publication/career-academies-long-term-impacts-work-education-and-transitions-adulthood

Rosen, R., & Molina, F. (2019). Practitioner perspectives on equity in career and technical education. New York: MDRC. https://eric.ed.gov/?id=ED596458


Written by Equity in CTE Workgroup, on behalf of the CTE Research Network

This is the fifth in a series of blog posts that stems from the 2020 Annual Principal Investigators Meeting. The theme of the meeting was Closing the Gaps for All Learners and focused on IES’s objective to support research that improves equity in access to education and education outcomes. Other posts in this series include Addressing Persistent Disparities in Education Through IES ResearchWhy I Want to Become an Education ResearcherDiversify Education Sciences? Yes, We Can!, and Closing the Opportunity Gap Through Instructional Alternatives to Exclusionary Discipline.

 

 

CTE Research Network Identifies Four Sites Ready to be Evaluated

 

In 2018, the IES awarded a grant1 to the American Institutes for Research (AIR) to lead a research network focused on career and technical education (CTE), the Expanding the Evidence Base for Career and Technical Education Network (CTE Research Network). The mission of the CTE Research Network is to increase the number of CTE impact studies and strengthen the capacity of the field to conduct and use rigorous CTE research.

One of AIR’s primary tasks as the CTE Network’s Lead is to conduct an evaluability study (also called a feasibility study) to identify CTE models or programs that could be evaluated using a rigorous experimental design. The purpose of the study is to ease the way for other researchers to evaluate CTE by doing the advance work to find suitable sites that may be interested in participating in research. Any interested research team may approach one of these sites to partner in an evaluation. IES and AIR hope that qualified teams will submit an application to the IES Education Research Grants program, under the CTE topic, for grant funding to conduct an evaluation.

In a preliminary report released today, the CTE Network Lead describes the method they used to identify a broad range of programs and models, the vetting criteria, and the reasons for selecting the four sites. For each of the selected sites, the report also describes the scope of the program and student enrollment, the CTE programs offered, the data available, and the willingness of the sites to welcome researchers to evaluate the CTE program. In addition, the report includes the suggested next steps for researchers and possible limitations in carrying out an evaluation of the particular site or program model.

Prior research on CTE over the last half century has mostly been exploratory in nature or, at best, quasi-experimental. One of the primary reasons for the lack of experimental research is that it is difficult to assign students to elective courses. Even quasi-experimental designs are challenging, as it is difficult to statistically control for all the reasons a student might choose to enroll in CTE. See here and here for further discussion of the challenges in conducting CTE research.

The CTE Research Network has another upcoming effort to help increase the CTE evidence base: a free training on causal methods for CTE research. The training will take place online in August 2020; the deadline for applications is June 30, 2020.

News about the CTE Research Network and resources to help CTE researchers can be found on the Network’s website; IES also occasionally blogs about the research findings of Network members. Although most of the CTE Network members are currently studying CTE at the secondary level, we hope that more research will be conducted at the postsecondary level. Researchers interested in applying to IES for a grant to study CTE are welcome to contact Corinne Alfeld (contact information below).


1Using Perkins funds from the Office of Career, Technical, and Adult Education (OCTAE) in partial fulfillment of the legislative requirement for a national research center to carry out scientifically-based research and evaluation for the purpose of developing, improving, and identifying the most successful methods for addressing the education, employment, and training needs of career and technical education (CTE) participants in CTE programs [Sec. 114(d)(4)].

 

Written by Corinne Alfeld (corinne.alfeld@ed.org), IES program officer, and Katherine Hughes (khughes@air.org), principal investigator for the CTE Network Lead at AIR

 

IES is Providing Digital Technical Assistance for FY 2021 Research Grant Applicants

Given the many challenges that this year has brought, including the difficulties and uncertainties due to the COVID-19 pandemic, IES is providing different resources and options to assist applicants as they begin preparing their applications. To ensure that program officers can focus their time on project-specific questions, applicants should review these resources first before seeking individual feedback.

First, have a copy of the documents that are needed to submit a proposal. Download a copy of the relevant request for applications (RFA) and the IES Application Submission Guide. This page has PDFs of these documents: https://ies.ed.gov/funding/21rfas.asp. Also, download the application package (search for CFDA 84.305) from https://grants.gov/. Contact Grants.gov (1-800-518-4726; support@grants.gov) if you need help with your electronic grant submission.

 

Next, take advantage of our digital technical assistance options.

  • On-demand webinars. These pre-recorded webinars answer questions about the grant competitions, how to apply, and how to prepare a strong application. You can access them here: https://ies.ed.gov/funding/webinars/.  

 

  • Virtual office hours. This year, we will host a series of drop-in hours during which a program officer will answer questions and give technical assistance. These office hours will help determine which competition or project type is the best fit and also understand some of the requirements and recommendations in the RFAs. Please see the schedule below along with the call-in information. This information is also posted here.

 

  • Cost analysis/Cost-effectiveness analysis. Many RFAs require a cost analysis plan, and some also require a cost effectiveness plan.  Please refer to our list of resources for developing these plans: https://ies.ed.gov/seer/cost_analysis.asp.

 

 

Finally, please make sure that you attend to the application due dates: https://ies.ed.gov/funding/futureComp.asp because IES does not accept late applications.

 

Virtual Office Hours

Staff from the research centers will host hour-long drop-in virtual sessions to provide technical assistance around particular competitions or research project types or for general purposes. Applicants are encouraged to join in the discussion and ask questions. These sessions are especially helpful if you are unsure of which competition or project type is the best match for you or if you are unclear on any changes to the requirements or recommendations. Below is a list of the current sessions and their topics. Please attend as many sessions as you would like.

All office hours will use the same call-in details. The program officer will allow participants into the meeting from the “lobby” at the beginning. We recommend you do not use video so that there is sufficient bandwidth. All times are shown in Eastern Standard time.

 

Join Microsoft Teams Meeting

+1 202-991-0393   United States, Washington DC (Toll)

Conference ID: 915 412 787#

 

If you would like to request accommodations (e.g., TTY), please send an email to NCER.Commissioner@ed.gov with this request as soon as possible.

You may have to download a free mobile application to use Microsoft Teams if you want the full audio and visual experience from your phone. Clicking on the linked “Join” hyperlink below should prompt you to do this. You can also refer to this article for information: https://support.microsoft.com/en-gb/office/set-up-your-teams-mobile-apps-1ba8dce3-1122-47f4-8db6-00a4f93117e8

 

 

Virtual Office Hours Schedule

 

 

Monday, June 22

Tuesday, June 23

Wednesday, June 24

Thursday, June 25

12:30 – 1:30 pm ET

Competition fit: this will cover all NCER grant competitions and items such as applicant eligibility, general requirements, submission questions, and the IES review process.

Efficacy/Follow-Up and Replication: this will cover characteristics of high-quality projects of these types.

Exploration projects: this will cover characteristics of high-quality projects of this type.

Development projects: this will cover characteristics of high-quality projects of this type.

2:00 – 3:00 pm ET

Exploration projects: this will cover characteristics of high-quality projects of this type.

Development projects: this will cover characteristics of high-quality projects of this type.

Is 305A (Education Research Grants) right for me? This will address general questions about CFDA 84.305A

Measurement projects: this will cover characteristics of high-quality projects of this type.

 

 

Monday, June 29

Tuesday, June 30

Wednesday, July 1

Thursday, July 2

12:30 – 1:30 pm ET

Development projects: this will cover characteristics of high-quality projects of this type.

Exploration projects: this will cover characteristics of high-quality projects of this type.

Measurement projects: this will cover characteristics of high-quality projects of this type.

 

2:00 – 3:00 pm ET

Competition fit: this will cover all NCER grant competitions and items such as applicant eligibility, general requirements, submission questions, and the IES review process.

Systematic Replication: this will focus on the requirements for a 305R or 324R application

Efficacy/Follow-Up: this will cover characteristics of high-quality projects of this type.

Pathways to the Education Sciences: this will address common questions about this training program opportunity.  

 

How to Seek Funding to Support CTE Research Partnerships

Over the past six months, Advance CTE and IES have worked together to highlight the power of CTE research partnerships in improving quality and equity in CTE. In Michigan, years of close collaboration between the Department of Education and the University of Michigan has enabled state leaders to address critical policy questions like choosing a secondary CTE program quality performance indicator. South Dakota leveraged relationships in the research community to improve data quality and foster a data-driven culture at the state level. And in Massachusetts, state leaders are working alongside long-time research partners to identify critical access and opportunity gaps and build solutions that enable equitable access to high-quality CTE.

Partnerships like these provide measurable benefits by allowing state policymakers to make informed decisions that impact learner success and bolster state talent pipelines – but they do come at a cost. The partnerships highlighted in this series were supported via a combination of state, federal, and foundation funds. Research grant funds are most often used to cover personnel time for work on the research project, both at the university or research organization and at the partner education agency. As many of our state agency interviewees mentioned, it is difficult to carve time out of their regular responsibilities to work on a research project. By securing dedicated funding to cover part or all of a person’s salary, a state agency can afford to spend time on a research project. In addition, research grant funds can be used to provide incentives for students, teachers, and schools to participate in a research study, for the development and administration of surveys or classroom observation tools (to complement information available in administrative data systems), and for software and hardware to analyze and house the data.

With growing public support for CTE, fueled by urgent needs for skilled labor, CTE programs will be called upon to do even more. States should be prepared with a research and evaluation strategy to determine whether and which strategies are most effective (and cost-effective). So how should states go about establishing and funding new CTE research partnerships?  

Options for Financing State CTE Research Partnerships

There are a number of avenues states can take to finance CTE research. Federal sources of funding for CTE-related research include the Department of Education, the Department of Labor, and the National Science Foundation. Education research funding may also be available at other agencies, such as the Department of Agriculture. Private funding for CTE research projects is also available from foundations such as the ECMC Foundation1, the Smith Richardson Foundation, and the Laura and John Arnold Foundation.

The National Center for Education Research (NCER) at IES launched a special CTE topic in its Education Research Grants program in 2017 to encourage researchers to study CTE. Funded grants under this topic have examined CTE-related issues such as industry certifications, applied-STEM (Science, Technology, Engineering and Mathematics) CTE pathways and work-based learning. IES also funds CTE research under other programs and maintains a CTE Statistics webpage. In 2018, in partnership with the Office of Career, Technical, and Adult Education (OCTAE), IES formed a CTE Research Network to increase the amount and quality of causal research in CTE. CTE Research Network members have been studying the impact of various CTE programs and delivery models on student high school, postsecondary and labor market outcomes. The National Center for Special Education Research (NCSER) began funding CTE research for students with disabilities in 2019.

New Opportunity to Apply for Federal Funding to Study CTE!

There is good news for state leaders and researchers interested in initiating CTE research partnerships. NCER has just released its Fiscal Year 2021 Request for Applications (RFA) for its Education Research Grants Program (CFDA 84.305A). This grants program, one of several in NCER, was established in 2002 to produce research that is scientifically rigorous and relevant to the needs of education practitioners and decisionmakers. NCER welcomes CTE-related research proposals under the CTE topic or under other topics (such as STEM, Improving Education Systems, and Postsecondary and Adult Education). NCSER has a separate RFA for its special education research grants program (CFDA 84.324A) and welcomes applications to study CTE for students with disabilities.

Research grant applications are due at midnight (Eastern time) on August 20, 2020. Letters of intent (not required but encouraged) are due on June 11, 2020. Each of the open RFAs, as well as archived webinars for applicants about the IES grant process, are available on the IES funding opportunities page.

Applicants should start early to make sure they have everything they need. In addition to viewing on-demand webinars, applicants should be sure to read the RFA closely and pay attention not only to the application requirements but also to the IES recommendations for a strong application. For example, applicants should describe their theory of change and any prior research on the issue; align their research methods to the research questions; describe measures and data source; and make sure the sample size offers adequate statistical power. This grants program is very competitive, and peer reviewers will be paying attention to whether applicants follow the recommendations. Everyone involved in the submission process should also familiarize themselves with the IES submission guide, which details the steps necessary to successfully submit an application online.

We are eager to hear any and all ideas! Corinne Alfeld (Corinne.alfeld@ed.gov) and Austin Estes (aestes@careertech.org) would be happy to discuss them, and Corinne can also provide technical assistance in writing a research grant application to IES. She can be reached by email to set up a phone call to discuss project ideas.


This final blog post wraps up our series aimed at increasing state Career Technical Education (CTE) research partnerships by highlighting ways to seek research funding. Corinne Alfeld, Research Analyst at the Institute of Education Sciences (IES) and Austin Estes, Manager of Data & Research at Advance CTE, collaborated to create this blog series in the hopes that more state agencies would partner with researchers to examine research questions related to CTE using state data.

 

1The ECMC Foundation is a funder of Advance CTE’s work.