Inside IES Research

Notes from NCER & NCSER

CTE Research Is Flourishing at IES!

Since its inception in 2017, the CTE portfolio in the National Center for Education Research (NCER) at IES has grown to 11 research grants and a research network! Several other CTE-related grants have been funded under other topics, such as “Postsecondary/Adult Education” and “Improving Education Systems” in the education research grants program, and in other grant programs such as “Using SLDS to Support State Policymaking.” Two CTE-related grants under the latter program were awarded in FY21—

The newest grants funded in FY21 in the CTE topic of the Education Research Grants program include—

As a causal impact study, the last project (on Virtual Enterprises) has been invited to join NCER’s CTE Research Network as its sixth and final member. Funded in 2018 to expand the evidence base for CTE, the CTE Research Network (led by PI Kathy Hughes at the American Institutes for Research) includes five other CTE impact studies (one project’s interim report by MDRC was recently reviewed by the What Works Clearinghouse and was found to meet standards without reservations). You can read more about the network’s mission and each of its member projects here.  

On AIR’s CTE Research Network website, you can find several new resources and reports, such as: 

The CTE Research Network has also been conducting training, including workshops in causal design for CTE researchers and online modules on data and research for CTE practitioners, shared widely with the field by a Network Lead partner, the Association for Career and Technical Education (ACTE). 

Last but certainly not least, if you are interested in getting your CTE project funded by IES, see the new FY22 research grant opportunities on the IES funding page. To apply to the CTE topic in the Education Research Grants program specifically, click on the PDF Request for Applications (ALN 84.305A). Contact Corinne Alfeld with any questions you might have.


Written by Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER Program Officer 

 

New Project Exploring Adult Basic Skills in STEM-Related Postsecondary CTE®

In celebration of CTE® (career and technical education) month, we would like to highlight the launch of an NCER project that aims to help us understand how to best support adults seeking additional CTE education and training.

Through their exploratory project, Literacy, Numeracy, and Problem-Solving Skills in Technology-Rich Environment in the STEM-Related Subbaccalaureate Programs in the United States, researchers will use a mixed-method design to gather information about the distribution of basic skills (literacy, numeracy, and problem solving) for adults in STEM occupations and students enrolled in STEM-related sub-baccalaureate programs at community colleges. Their goal is to help identify the needs of students and the programming practices at community colleges that may promote basic skill development in STEM programs.

The team will be leveraging data from the Program for the International Assessment of Adult Competencies (PIAAC) to survey the distribution of skills and abilities across a nationally representative sample of adults in STEM fields. They will also be collecting primary data from adults and programs in multiple locations, including Indiana, Ohio, and Washington states.

 

 

To help inform the public about their project, the researchers have created a short YouTube video for the public. This project began in July 2020 and may have initial results ready as early as late 2021.

 


Written by Meredith Larson (Meredith.Larson@ed.gov), Program Officer for Postsecondary and Adult Education, NCER.

Facilitating Causal Research in CTE: Notes from the Network

The Career and Technical Education (CTE) Research Network just entered its third year, and it is time to share recent accomplishments with the IES community! As a reminder, the CTE Research Network (led by the American Institutes for Research, or AIR) was created to expand the evidence base on the impact of CTE programs on student outcomes using causal research methods.

 

Research

In June 2020, AIR released a preliminary report of CTE sites that are ready for causal evaluation. Designed to support researchers interested in studying the impact of CTE on student outcomes, the report details the history, theory of change, student enrollments, and other information for 4 selected CTE sites around the country. The Network hopes that researchers will use the information in the report to design evaluation studies of these programs.

In July 2020, a fifth research project joined the network. A team from MDRC, led by Rachel Rosen, was recently funded by IES to study the Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance. In partnership with Communities in Schools (CIS), the study will use a three-arm, school-level random assignment research design (RCT) to assess the effects of Navience and YouScience on students' self-expressed attitude and interest in career pathways, CTE course taking patterns, and engagement with and progress towards graduation. We welcome the team to the Network and look forward to learning whether and how these advising tools influence student thinking about career options, choice of relevant CTE coursework and work-based learning options, and decisions about CTE concentration in available pathways and programs of study.

A small group of researchers from different Network teams collaborated on and recently released a technical working paper on counterfactuals in CTE. It can be challenging to identify comparison groups for CTE students because it is an elective into which they self-select. The paper describes a variety of rigorous methods of comparing CTE students to valid counterparts and provides case studies that illustrate how to use these methods.

 

Training

The CTE Research Network is committed to increasing the number of researchers trained to study CTE using causal methods. It is notoriously challenging to isolate the effects of CTE from other influences on student outcomes. In August 2020, the Network hosted 18 researchers for a week-long virtual summer training institute on how to design a causal study to examine the impact of CTE. During the week, participants learned how to implement randomized-control trials (RCTs), regression discontinuity designs (RDDs), and comparative interrupted time series (CITS) in a CTE context. After learning about each method, participants worked in small groups to apply the method to real data and had access to the instructors to ask questions. The feedback about the training was overwhelmingly positive. The lecture portions of the training will be posted soon to the training page of the Network’s website. Another week-long training institute will be held in summer 2021 (hopefully, in person!)

The Network is currently developing a series of online modules for CTE practitioners and state agency staff to strengthen capacity to access, conduct, understand, and use CTE research. There will be a presentation to preview the modules at the Association of Career and Technical Education (ACTE) Best Practices and Innovations Conference on October 9, 2020 and a longer and more in-depth session at ACTE’s Career Tech VISION conference the first week of December. These practitioner training modules will be available for free on the website in late Fall 2020.

 

Leadership and Dissemination

The CTE Research Network is regularly updating its resources page with publications of interest to the CTE research field. The most recent is a report of findings from MDRC’s study of P-Tech high schools. The Network’s equity workgroup (a group of researchers from across the Network’s member projects) also published a popular blog this summer on applying an equity lens to CTE research. The Network also posts outside resources such as a REL self-study tool on career readiness and evaluation reports from other researchers.

 


For more about the CTE Research Network, you can sign up to receive the Network’s quarterly newsletter at the bottom of their website’s home page and follow them on Twitter (@CTEResNetwork) and LinkedIn.

If you are interested in learning more about the CTE Research Network, contact the Director, Kathy Hughes (khughes@air.org).

If you are interested in discussing CTE research opportunities at IES, contact Corinne Alfeld (Corinne.Alfeld@ed.gov).

Cost Analysis in Practice (CAP) Project Provides Guidance and Assistance

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

 

STAGES FOR CONDUCTING A COST ANALYSIS

 

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

 

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

 

*More resources available here.


The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

Equity: Alignment of Mission and Methods

Editor's Note: The following post was originally posted on the IES-funded CTE Research Network. The grantee has given us permission to post it on the IES blog.

Funded in 2018 by the Institute of Education Sciences (IES), the Career and Technical Education (CTE) Research Network aims to conduct and promote high-quality casual studies examining the impact of career and technical education. Aligned with the theme of the January 2020 IES Principal Investigators Meeting – Closing the Gaps for All Learners – the Network’s activities include working to deepen the field’s understanding of issues of equity and inequity in CTE research and evaluation.

 

The importance of understanding equity in CTE research

The Wisconsin Department of Public Instruction defines equity in the following way:

“Every student has access to the educational resources and rigor they need at the right moment in their education across race, gender, ethnicity, language, disability, sexual orientation, family background, and/or family income.”

An explicit focus on equity in CTE is particularly important considering that in the not so distant past, vocational education (a precursor to the term career and technical education, or CTE) often served as the track for youth deemed “unable to learn” or “not college material.” In many cases, vocational education was used to systematically relegate students—many of whom were low-income, Black or African American, Latinx, or American Indian—into low-wage jobs that offered limited opportunities for growth.

Today, the focus of CTE has expanded to include fields in science, technology, engineering and mathematics (STEM) and represents for many young people an opportunity to graduate high school and enter postsecondary education or the labor market with highly valued skills and certifications in numerous fields. As CTE has evolved, participation has become associated with a variety of positive outcomes. For example, researchers have found that CTE course taking is associated with higher high school graduation and postsecondary enrollment rates, higher labor market earnings, and better overall student outcomes.

While these positive CTE outcomes are promising, there is more to understand about the causal outcomes associated with CTE participation, especially among subgroups of students based on race, gender, socioeconomic level, and ability status. IES and the CTE Research Network are committed to deepening the field’s understanding of equity and inequity in CTE studies. Along with acknowledging the pernicious ways in which vocational education has historically been used to discriminate against some students and disaggregating outcome data by student subpopulation (an emphasis in recent Perkins V legislation), the network concludes that at a minimum, engaging in equity-minded research and evaluation requires:

  • Establishing diverse research teams: Research has shown that diversity on teams yields greater innovation, more productivity, and better financial results (Levine, 2020). With these benefits in mind, it is important to be intentional in creating diverse research teams that can bring new perspectives, voices, and approaches to studies that aim to identify, analyze and interpret equity data.
  • Adopting an equity mindset in research and evaluation: To inform the field’s understanding of how CTE may promote or inhibit equitable student outcomes, researchers must commit to recognizing their own biases and examining how those biases may influence their research designs and analyses. An equity mindset also requires capturing and analyzing patterns of inequities that appear in administrative and implementation data.
  • Exploring intersectionality: Adopting an equity mindset—as important for research as is using valid and reliable measures—also requires conducting analyses of CTE outcomes that go beyond merely examining differences between subpopulations. Rather, analyses should also examine intersectionality within subpopulations (for example, by gender and race), which affords the field a more nuanced understanding of how outcomes for members of the same subpopulation may vary by other dimensions of identity (such as gender or ability status). Such analyses can help the field understand what works and for whom—information that can help drive policy and practice.
  • Addressing the systems, policies, and procedures that promote inequities: Inequities do not exist in a vacuum. Thus, it is important to contextualize causal CTE studies, acknowledging how systems, policies, and procedures may create barriers to success for some students. Analyses that take an ecosystems approach—focusing on how the social, economic, and geographic environment shapes outcomes—provide valuable insight into the nature of inequities that exist and how these inequities might be overcome. Equally important is to identify the possible or probable causes of inequities to understand how race, gender, and other variables influence students’ experiences in CTE. Analyses must also extend beyond merely identifying average effect sizes to investigating variation in treatment experiences by subpopulations, an approach that provides valuable insights into how young people in different subpopulations fare relative to their peers in specific contexts. Using data and analysis in this way can provide the evidence needed to support policy recommendations aimed at closing equity gaps and creating the conditions that all students need to transition successfully into adulthood.
  • Engaging the communities that participate in our studies: Because evidence is critical for making data-driven decisions, it is important when designing causal studies to include the participating communities and other stakeholders in the knowledge generation and interpretation processes. These communities and stakeholders can also play an important role in informing researchers’ understanding of the specific causes of inequities identified in study findings. Research should be an inclusive process—the communities being studied and those directly affected by research findings should be included in the planning, implementation, and interpretation of research.
  • Asking what more is needed to promote equity: Embracing equity as a measure of success in education research will take time and will require a significant shift in the way research is conceptualized, designed, and conducted. However, to promote a more just society, it is imperative that researchers keep equity at the center of their work.

Although the CTE Research Network is funded to conduct causal studies, which can play a role in identifying inequities, we realize that other research methods also play a role in deepening the field’s understanding of such inequities. For example, qualitative and implementation research can be used to gain important insight into the contextual factors that shape or reinforce inequities and can also be used to engage stakeholders as informants on the topic. Therefore, building the field’s knowledge of these issues will require employing a range of data collection efforts.

In the meantime, the CTE Research Network is taking the following action steps to continue to advance our equity-minded approach to CTE research:

  • Developing a set of equity questions to consistently consider during network convenings
  • Elevating issues of equity in all network presentations
  • Sharing resources on equity to help network members think critically about how best to bring an equity lens to bear on research and evaluation studies
  • Creating and promoting opportunities to help diversify researchers engaged in causal CTE research

As a network, we believe these research practices will shine a light on (in)equity in CTE. Where inequities exist, we hope our work will inform education policymaking that aims not only to close existing equity gaps but also to prevent the perpetuation of inequities in CTE. We invite other researchers to join us in this effort by taking similar action steps as part of their own research and evaluation endeavors. The following resources can inform researchers’ understanding of equity issues in general and in CTE studies in particular:

 

References

Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity perspective in research: Practical guidance for the research process. Washington, DC: Child Trends.

Dougherty, S. M. (2016). Career and technical education in high school: Does it improve student outcomes? Washington, DC: Thomas B. Fordham Institute. https://eric.ed.gov/?id=ED570132

Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2017). Building better bridges to life after high school: Experimental evidence on contemporary career academies. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. https://eric.ed.gov/?id=ED572934

Hodge, E., Dougherty, S., & Burris, C. (2020). Tracking and the future of career and technical education: How efforts to connect school and work can avoid the past mistakes of vocational education. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/cte

Kemple, J. (2008). Career academies: Long-term impacts on work, education, and transitions to adulthood. New York: MDRC. Retrieved from https://www.mdrc.org/publication/career-academies-long-term-impacts-work-education-and-transitions-adulthood

Rosen, R., & Molina, F. (2019). Practitioner perspectives on equity in career and technical education. New York: MDRC. https://eric.ed.gov/?id=ED596458


Written by Equity in CTE Workgroup, on behalf of the CTE Research Network

This is the fifth in a series of blog posts that stems from the 2020 Annual Principal Investigators Meeting. The theme of the meeting was Closing the Gaps for All Learners and focused on IES’s objective to support research that improves equity in access to education and education outcomes. Other posts in this series include Addressing Persistent Disparities in Education Through IES ResearchWhy I Want to Become an Education ResearcherDiversify Education Sciences? Yes, We Can!, and Closing the Opportunity Gap Through Instructional Alternatives to Exclusionary Discipline.