Inside IES Research

Since its inception in 2017, the CTE portfolio in the National Center for Education Research (NCER) at IES has grown to 11 research grants and a research network! Several other CTE-related grants have been funded under other topics, such as “Postsecondary/Adult Education” and “Improving Education Systems” in the education research grants program, and in other grant programs such as “Using SLDS to Support State Policymaking.” Two CTE-related grants under the latter program were awarded in FY21—

• The Distributional Effects of Secondary Career and Technical Educational (CTE) Programs on Postsecondary Educational and Employment Outcomes: An Evaluation of Delaware's CTE Programs of Study (PI: Luke Rhine, Delaware)
• Analyzing and Understanding the Educational and Economic Impact of Regional Career Pathways (PI: Jonathon Attridge, Tennessee)

The newest grants funded in FY21 in the CTE topic of the Education Research Grants program include—

• College and Career Readiness:  Investigating California's Efforts to Expand Career Technical Education Through Dual Enrollment (PI: Michal Kurlaender, University of California at Davis) 
• Postsecondary and Labor Market Effects of Career and Technical Education in Baltimore City Public Schools (PI: Marc Stein, Johns Hopkins University) 
• SREB Career and Technical Education Leadership Academy Study (PI: James Stone, SREB)
• Sub-baccalaureate Career and Technical Education: A Study of Institutional Practices, Labor Market Demand, and Student Outcomes in Florida (PI: Angela Estacion, WestEd)
• An Experimental Evaluation of the Efficacy of Virtual Enterprises (PI: Fatih Unlu, RAND) 

As a causal impact study, the last project (on Virtual Enterprises) has been invited to join NCER’s CTE Research Network as its sixth and final member. Funded in 2018 to expand the evidence base for CTE, the CTE Research Network (led by PI Kathy Hughes at the American Institutes for Research) includes five other CTE impact studies (one project’s interim report by MDRC was recently reviewed by the What Works Clearinghouse and was found to meet standards without reservations). You can read more about the network’s mission and each of its member projects here.  

On AIR’s CTE Research Network website, you can find several new resources and reports, such as: 

• Ready for Causal Research: A National Evaluability Assessment of Career and Technical Education Programs (Final Report). This report describes four CTE programs identified as most immediately viable for causal research—and all these programs are willing to be studied!  
• Identification and Counterfactuals for Program Evaluation of Career and Technical Education. Identifying a control group, or counterfactual, is tricky in a subject like CTE into which students self-select. To guide researchers, this paper presents a series of case studies assessing the counterfactual for a selection of experimental and quasi-experimental CTE program evaluations. 
• Improving Measurement in Career and Technical Education to Support Rigorous Research. This paper catalogs the measures of CTE exposure that prior studies have used, documents measures and outcomes in more recent causal studies, and offers a framework for how to measure CTE exposure and outcomes in the future. 

The CTE Research Network has also been conducting training, including workshops in causal design for CTE researchers and online modules on data and research for CTE practitioners, shared widely with the field by a Network Lead partner, the Association for Career and Technical Education (ACTE). 

Last but certainly not least, if you are interested in getting your CTE project funded by IES, see the new FY22 research grant opportunities on the IES funding page. To apply to the CTE topic in the Education Research Grants program specifically, click on the PDF Request for Applications (ALN 84.305A). Contact Corinne Alfeld with any questions you might have.

Written by Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER Program Officer 

February marks Career and Technical Education Month^®, a public awareness campaign that highlights and celebrates career and technical education (CTE). CTE programs emphasize career preparation, skill trades, applied sciences, and modern technologies. CTE coursework integrates academic knowledge with post-secondary pathways and careers by directly preparing middle and high school students with coursework related to high-demand industries.

To encourage research to improve career readiness skills and transition outcomes for students with or at risk for disabilities, the National Center for Special Education Research (NCSER) competed a special topic on Career and Technical Education for Students with Disabilities in FY 2019 and FY 2020. A previous blog post focused on the FY 2019 grant. In FY 2020, NCSER awarded two new research grants through this special topic:

CTE Teachers and Long-Term Outcomes for Students with Disabilities

The purpose of this exploratory project is to assess CTE teacher effectiveness for students with disabilities. Principal Investigator Dan Goldhaber at the University of Washington and his colleagues (co-PIs Kristian Holden and Roddy Theobold) will estimate CTE teacher effectiveness using high school attendance, GPA, persistence, and graduation probability as outcomes. They will also consider longer-term outcomes including college enrollment and employment, as well as whether CTE teacher effectiveness varies according to teacher licensure, pathway into teaching, and prior work experiences.

Supported College and Career Readiness for Secondary Students with Emotional and Behavioral Problems

In this study, Principal Investigator Lee Kern at Lehigh University and her colleagues (co-PIs Chris Liang and Jennifer Freeman) will develop and pilot test a multi-component program, Supported College and Career Readiness, that augments typical school-based college and career readiness activities (such as those associated with CTE). The research team aims to further support high school age students with or at risk for emotional and behavioral disorders (EBD), whom research suggests are insufficiently benefiting from college and career readiness activities. As a result, students with or at risk for EBD are frequently unprepared for career or postsecondary education pathways.

We asked each Principal Investigator to share the motivation for studying this research topic and what makes their study unique and impactful.

What inspired you to study this research topic?

Dr. Goldhaber: At CALDER, we have contributed to a large body of research on the impacts of math and ELA teachers on students. This body of evidence suggests that the quality of the teacher workforce is the most important schooling factor influencing students test and non-test outcomes. But we were surprised to find that, despite growing interest in CTE, there is a surprising lack of empirical research on CTE teacher contributions to student learning. We think this is a particularly important issue for students with disabilities for two reasons. First, CTE teachers have different pathways to teaching than academic teachers, and these pathways may provide less pedagogical preparation for the unique needs of students with disabilities. And second, students with disabilities are significantly more likely to be enrolled in these courses relative to students without disabilities. As such, this project lies at the intersection of our prior work on academic teachers, and our recent work on CTE participation for students with disabilities.

Dr. Kern: Spending time with high school age students with emotional and behavioral problems brought me to truly understand how little they consider their futures. And, even when they have goals and dreams, they see scarce connections between what they are being asked to do in high school and realizing their goals. It occurred to me that finding ways to strengthen this connection might be an avenue not only to better prepare students for post-high school life, but also to reduce dropout. Indeed, the need to better prepare youth for life after high school has been increasingly embraced, especially in the last decade, evidenced in part by the adoption of career and college readiness (CCR) standards in almost every U.S. state. I witnessed many impressive CCR efforts and programs in high schools, yet it appeared that students with emotional and behavioral problems were failing to access these school-based CCR supports, some of which did not seem well aligned with their needs. With support from a Lehigh University faculty grant, I began to research exactly how this subset of students regarded, accessed, and benefitted from CCR activities and supports. This research underscored the need to supplement important components of CCR to better align with the needs of students with emotional and behavioral problems. Our goal in the current project is to develop and evaluate an intervention package that supplements high school CCR activities to better prepare students with emotional and behavioral problems for community, college, and/or career.

What makes your project unique and exciting? 

Dr. Goldhaber: This project will use unique data from Washington state that allows us to track students with disabilities in high school, postsecondary education, and employment. Access to long-term outcomes is both unique and important given that test scores are not likely to be a very good measure of the contributions that CTE teachers make toward student education. Our project is also closely focused on CTE workforce issues. The staffing of CTE courses is quite challenging—in Washington state, for instance, over half of all CTE teachers have not completed the state’s teacher licensure requirements and hold limited CTE licenses. Thus, we believe the work will garner a lot of attention from policymakers.

Dr. Kern: I am especially enthusiastic about this project because it targets areas in which empirical research tells us that students with emotional and behavioral problems have insufficient CCR skills, perceptions, and knowledge. I am also fortunate to work with my co-PIs, Drs. Freeman and Liang, who bring diverse and unique expertise in the areas of CCR assessment, CCR counseling, racial/ethnic identity development, and more. I am also optimistic about the feasibility of this project because it capitalizes on existing school resources in the form of school-based CCR programs. So, rather than adding programs or interventions, it adapts and expands existing CCR components.

Tell us how your research results could possibly shape CTE-related policy.

Dr. Goldhaber: In broad terms, we do not know much about the impact of CTE teacher quality, so our study has the potential to inform broadly the myriad ways we think about the heterogeneity of CTE teacher effects. But, more specifically and narrowly, we will be looking at the connections between CTE effects and CTE teacher pathways, which should inform policies and practices around CTE preparation and licensure.

Dr. Kern: We hope that our efforts will shape future CCR policy. Our aim is to build the evidence-base in the area of CCR supports that address the needs of students at risk for emotional and behavioral problems. Ultimately, we would like to see federal policies that guarantee at-risk students in all high schools in the U.S. receive comprehensive, evidence-based CCR interventions and supports that fully prepare them for life after high school.

For more information on CTE and students with disabilities, the Association for Career and Technical Education (ACTE) and the National Technical Assistance Center on Transition (NTACT) provide useful resources. In addition, ACTE, NTACT, and Penn State University’s Workforce Education program hosted a five-part webinar series in 2019 about programs, practices, and partnerships that support students with disabilities in CTE. This series can be viewed here.

This blog was authored by Alice Bravo (University of Washington), IES intern through the Virtual Student Federal Service, and Akilah Nelson, Program Officer at NCSER. For more information about the Career and Technical Education for Students with Disabilities topic area, contact Akilah Nelson.

The views expressed by the investigators do not necessarily reflect those of IES.

In celebration of CTE^® (career and technical education) month, we would like to highlight the launch of an NCER project that aims to help us understand how to best support adults seeking additional CTE education and training.

Through their exploratory project, Literacy, Numeracy, and Problem-Solving Skills in Technology-Rich Environment in the STEM-Related Subbaccalaureate Programs in the United States, researchers will use a mixed-method design to gather information about the distribution of basic skills (literacy, numeracy, and problem solving) for adults in STEM occupations and students enrolled in STEM-related sub-baccalaureate programs at community colleges. Their goal is to help identify the needs of students and the programming practices at community colleges that may promote basic skill development in STEM programs.

The team will be leveraging data from the Program for the International Assessment of Adult Competencies (PIAAC) to survey the distribution of skills and abilities across a nationally representative sample of adults in STEM fields. They will also be collecting primary data from adults and programs in multiple locations, including Indiana, Ohio, and Washington states.

To help inform the public about their project, the researchers have created a short YouTube video for the public. This project began in July 2020 and may have initial results ready as early as late 2021.

Written by Meredith Larson (Meredith.Larson@ed.gov), Program Officer for Postsecondary and Adult Education, NCER.

The Career and Technical Education (CTE) Research Network just entered its third year, and it is time to share recent accomplishments with the IES community! As a reminder, the CTE Research Network (led by the American Institutes for Research, or AIR) was created to expand the evidence base on the impact of CTE programs on student outcomes using causal research methods.

Research

In June 2020, AIR released a preliminary report of CTE sites that are ready for causal evaluation. Designed to support researchers interested in studying the impact of CTE on student outcomes, the report details the history, theory of change, student enrollments, and other information for 4 selected CTE sites around the country. The Network hopes that researchers will use the information in the report to design evaluation studies of these programs.

In July 2020, a fifth research project joined the network. A team from MDRC, led by Rachel Rosen, was recently funded by IES to study the Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance. In partnership with Communities in Schools (CIS), the study will use a three-arm, school-level random assignment research design (RCT) to assess the effects of Navience and YouScience on students' self-expressed attitude and interest in career pathways, CTE course taking patterns, and engagement with and progress towards graduation. We welcome the team to the Network and look forward to learning whether and how these advising tools influence student thinking about career options, choice of relevant CTE coursework and work-based learning options, and decisions about CTE concentration in available pathways and programs of study.

A small group of researchers from different Network teams collaborated on and recently released a technical working paper on counterfactuals in CTE. It can be challenging to identify comparison groups for CTE students because it is an elective into which they self-select. The paper describes a variety of rigorous methods of comparing CTE students to valid counterparts and provides case studies that illustrate how to use these methods.

Training

The CTE Research Network is committed to increasing the number of researchers trained to study CTE using causal methods. It is notoriously challenging to isolate the effects of CTE from other influences on student outcomes. In August 2020, the Network hosted 18 researchers for a week-long virtual summer training institute on how to design a causal study to examine the impact of CTE. During the week, participants learned how to implement randomized-control trials (RCTs), regression discontinuity designs (RDDs), and comparative interrupted time series (CITS) in a CTE context. After learning about each method, participants worked in small groups to apply the method to real data and had access to the instructors to ask questions. The feedback about the training was overwhelmingly positive. The lecture portions of the training will be posted soon to the training page of the Network’s website. Another week-long training institute will be held in summer 2021 (hopefully, in person!)

The Network is currently developing a series of online modules for CTE practitioners and state agency staff to strengthen capacity to access, conduct, understand, and use CTE research. There will be a presentation to preview the modules at the Association of Career and Technical Education (ACTE) Best Practices and Innovations Conference on October 9, 2020 and a longer and more in-depth session at ACTE’s Career Tech VISION conference the first week of December. These practitioner training modules will be available for free on the website in late Fall 2020.

Leadership and Dissemination

The CTE Research Network is regularly updating its resources page with publications of interest to the CTE research field. The most recent is a report of findings from MDRC’s study of P-Tech high schools. The Network’s equity workgroup (a group of researchers from across the Network’s member projects) also published a popular blog this summer on applying an equity lens to CTE research. The Network also posts outside resources such as a REL self-study tool on career readiness and evaluation reports from other researchers.

For more about the CTE Research Network, you can sign up to receive the Network’s quarterly newsletter at the bottom of their website’s home page and follow them on Twitter (@CTEResNetwork) and LinkedIn.

If you are interested in learning more about the CTE Research Network, contact the Director, Kathy Hughes (khughes@air.org).

If you are interested in discussing CTE research opportunities at IES, contact Corinne Alfeld (Corinne.Alfeld@ed.gov).

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

STAGES FOR CONDUCTING A COST ANALYSIS

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

*More resources available here.

The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.