IES Blog

In honor of CTE month, we wanted to provide more information about the recent additions to NCER’s CTE research portfolio: 7 new grants awarded across 2 grant programs in FY2021. A previous blog last summer announced these grants; this blog briefly describes each project (the hyperlinks will take you to the full online project abstract). Be sure to read to the end for links to other CTE work across IES!

Within the Education Research Grants program (305A), the following projects were funded in 2021:

College and Career Readiness: Investigating California's Efforts to Expand Career Technical Education Through Dual Enrollment

(PI: Michal Kurlaender, University of California, Davis)

This project is examining the result of policy changes due to California Assembly Bill 288 (AB288), enacted in 2015 to create the College and Career Access Pathways (CCAP) partnership. CCAP increased the prominence of Career and Technical Education (CTE) at the high school level by allowing high schools and community colleges to enter joint partnerships and offer dual enrollment courses that count towards both a high school diploma and an associate degree from California community colleges. 

Postsecondary and Labor Market Effects of Career and Technical Education in Baltimore City Public Schools

(PI: Marc Stein, Johns Hopkins University)

This project uses a unique selection process into CTE Centers within a large school district, linked with longitudinal state data, to provide strong evidence on the benefits and mechanisms of CTE participation on secondary education, postsecondary education, and labor market outcomes.

SREB Career and Technical Education Leadership Academy Study

(PI: James Stone, Southern Regional Education Board)

This project is developing, piloting, and studying the promise of the Southern Regional Education Board (SREB) Career and Technical Education Leadership Academy to increase the capacity of school leaders at career and technology centers to work with their teachers as instructional leaders and thereby improve CTE outcomes for their students.

An Experimental Evaluation of the Efficacy of Virtual Enterprises

(Co-PIs: Fatih Unlu, RAND Corporation and Kathy Hughes, AIR)

In this project, the research team will provide the first causal evidence on the efficacy and cost-effectiveness of Virtual Enterprises, a virtual school-based enterprise (SBE) program. SBEs are one form of work-based learning (WBL) in which students run a business that produces and sells goods or services. SBEs may offer unique benefits relative to other types of WBL by providing opportunities for more students to participate, reducing the need for transportation, and allowing students more room to make and learn from their mistakes. Note: Since this is an efficacy (causal impact) study, it has joined the CTE Research Network.

Sub-baccalaureate Career and Technical Education: A Study of Institutional Practices, Labor Market Demand, and Student Outcomes in Florida

(PI: Angela Estacion, WestEd)

The purpose of this project is to address existing policy and research gaps by, first, administering a statewide survey to catalogue the institutional practices that Florida community and technical colleges use to align CTE programming to the labor market. Second, by combining the survey data with student-level program participation and outcome data, the project team will ascertain the degree to which institutional practices and labor market conditions in students' geographical areas are correlated with students' choices and outcomes. Finally, the project team will analyze qualitative data collected from case studies of Florida community and technical colleges to describe the practices cited in the survey data and understand the process of aligning courses and programs with local labor market demand.

Within the “Using SLDS to Support State Education Policymaking” (305S) grant program, the following projects were funded in 2021:

The Distributional Effects of Secondary Career and Technical Educational (CTE) Programs on Postsecondary Educational and Employment Outcomes: An Evaluation of Delaware's CTE Programs of Study

(PI: Luke Rhine, Delaware Department of Education)

The Delaware Department of Education and University of Delaware is examining variability in participation rates among student subgroups in Delaware public high school CTE programs and link CTE high school participation to high school graduation, postsecondary enrollment, employment, and wages. Stay tuned to delawarepathways.org for more information as the project unfolds.

Analyzing and Understanding the Educational and Economic Impact of Regional Career Pathways

(PI: Jonathan Attridge, Tennessee)

The research team is conducting an evaluation of Tennessee Pathways (along with earlier career pathway programs), a state initiative to align K-12 education, postsecondary education, and employers so that high school students have a clear pathway to move into the workforce.

For a full list of CTE-related grants funded by NCER and NCSER across years, topics, and grant competitions, you can explore our “funded projects” search pages for NCER or NCSER. Here is a recent blog post about students with disabilities in CTE from NCSER. And don’t forget to visit the CTE Research Network frequently for many new CTE-related findings and resources!

NCEE and NCES have published multiple reports on CTE, including Career and Technical Education Credentials in Virginia High Schools: Trends in Attainment and College Enrollment Outcomes, as well as some great blogs such as this one on exploring the growing impact of career pathways from NCEE. You can see more on career readiness from NCEE here and from NCES here.

We are so pleased with the growth of this portfolio since our first call for more CTE research 5 years ago! However, there continues to be a need to better understand CTE. For instance, research is still needed on CTE measures and assessments, quality of programs (content, instruction, and opportunities for WBL), and variation in impacts across student subgroups and career clusters.

For more information about CTE research grants, including feedback on new project proposal ideas, please contact NCER program officer Corinne.Alfeld@ed.gov.

Research identifies benefits of access to same-race/ethnicity teachers for Black and Hispanic students. However, the teacher workforce is overwhelmingly White, and little is known about the system-level strategies that are successful at diversifying the profession. In recognition of Black History Month, we asked researcher Dr. David Blazar to discuss his recently awarded IES project that aims to advance the literature base on how school systems can recruit more Black teachers. This is what he shared.

What does existing research say about the need for more Black teachers?

Building on a longstanding theoretical and qualitative literature base from scholars including Gloria Ladson-Billings, Geneva Gay, Richard Milner, and many others, researchers have gathered causal evidence to support the claim of the benefit of Black teachers to Black students. Analyzing test score data from Tennessee's Project STAR experiment, Dee (2004) found that assignment to a Black teacher significantly increased the math and reading achievement of Black students.

Fast forward 18 years, and the research findings largely remain the same while the evidence base has grown substantially (see one meta-analysis, and a research synthesis). In the second experiment on this topic after Dee, my own recent analyses currently available in a working paper not only replicate the earlier test-score impacts, but also show that

• Test-scores effects (roughly 0.2 SD) persist at very similar magnitudes 6 years later when students are in high school, a rare pattern in education research
• Black and other underrepresented teachers of color have even larger effects (upwards of 0.45 SD) on the social-emotional development of their students of color and their White students
• Black and other teachers of color are much more likely than White teachers to hold mindsets and engage in classroom practices aligned to “culturally responsive teaching,” which in turn benefits a range of student outcomes

In short: The effects of Black teachers on the outcomes of Black students are larger than those of most other interventions as documented in the broader education research literature (generally no higher than 0.1 SD).

I pair these hugely meaningful findings with three more sobering facts:

• Black teachers are underrepresented in the teacher workforce. Roughly 7% of teachers nationally are Black, compared to roughly 15% of students. These patterns have not shifted much over the last several decades, even though calls to diversify the teacher workforce started over 30 years ago.
• The mismatch between student and teacher demographics may be due to “leaks” at multiple stages of the school-to-career pipeline, including lower rates of high school graduation amongst Black students relative to their White peers, similar gaps in college graduation rates, less interest in teaching as a career, and greater financial barriers and opportunity costs even when the interest is there.
• Despite impressive work by educators, scholars, and policymakers to design multiple strategies for recruiting Black individuals into teaching, the bulk of these remain “promising practices” rather than evidence-based best practices.

How will your IES-funded study address the need for more Black teachers?

Because the underrepresentation of Black teachers in U.S. schools is notable and longstanding, researchers and school systems must work together—and quickly—to consider multiple strategies. Stating that we need to diversify the teacher workforce is neither new nor novel. The imperative was posed several decades ago, and it is time that we figure out how best to do it.

To address this challenge head on, I am collaborating with Ramon Goings, Seth Gershenson, and other scholars, as well as with state agencies and policy actors in Maryland to explore several recruitment strategies aimed at diversifying the teacher workforce, implemented at different stages of the school-to-career pipeline.

Aligned to the theoretical literature, a core feature of our study is that we focus on strategies that look locally for prospective teaching talent and are therefore known as “grow-your-own” programs. These approaches aim to align the demographics of incoming teachers with the demographics of current student populations and ensure that those incoming teachers are familiar with the local area. We further designed our study to explore multiple components of and potential solutions to the policy problem, given that recruitment is unlikely to be addressed with a one-size-fits-all approach. Even though the partnership and data come from Maryland, the recruitment strategies and our study are relevant to the recruitment strategies used in states across the country.

The three strategies are—

• Early exposure to teaching in high school through the Teacher Academy of Maryland—a career and technical education program of study—that provides high school students with an opportunity to learn about teaching as a career, gain teaching experience in a real-world classroom, and earn an associate’s degree in teaching alongside their high school diploma.
• Financial support and incentives for college students, including the recently implemented Teaching Fellows for Maryland Scholarship. Scholarships aim to decrease financial barriers and opportunity costs that may prevent Black individuals from becoming teachers.
• Career-changer programs, such as alternative-route teacher certification and residency programs that both decrease barriers to entry into the profession and focus on recruiting locally.

Our analyses will provide some of the first quantitative data linking the rollout of varied recruitment strategies and the workforce decisions of prospective Black teachers. Beyond analyses of each individual program, our findings will provide important guidance not only about how best to intervene but also when to do so. We look forward to sharing what we find and to building an evidence base alongside other scholars and funding agencies tackling this important issue.

David Blazar is an Assistant Professor at the University of Maryland College Park (UMCP) in the Education Policy and Leadership program. He also is the Faculty Director of the Maryland Equity Project, a UMCP initiative to improve educational outcomes and close achievement gaps through research.

This interview blog is part of a larger IES blog series on diversity, equity, inclusion and accessibility (DEIA) in the education sciences. It was produced by Katina Stapleton (Katina.Stapleton@ed.gov), co-Chair of the IES Diversity and Inclusion Council, and Wai-Ying Chow (Wai-Ying.Chow@ed.gov), the Effective Instruction program officer within the National Center for Education Research.

This is the second of a two-part blog series from an IES-funded partnership project. The first part described how the process of cost-effectiveness analysis (CEA) provided useful information that led to changes in practice for a school nurse program and restorative practices at Jefferson County Public Schools (JCPS) in Louisville, KY. In this guest blog, the team discusses how the process of conducting value-added analysis provided useful program information over and above the information they obtained via CEA or academic return on investment (AROI).

Since we know you loved the last one, it’s time for another fun thought experiment! Imagine that you have just spent more than a year gathering, cleaning, assembling, and analyzing a dataset of school investments for what you hope will be an innovative approach to program evaluation. Now imagine the only thing your results tell you is that your proposed new application of value-added analysis (VAA) is not well-suited for these particular data. What would you do? Well, sit back and enjoy another round of schadenfreude at our expense. Once again, our team of practitioners from JCPS and researchers from Teachers College, Columbia University and American University found itself in a very unenviable position.

We had initially planned to use the rigorous VAA (and CEA) to evaluate the validity of a practical measure of academic return on investment for improving school budget decisions on existing school- and district-level investments. Although the three methods—VAA, CEA, and AROI—vary in rigor and address slightly different research questions, we expected that their results would be both complementary and comparable for informing decisions to reinvest, discontinue, expand/contract, or make other implementation changes to an investment. To that end, we set out to test our hypothesis by comparing results from each method across a broad spectrum of investments. Fortunately, as with CEA, the process of conducting VAA provided additional, useful program information that we would not have otherwise obtained via CEA or AROI. This unexpected information, combined with what we’d learned about implementation from our CEAs, led to even more changes in practice at JCPS.

Data Collection for VAA Unearthed Inadequate Record-keeping, Mission Drift, and More

Our AROI approach uses existing student and budget data from JCPS’s online Investment Tracking System (ITS) to compute comparative metrics for informing budget decisions. Budget request proposals submitted by JCPS administrators through ITS include information on target populations, goals, measures, and the budget cycle (1-5 years) needed to achieve the goals. For VAA, we needed similar, but more precise, data to estimate the relative effects of specific interventions on student outcomes, which required us to contact schools and district departments to gather the necessary information. Our colleagues provided us with sufficient data to conduct VAA. However, during this process, we discovered instances of missing or inadequate participant rosters; mission drift in how requested funds were actually spent; and mismatches between goals, activities, and budget cycles. We suspect that JCPS is not alone in this challenge, so we hope that what follows might be helpful to other districts facing similar scenarios.

More Changes in Practice 

The lessons learned during the school nursing and restorative practice CEAs discussed in the first blog, and the data gaps identified through the VAA process, informed two key developments at JCPS. First, we formalized our existing end-of-cycle investment review process by including summary cards for each end-of-cycle investment item (each program or personnel position in which district funds were invested) indicating where insufficient data (for example, incomplete budget requests or unavailable participation rosters) precluded AROI calculations. We asked specific questions about missing data to elicit additional information and to encourage more diligent documentation in future budget requests. 

Second, we created the Investment Tracking System 2.0 (ITS 2.0), which now requires budget requesters to complete a basic logic model. The resources (inputs) and outcomes in the logic model are auto-populated from information entered earlier in the request process, but requesters must manually enter activities and progress monitoring (outputs). Our goal is to encourage and facilitate development of an explicit theory of change at the outset and continuous evidence-based adjustments throughout the implementation. Mandatory entry fields now prevent requesters from submitting incomplete budget requests. The new system was immediately put into action to track all school-level Elementary and Secondary School Emergency Relief (ESSER)-related budget requests.

Process and Partnership, Redux

Although we agree with the IES Director’s insistence that partnerships between researchers and practitioners should be a means to (eventually) improving student outcomes, our experience shows that change happens slowly in a large district. Yet, we have seen substantial changes as a direct result of our partnership. Perhaps the most important change is the drastic increase in the number of programs, investments, and other initiatives that will be evaluable as a result of formalizing the end-of-cycle review process and creating ITS 2.0. We firmly believe these changes could not have happened apart from our partnership and the freedom our funding afforded us to experiment with new approaches to addressing the challenges we face.   

Stephen M. Leach is a Program Analysis Coordinator at JCPS and PhD Candidate in Educational Psychology Measurement and Evaluation at the University of Louisville.

Dr. Robert Shand is an Assistant Professor at American University.

Dr. Bo Yan is a Research and Evaluation Specialist at JCPS.

Dr. Fiona Hollands is a Senior Researcher at Teachers College, Columbia University.

If you have any questions, please contact Corinne Alfeld (Corinne.Alfeld@ed.gov), IES-NCER Grant Program Officer.

This is the first of a two-part guest blog series from an IES-funded partnership project between Teachers College, Columbia University, American University, and Jefferson County Public Schools in Kentucky. The purpose of the project is to explore academic return on investment (AROI) as a metric for improving decision-making around education programs that lead to improvements in student education outcomes. In this guest blog entry, the team showcases cost analysis as an integral part of education program evaluation.

Here’s a fun thought experiment (well, at least fun for researcher-types). Imagine you just discovered that two of your district partner’s firmly entrenched initiatives are not cost-effective. What would you do? 

Now, would your answer change if we told you that the findings came amidst a global pandemic and widespread social unrest over justice reform, and that those two key initiatives were a school nurse program and restorative practices? That’s the exact situation we faced last year in Jefferson County Public Schools (JCPS) in Louisville, KY. Fortunately, the process of conducting rigorous cost analyses of these programs unearthed critical evidence to help explain mostly null impact findings and inform very real changes in practice at JCPS.

Cost-Effectiveness Analysis Revealed Missing Program Components

Our team of researchers from Teachers College, Columbia University and American University, and practitioners from JCPS had originally planned to use cost-effectiveness analysis (CEA) to evaluate the validity of a practical measure of academic return on investment for improving school budget decisions. With the gracious support of JCPS program personnel in executing our CEAs, we obtained a treasure trove of additional quantitative and qualitative cost and implementation data, which proved to be invaluable.

Specifically, for the district’s school nurse program, the lack of an explicit theory of change, of standardized evidence-based practices across schools, and of a monitoring plan were identified as potential explanations for our null impact results. In one of our restorative practices cost interviews, we discovered that a key element of the program, restorative conferences, was not being implemented at all due to time constraints and staffing challenges, which may help explain the disappointing impact results.

Changes in Practice

In theory, our CEA findings indicated that JCPS should find more cost-effective alternatives to school nursing and restorative practices. In reality, however, both programs were greatly expanded; school nursing in response to COVID and restorative practices because JCPS leadership has committed to moving away from traditional disciplinary practices. Our findings regarding implementation, however, lead us to believe that key changes can lead to improved student outcomes for both.

In response to recommendations from the team, JCPS is developing a training manual for new nurses, a logic model illustrating how specific nursing activities can lead to better outcomes, and a monitoring plan. For restorative practices, while we still have a ways to go, the JCPS team is continuing to work with program personnel to improve implementation.

One encouraging finding from our CEA was that, despite imperfect implementation, suspension rates for Black students were lower in schools that had implemented restorative practices for two years compared to Black students in schools implementing the program for one year. Our hope is that further research will identify the aspects of restorative practices most critical for equitably improving school discipline and climate.

Process and Partnership

Our experience highlights unexpected benefits that can result when researchers and practitioners collaborate on all aspects of cost-effectiveness analysis, from collecting data to applying findings to practice. In fact, we are convinced that the ongoing improvements discussed here would not have been possible apart from the synergistic nature of our partnership. While the JCPS team included seasoned evaluators and brought front-line knowledge of program implementation, information systems, data availability, and district priorities, our research partners brought additional research capacity, methodological expertise, and a critical outsider’s perspective.

Together, we discovered that the process of conducting cost-effectiveness analysis can provide valuable information normally associated with fidelity of implementation studies. Knowledge gained during the cost analysis process helped to explain our less-than-stellar impact results and led to key changes in practice. In the second blog of this series, we’ll share how the process of conducting CEA and value-added analysis led to changes in practice extending well beyond the specific programs we investigated.

Stephen M. Leach is a Program Analysis Coordinator at JCPS and PhD Candidate in Educational Psychology Measurement and Evaluation at the University of Louisville.

Dr. Fiona Hollands is a Senior Researcher at Teachers College, Columbia University.

Dr. Bo Yan is a Research and Evaluation Specialist at JCPS.

Dr. Robert Shand is an Assistant Professor at American University.

If you have any questions, please contact Corinne Alfeld (Corinne.Alfeld@ed.gov), IES-NCER Grant Program Officer.

School environments are places in which students, particularly students of color, are exposed to implicit bias and discrimination that can negatively impact their academic outcomes. In this interview blog, we asked prevention scientist Dr. Chynna McCall to discuss how her career journey and her experiences working with children and families from diverse populations inspired her research on creating equitable school environments.   

How did you begin your career journey as a prevention scientist?

Perhaps my most valued professional experience is serving as a licensed school psychologist in public schools in Colorado, working with children and families from racially, culturally, and linguistically diverse populations. This experience inspired me to join the Missouri Prevention Science Institute in 2018 as an Institute of Education Sciences postdoctoral fellow, where I studied how to use research to solve real-world problems. More specifically, I learned how to use prevention science to develop and evaluate evidence-based practices and interventions that prevent negative social and emotional impacts before they happen. After my fellowship, I was hired and promoted to a senior research associate position at the Missouri Prevention Science Institute. In this role, I have operational responsibilities for various federally funded grants and conduct my own grant-funded research. Presently, I am working on the development and testing of an equity-focused social-emotional learning curriculum for 3rd through 5th grade students.

What challenges did you observe as a school psychologist?

As a school psychologist, I worked in two vastly different school districts. In one, most students came from low-income families, spoke English as a second language, and the school's performance on standardized tests was significantly below average. Most of the challenges I tackled during my time there could be categorized as social-emotional; most students had unbalanced home lives, and many suffered emotional or physical trauma. Because the school district pressured teachers to improve test scores, focus on behavior and classroom management unilaterally shifted towards scholastics. The unfortunate outcome was neglecting to acknowledge the role that student behavior and the root causes of those behaviors play in affecting academic outcomes. While the second district I worked for was a high-performing one with generally high socioeconomic status, I chose to work for the school designated for those children in the district who have serious emotional disabilities.

Even though there are stark differences between the two districts, I consistently encountered a need for students to develop better relationships with their teachers, peers, and parents, develop a better sense of self, and for teachers, other school personnel, students, and parents to have a better understanding of how their practices and interactions are impacting student social-emotional and academic outcomes.

How does your background as a school psychologist influence your research?

My experience as a school psychologist has reinforced my understanding of what is needed to improve public education and what research questions are of utmost importance. Through my time as a school psychologist, it helped me define the goals of my research, which include 1) understanding the influence of prejudice and discrimination on student internal and external behaviors and outcomes, 2) understanding how school personnel expression of prejudice and discrimination influence student internal and external behaviors and outcomes, and 3) determining how to most effectively develop an equitable school environment that positively influences marginalized and minoritized youth outcomes.

My research examines how school environment—including the prejudicial and discriminatory thoughts and behaviors of school staff, students, and guardians—influences identity development, identity expression (for example, racial identity, gender identity, sexuality, and intersectionality) and internal and external behaviors. The objective is to use this knowledge to create a school environment that facilitates prosocial student identity development. My research hinges on my observations and experiences as a practicing school psychologist to focus on how to shift differential outcomes observed in public education due to experiences of discrimination both in and out of the school setting.

In your area of research, what do you see as the greatest research needs or recommendations to address diversity and equity and improve the relevance of education research for diverse communities of students and families?

I believe schools at every level of education are microcosms for the greater society. How students traverse through the school system dictates how they will navigate through the macrocosm of society. How students navigate the school system can be improved if school systems are equipped with the tools that allow staff to prepare the students better academically, socially, and emotionally. These tools are essential for students who are having a difficult time because of cultural, linguistic, psychological, or physical differences from their peers. It is crucial for the research community to continually advocate for positive change in our education system, work towards better understanding student needs, and develop effective and efficient tools that better promote student growth and outcomes.

I also believe that researchers who study school environments must explicitly study bias. We have to look at whether and how school professionals are becoming aware of and challenging their implicit biases, as well as how students are becoming aware of bias and how they deal with it—either by internalizing it or challenging it. We also must look into how challenging or accepting bias affects students emotionally, behaviorally, and academically.

What advice would you give to emerging scholars from underrepresented, minoritized groups that are pursuing a career in education research?

See your perspective and experience as assets. Your perspective is underrepresented and is needed in making necessary changes to education and education outcomes. When you view your perspective as something of value, you are better able to determine what unaddressed research questions need to be asked and to move education research in a direction that is more inclusive of every student.

This year, Inside IES Research is publishing a series of interviews (see here, here, and here) showcasing a diverse group of IES-funded education researchers and fellows that are making significant contributions to education research, policy, and practice. As part of our Black History Month blog series, we are focusing on African American/Black researchers and fellows as well as researchers who focus on the education of Black students.

Dr. Chynna McCall is a Senior Research Associate with the Missouri Prevention Science Institute at the University of Missouri. Prior to this position, she was an IES postdoctoral fellow in the Missouri Interdisciplinary Postdoctoral Research and Training Program training program.

Produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), postdoctoral training program officer, and Katina Stapleton (Katina.Stapleton@ed.gov), co-Chair of the IES Diversity and Inclusion Council and predoctoral training program officer.