# IES Blog

### Institute of Education Sciences

For nearly 15 years, NCER has supported the development and improvement of innovative methodological and statistical tools and approaches that will better enable applied education researchers to conduct high-quality, rigorous education research. This blog spotlights the work of Andrew Gelman, a professor of statistics and political science at Columbia University, and Sophia Rabe-Hesketh, a professor of statistics at the School of Education at the University of California, Berkeley. IES has supported their research on hierarchical modeling and Bayesian computation has for many years. In this interview blog, Drs. Gelman and Rabe-Hesketh reflect on how Bayesian modeling applies to educational data and describe the general principles and advantages of Bayesian analysis.

What motivates your research on hierarchical modeling and Bayesian computation?

Education data can be messy. We need to adjust for covariates in experiments and observational studies, and we need to be able to generalize from non-random, non-representative samples to populations of interest.

The general motivation for multilevel modeling is that we are interested in local parameters, such as public opinion by states, small-area disease incidence rates, individual performance in sports, school-district-level learning loss, and other quantities that vary among people, across locations, and over time. In non-Bayesian settings, the local parameters are called random effects, varying intercepts/slopes, or latent variables.

Bayesian and non-Bayesian models differ in how completely the researcher using them must specify the probability distributions of the parameters. In non-Bayesian models, typically only the data model (also called the likelihood function) must be specified. The underlying parameters, such as the variances of random intercepts, are treated as unknown constants. On the other hand, the Bayesian approach requires specifying a full probability model for all parameters.

A researcher using Bayesian inference encodes additional assumptions about all parameters into prior distributions, then combines information about the parameters from the data model with information from the prior distributions. This results in a posterior distribution for each parameter, which, compared to non-Bayesian model results, provides more information about the appropriateness of the model and supports more complex inferences.

What advantages are there to the Bayesian approach?

Compared to other estimates, Bayesian estimates are based on many more assumptions. One advantage of this is greater stability at small sample sizes. Another advantage is that Bayesian modeling can be used to produce flexible, practice-relevant summaries from a fitted model that other approaches cannot produce. For instance, when modeling school effectiveness, researchers using the Bayesian approach can rely on the full probability model to justifiably obtain the rankings of schools or the probabilities that COVID-related declines in NAEP mean test scores for a district or state have exceeded three points, along with estimates for the variability of these summaries.

Further, Bayesian inference supports generalizability and replicability by freely allowing uncertainty from multiple sources to be integrated into models. Without allowing for uncertainty, it’s difficult to understand what works for whom and why. A familiar example is predicting student grades in college courses. A regression model can be fit to obtain a forecast with uncertainty based on past data on the students, and then this can be combined with student-specific information. Uncertainties in the forecasts for individual students or groups of students will be dependent and can be captured by a joint probability model, as implemented by posterior simulations. This contrasts with likelihood-based (non-Bayesian) inference where predictions and their uncertainty are typically considered only conditionally on the model parameters, with maximum likelihood estimates plugged in. Ignoring uncertainty leads to standard error estimates that are too small on average (see this introduction to Bayesian multilevel regression for a detailed demonstration and discussion of this phenomenon).

What’s an important disadvantage to the Bayesian approach?

Specifying a Bayesian model requires the user to make more decisions than specifying a non-Bayesian model. Until recently, many of these decisions had to be implemented using custom programming, so the Bayesian approach had a steep learning curve. Users who were not up to the programming and debugging task had to work within some restricted class of models that had already been set up with existing software.

This disadvantage is especially challenging in education research, where we often need to adapt and expand our models beyond a restricted class to deal with statistical challenges such as imperfect treatment assignments, nonlinear relations, spatial correlations, and mixtures, along with data issues such as missingness, students changing schools, guessing on tests, and predictors measured with error.

In 2011, we developed Stan, our open-source Bayesian software, with funding from a Department of Energy grant on large-scale computing. With additional support from the National Science Foundation and IES, we have developed model types, workflows, and case studies for education researchers and also improved Stan’s computational efficiency.

By combining a state-of-the-art inference engine with an expressive modeling language, Stan allows education researchers to build their own models, starting with basic linear and logistic regressions and then adding components of variation and uncertainty and expanding as needed to capture challenges that arise in applied problems at hand.  We recommend the use of Stan as part of a Bayesian workflow of model building, checking, and expansion, making use of graphs of data and fitted models.

Stan can be accessed using R, Python, Stata, Julia, and other software. We recommend getting started by looking at the Stan case studies. We also have a page on Stan for education research and a YouTube channel.

In terms of dealing with the issues that arise in complex educational data, where do we stand today?

Put all this together, and we are in the business of fitting complex models in an open-ended space that goes beyond any existing menu of statistical procedures. Bayesian inference is a flexible way to fit such models, and Stan is a flexible tool that we have developed, allowing general models to be fit in reasonable time using advanced algorithms for statistical computing.  As always with research, there are many loose ends and there is more work to be done, but we can now routinely fit, check, and display models of much greater generality than was before possible, facilitating the goals of understanding processes in education.

This blog was produced by Charles Laurin (Charles.Laurin@ed.gov), NCER program officer for the Statistical and Research Methodology in Education grant program.

NCES is committed to providing reliable and up-to-date national-level estimates of bullying. As such, a new set of web tables focusing on bullying victimization at school was just released.

These tables use data from the School Crime Supplement to the National Crime Victimization Survey, which collects data on bullying by asking a nationally representative sample of students ages 12–18 who were enrolled in grades 6–12 in public and private schools if they had been bullied at school. This blog post highlights data from these newly released web tables.

Some 19 percent of students reported being bullied during the 2021–22 school year. More specifically, bullying was reported by 17 percent of males and 22 percent of females and by 26 percent of middle school students and 16 percent of high school students. Moreover, among students who reported being bullied, 14 percent of males and 28 percent of females reported being bullied online or by text.

Students were also asked about the recurrence and perpetrators of bullying and about the effects bullying has on them. During the 2021–22 school year, 12 percent of students reported that they were bullied repeatedly or expected the bullying to be repeated and that the bullying was perpetrated by someone who was physically or socially more powerful than them and who was not a sibling or dating partner. When these students were asked about the effects this bullying had on them,

• 38 percent reported negative feelings about themselves;
• 27 percent reported negative effects on their schoolwork;
• 24 percent reported negative effects on their relationships with family and friends; and
• 19 percent reported negative effects on their physical health.

Explore the web tables for more data on how bullying victimization varies by student characteristics (e.g., sex, race/ethnicity, grade, household income) and school characteristics (e.g., region, locale, enrollment size, poverty level) and how rates of bullying victimization vary by crime-related variables such as the presence of gangs, guns, drugs, alcohol, and hate-related graffiti at school; selected school security measures; student criminal victimization; personal fear of attack or harm; avoidance behaviors; fighting; and the carrying of weapons.

Find additional information on this topic in the Condition of Education indicator Bullying at School and Electronic Bullying. Plus, explore more School Crime and Safety data and browse the Report on Indicators of School Crime and Safety: 2022.

In recognition of Black History Month, we interviewed Dr. Tamara Bertrand Jones, an associate professor of higher education in the Department of Educational Leadership and Policy Studies at Florida State University and co-principal investigator of the Partners United for Research Pathways Oriented to Social Justice in Education (PURPOSE) program, funded by the IES Pathways to the Education Sciences Research Training Program. In this blog, Dr. Bertrand Jones discusses her experiences conducting research on the professional experiences of underrepresented populations as well as her work supporting emerging scholars of color.

How has your background and experiences shaped your research on the graduate education and professional experiences of underrepresented populations, particularly Black women, in academia?

My dissertation research centered Black perspectives on cultural competence in evaluation. For the first 10 years of my academic career, I worked as an administrator, primarily in student affairs. When I transitioned from administration to faculty, I extended my research to Black experiences in academia. Microaggressions (such as unsolicited advice) that derailed my productivity and diminished my self-confidence immediately greeted me. For example, I was told that my research would be labeled navel-gazing (excessive self-contemplation) because I was a Black woman studying Black women and that this negative label may present challenges for my career. It took time, lots of positive self-affirmation, and validation from my mentors and close Black women colleagues to silence those voices and walk confidently in my contribution as a scholar and my personal purpose for pursuing an academic career. After these experiences, I doubled down on my commitment to demystify the hidden curriculum in the academy and support emerging scholars by being responsive to their identities, experiences, needs, and aspirations.

What is the PURPOSE training program, and what have you learned from administering PURPOSE?

We created PURPOSE to help develop more underrepresented and minoritized education researchers. To date, we have had seven cohorts of PURPOSE Fellows, totaling more than 80 fellows. The program includes critical discussions about social justice and educational inequities, mentoring, professional development, and service-learning research apprenticeships. During their training, we also encourage fellows to reflect on their own identities in terms of race, gender, and social class among other identities while they develop their individual researcher identities. These experiences culminate in capstone research projects related to social justice in education that fellows develop from inception to dissemination during the fellowship year. Taken together, these experiences foster capacities to conduct meaningful research and provide socialization into the rigors of research and graduate school.

We found that fellows experience socialization into education research in ways that help them 1) develop a researcher-identity, and 2) prepare products that demonstrate strong research potential for graduate school. Our fellows have experienced positive gains in their self-efficacy for carrying out a variety of research skills such as conducting literature reviews and working independently and in teams. We believe our approach to culturally relevant education and research methods and valuing the voices of our diverse fellows and mentors will lead to changes in future teaching and research practices.

Based on your research and experiences, what do you see as the greatest needs to improve the education and professional pathways for Black scholars?

In the over 14 years I have been a faculty member, I recognize that there are myriad ways to be successful in academia while remaining true to who you are. Through my work on early career professionals in partnership with Sisters of the Academy (SOTA), a community of Black women in higher education, I strive to create an environment where emerging scholars are exposed to scholars who represent diverse ways of being in academia. These models can shape emerging scholars’ vision of their future possible selves and help them develop their own pathways that are congruent with who they are. If institutions lack those models in their faculty, I urge leaders to intentionally connect with groups or organizations like SOTA that have the expertise and access to individuals who can serve in those roles from their emerging scholars.

What advice would you give to emerging scholars from underrepresented, minoritized groups that are pursuing a career in education research?

Often emerging scholars from underrepresented, minoritized groups are not encouraged to engage in work that speaks to their soul or can meaningfully impact the communities they serve. As in my experience, underrepresented emerging scholars are often told that doing research on our identity groups or researching issues that these groups experience is limiting, pigeonholing, and too self-reflective. Emerging Black scholars, in particular, are told they must approach their work in ways that are contradictory to their values or diminish their self-concepts. These messages can stunt growth and hinder the ability to identify innovative solutions to education’s most-pressing problems.

Because of this, I encourage all emerging scholars to consider the following reflective questions, guided by my emerging professional development framework—the 5 I’s, to help align their education research careers with how they see themselves, individually and in community.

• Identity: How does my identity influence my research?
• Intention: How can I create synergy between my research and scholarship, courses I teach, service I perform, and who I am as a scholar?
• Implementation: How does my positionality influence my research design choices?
• Influence: Who needs to know about my work? How can partnership extend the impact of my work?
• Impact: How can my work be used to create better educational environments for marginalized or minoritized communities, or change education policy, research, or practice in meaningful ways?

This interview blog is part of a larger IES blog series on diversity, equity, inclusion and accessibility (DEIA) in the education sciences. It was produced by Akilah Nelson (akilah.nelson@ed.gov), a program officer within the National Center for Special Education Research.

With rapid technological advances, the U.S. labor market exhibits a growing need for more frequent and ongoing skill development. Community college noncredit career and technical education (CTE) programs that allow students to complete workforce training and earn credentials play an essential role in providing workers with the skills they need to compete for jobs in high-demand fields. Yet, there is a dearth of research on these programs because noncredit students are typically not included in state and national postsecondary datasets. In this guest blog for CTE Month, researchers Di Xu, Benjamin Castleman, and Betsy Tessler discuss their IES-funded exploration study in which they build on a long-standing research partnership with the Virginia Community College System and leverage a variety of data sources to investigate the Commonwealth’s FastForward programs. These programs are noncredit CTE programs designed to lead to an industry-recognized credential in one of several high-demand fields identified by the Virginia Workforce Board.

In response to the increasing demand for skilled workers in the Commonwealth, the Virginia General Assembly passed House Bill 66 in 2016 to establish the New Economy Workforce Credential Grant Program (WCG) with the goal of providing a pay-for-performance model for funding noncredit training. The WCG specifically funds FastForward programs that lead to an industry-recognized credential in a high-demand field in the Commonwealth. Under this model, funding is shared between the state, students, and training institutions based on student performance, with the goal of ensuring workforce training is affordable for Virginia residents. An important implication of WCG is that it led to systematic, statewide collection of student-level data on FastForward program enrollment, program completion, industry credential attainment, and labor market performance. Drawing on these unique data, coupled with interviews with key stakeholders, we generated findings on the characteristics of FastForward programs, as well as the academic and labor market outcomes of students enrolled in these programs. We describe the preliminary descriptive findings below.

FastForward programs enroll a substantially different segment of the population from credit-bearing programs and offer a vital alternative route to skill development and workforce opportunities, especially for demographic groups often underrepresented in traditional higher education. FastForward programs in Virginia enroll a substantially higher share of Black students, male students, and older students than short-duration, credit-bearing programs at community colleges that typically require one year or less to complete. Focus groups conducted with FastForward students at six colleges indicate that the students were a mix of workers sent by their employers to learn specific new skills and students who signed up for a FastForward program on their own. Among the latter group were older career changers and recent high school graduates, many of whom had no prior college experience and were primarily interested in landing their first job in their chosen field. Moreover, 61% of FastForward participants have neither prior nor subsequent enrollment in credit-bearing programs, highlighting the program’s unique role in broadening access to postsecondary education and career pathways.

FastForward programs offer an alternative path for students who are unsuccessful in credit-bearing programs. The vast majority of students (78%) enrolled in only one FastForward program, with the average enrollment duration of 1.5 quarters, which is notably shorter than most traditional credit-bearing programs. While 36% have prior credit-bearing enrollment, fewer than 20% of these students earned a degree or certificate from it, and less than 12% of FastForward enrollees transitioned to credit-bearing training afterward. Interviews with administrators and staff indicated that while some colleges facilitate noncredit-to-credit pathways by granting credit for prior learning, others prioritize employment-focused training and support over stackable academic pathways due to students’ primary interest in seeking employment post-training.

FastForward programs have a remarkable completion rate and are related to high industry credential attainment rates. Over 90% of students complete their program, with two-thirds of students obtaining industry credentials. Student focus groups echoed this success. They praised the FastForward program and colleges for addressing both their tuition and non-tuition needs. Many students noted that they had not envisioned themselves as college students and credited program staff, financial aid, and institutional support with helping them to be successful.

Earning an industry credential through FastForward on average increases quarterly earnings by approximately \$1,000. In addition, industry credentials also increase the probability of being employed by 2.4 percentage points on average. We find substantial heterogeneity in economic return across different fields of study, where the fields of transportation (for example, commercial driver’s license) and precision production (for example, gas metal arc welding) seem to be associated with particularly pronounced earnings premiums. Within programs, we do not observe significant heterogeneity in economic returns across student subgroups.

What’s Next?

In view of the strong economic returns associated with earning an industry credential and the noticeable variation in credential attainment between training institutions and programs, our future exploration intends to unpack the sources of variation in program-institution credential attainment rates and to identify specific program-level factors that are within the control of an institution and which are associated with higher credential rates and lower equity gaps. Specifically, we will collect additional survey data from the top 10 most highly-enrolled programs at the Virginia Community College System (VCCS) that will provide more nuanced program-level information and identify which malleable program factors are predictive of higher credential attainment rates, better labor market outcomes, and smaller equity gaps associated with these outcomes.

Di Xu is an associate professor in the School of Education at UC, Irvine, and the faculty director of UCI’s Postsecondary Education Research & Implementation Institute.

Ben Castleman is the Newton and Rita Meyers Associate Professor in the Economics of Education at the University of Virginia.

Betsy Tessler is a senior associate at MDRC in the Economic Mobility, Housing, and Communities policy area.

Note: A team of researchers, including Kelli Bird, Sabrina Solanki, and Michael Cooper contributed jointly to the quantitative analyses of this project. The MDRC team, including Hannah Power, Kelsey Brown, and Mark van Dok, contributed to qualitative data collection and analysis. The research team is grateful to the Virginia Community College System (VCCS) for providing access to their high-quality data. Special thanks are extended to Catherine Finnegan and her team for their valuable guidance and support throughout our partnership.

This project was funded under the Postsecondary and Adult Education research topic; questions about it should be directed to program officer James Benson (James.Benson@ed.gov).

This blog was produced by Corinne Alfeld (Corinne.Alfeld@ed.gov), NCER program officer for the CTE research topic.

Positive career and technical education (CTE) experiences have the potential to lead to long-term success for students with disabilities. Yet the pathways into this field for teachers are highly variable. In honor of CTE Awareness Month, we would like to share an interview with NCSER-funded principal investigators Dan Goldhaber (left below) and Roddy Theobald (right below), who have been investigating the relationship between preparation pathways for CTE teachers and student outcomes. In the interview below, Drs. Goldhaber and Theobald share their findings and how their research can influence CTE teacher licensure.

What led to your interest in studying CTE for students with disabilities?

A growing body of research—including prior work we’ve done with a NCSER grant on predictors of postsecondary outcomes for students with disabilities—has found that participation in a concentration of CTE courses in high school is a strong predictor of improved postsecondary outcomes for students with disabilities. Moreover, in another recent NCSER-funded project, we found that pre-service preparation of special education teachers can be a significant predictor of outcomes for students with disabilities in their classrooms. Our current project lies directly at the intersection of these two prior projects and asks the following question: Given the importance of both CTE courses and special education teachers for predicting outcomes for students with disabilities, what role do CTE teachers play in shaping these outcomes, and what types of CTE teacher preparation are most predictive of improved outcomes for these students? This question is important in Washington state because individuals with prior employment experience can become a CTE teacher through a "business and industry" (B&I) pathway that does not require as much formal teacher preparation as traditional licensure pathways. Likewise, this question is important nationally because over half of states offer a similar CTE-specific path to teacher licensure that relies on prior work experience as a licensure requirement.

Your research team published a report last year from your current research project with some surprising results related to the teacher preparation pathway and outcomes for students. Can you tell us about those findings?

In the first paper from this project, now published in Teacher Education and Special Education, we connected observable characteristics of CTE teachers in Washington to non-test outcomes (including absences, disciplinary incidents, grade point average, grade progression, and on-time graduation) of students with and without disabilities in their classrooms. The most surprising findin­g was that students with disabilities participating in CTE tended to have better non-test outcomes when they were assigned to a CTE teacher from the B&I pathway compared those assigned to a traditionally prepared CTE teacher.

What do you think may be the underlying reason for this finding?

We discussed several hypotheses for this result in the paper, including the possibility that the content knowledge and experience of B&I pathway teachers may matter more than traditional preparation for students with disabilities. This conclusion, however, comes with two caveats. First, preliminary results from the second paper (presented at the 2023 APPAM Fall Conference) suggest that these relationships do not translate to improved college enrollment or employment outcomes for these students. Second, we cannot disentangle the effects of B&I teachers' prior employment experiences from "selection effects" of who chooses to enter through this pathway.

In what ways can this research influence CTE policy and practice?

We have described teacher licensure as the "Wild West" of education policy because 50 different states are responsible for developing state teacher licensing systems. CTE teacher licensure is like the "Wild West of the Wild West" in that over half of states offer a CTE-specific pathway to licensure, which relies on prior industry experience as a requirement for licensure, each with different requirements and regulations. As states continue to navigate challenges with staffing CTE classrooms with qualified teachers, it is important to understand the implications of the unique CTE-specific pathways for student outcomes, particularly for students with disabilities. This project is an early effort to provide this evidence to inform CTE licensure policy.

How do you plan to continue this line of research?

The next steps of this project leverage data provided through the Washington state’s P-20 longitudinal data system maintained by the Washington Education Research and Data Center (ERDC). ERDC has connected high school students' CTE experiences (including their teacher) to college and employment records. This allows us to consider the implications of CTE teacher characteristics for students' postsecondary outcomes. Moreover, due to the question about the prior employment experiences of CTE teachers, ERDC has agreed to link records on CTE teachers’ prior employment so we can disentangle the importance of different pre-teaching employment experiences of CTE teachers.

Is there anything else you would like to add?

We are grateful to NCSER for their support of this project and the two prior projects that motivated it!

Dr. Dan Goldhaber is the director of the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research (AIR) and the director of the Center for Education Data and Research at the University of Washington.

Dr. Roddy Theobald is the deputy director of CALDER and a managing researcher at AIR. Thank you, Dr. Dan Goldhaber and Dr. Roddy Theobald, for sharing your experiences and findings about CTE!

This blog was authored by Skyler Fesagaiga, a Virtual Student Federal Service intern for NCSER and graduate student at the University of California, San Diego. Akilah Nelson, NCSER program officer, manages grants funded under the Career and Technical Education for Students with Disabilities special topic.