IES Blog

Each year, the Institute of Education Sciences (IES) recognizes an outstanding fellow from its Predoctoral Interdisciplinary Research Training Programs in the Education Sciences for academic accomplishments and contributions to education research. The 2020 awardee, Ken Holstein, completed his PhD at Carnegie Mellon University and is currently an assistant professor in the Human-Computer Interaction Institute at Carnegie Mellon University, where he directs a research lab focused on human-AI interaction.

Recently, we caught up with Dr. Holstein and asked him to discuss his research on human-computer interaction (HCI) and his experiences as a scholar.

How did you become interested in human-computer interaction and learning sciences research?

I have long been fascinated with human learning and expertise. As an undergraduate, I worked on research in computational cognitive science, with a focus on understanding how humans are often able to learn so much about the world from so little information (relative to state-of-the-art machine learning systems). Originally, I had planned on a career conducting basic research to better understand some of our most remarkable and mysterious cognitive capabilities. However, as I neared graduation, I became increasingly interested in pursuing research with more immediate potential for positive real-world impact. The fields of HCI and the learning sciences were a perfect fit to my interests. These areas provided opportunities to study how to support and enhance human learning and expertise in real-world settings, using a bricolage of research methods from a wide range of disciplines. 

Much of your lab’s research focuses on how humans and AI systems can augment each other’s abilities and learn from each other. What are the most promising applications of these ideas for education research and vice versa? 

I see a lot of potential for AI systems to augment the abilities of human teachers and tutors. In my PhD research, I worked with middle and high school teachers to understand their experiences working with AI-based tutoring software in their classrooms, and to co-design and prototype new possibilities together. Overall, teachers saw many opportunities to redesign AI tutoring software with the aim of augmenting and amplifying their own abilities as teachers, beyond simply automating instructional interactions with students. My research explored a small subset of these design directions, but there is a very rich design space that has yet to be explored.

In general, I believe that to design technologies that can effectively augment the abilities of human workers, such as teachers, it is critical to first understand what unique expertise and abilities they bring to the table as humans, which complement the capabilities of AI systems. This understanding can then inform the design of AI systems that explicitly support and draw upon the strengths of human workers (co-augmentation), and that can both learn from workers’ knowledge and support their professional learning (co-learning).

While I’ve described so far about ways the concepts of co-augmentation and co-learning can be applied to education research, I am also very excited about the opposite direction. I think that research on human-AI complementarity, AI-augmented work, and AI-assisted decision-making can benefit greatly by drawing upon ideas from education and the learning sciences. A lot of the research that we’re currently working on in my group involves bringing theories and approaches from the learning sciences to bear on open challenges in this space. To give just one example: there is a body of research that aims to design systems that support human-AI complementarity—configurations of humans and AI systems that yield better outcomes than working alone. So far, this research tends to focus on human ability as if it were static, rather than centering human learning. I believe this is a major missed opportunity, given that the human ability to learn and adapt based on incredibly scarce data is at the core of many of our most impressive capabilities relative to modern AI systems.

What advice would you give to emerging scholars that are pursuing a career in human-computer interaction? 

The field of human-computer interaction brings together a wide range of different topics, disciplines, research methods, and ways of knowing. As a junior scholar, this breadth can be both exciting and overwhelming. To navigate the overwhelm, I think it can be helpful to think about the forms of impact you would like your work to have. For example, are you interested in changing the way a research community thinks about a given topic? Are you interested in creating new technologies that can empower a particular group of people to do something that they could not have (easily) done otherwise? Are you interested in informing public policy with your research? Or are you interested in some combination of all of the above? Oftentimes, I have seen junior scholars in HCI start from a specific project idea, without having a clear sense of what impacts on the world their project might have if it is successful. Working “backwards” by considering and discussing desired impacts of research earlier on in the process can help to productively guide choices of research questions, methods, and lenses.

This blog was produced by IES training program officer Katina Stapleton (Katina.Stapleton@ed.gov). It is part of an Inside IES Research blog series showcasing a diverse group of IES-funded education fellows that are making significant contributions to education research, policy, and practice.

As education accountability policies continue to hold school leaders responsible for the success of their schools, it is crucial to assess and develop leadership throughout the school year. In honor of the IES 20^th Anniversary, we are highlighting NCER’s investment in leadership measures. This guest blog discusses the Vanderbilt Assessment of Leadership in Education (VALED). The VALED team was led by Andy Porter and included Ellen Goldring, Joseph Murphy and Steve Elliott, all at Vanderbilt University at the time. Other important contributors to the work are Xiu Cravens, Morgan Polikoff, Beth Minor Covay, and Henry May. The VALED was initially developed with funding from the Wallace Foundation and then further developed and validated with funding from IES.

What motivated your team to develop VALED?

There is currently widespread agreement that school principals have a major impact on schools and student achievement. However, at the time we developed VALED, we noticed that there were limited research-based instruments to measure principal leadership effectiveness aligned to both licensure standards and rooted in the evidence base. Prior to the VALED, principal leadership evaluation focused primarily on managerial tasks. However, we believed that principal leadership centered on improving teaching and learning, school culture, and community and parent engagement (often called learning-centered leadership) is at the core of leadership effectiveness.

What does VALED measure?

The VALED is a multi-rater assessment of learning-centered leadership behaviors. The principal, his/her supervisor, and teachers in the school complete it, which is why VALED is sometimes referred to as a 360 assessment or multi-source feedback.

VALED measures six core components and six key processes that define learning-centered leadership. The core components are high standards for student learning, rigorous curriculum, quality instruction, culture of learning and professional behavior, connections to external communities, and performance accountability. The key processes are planning, implementing, supporting, communicating, monitoring, and advocating.

How is the VALED different from other school leadership assessments?

The VALED is unique because it focuses on school leadership behaviors aligned to school improvement and school effectiveness, incorporates feedback and input from those who collaborate closely with the principal, includes a self- assessment, acknowledges the distributed work of leadership in a school, and has strong psychometric properties. We think there are several elements that contribute to the uniqueness of the instrument.

First, VALED is based on what we have learned from scholarship and academic research rather than less robust frameworks such as personal opinions and or unrepresentative samples. The VALED was crafted from concepts identified as important in that knowledge and understanding. The VALED model is based upon knowledge about connections between leadership and learning and provides a good deal of the required support for the accuracy, viability, and stability of the instrument.

Second, principals rarely receive data-based feedback, even though feedback is essential for growth and improvement. The rationale behind multi-source or 360-degree feedback is that information regarding leadership efficacy resides within the shared experiences of teachers and supervisors, collaborating with the principal, rather than from any one source alone. Data that pinpoint gaps between principal’s own self-assessment, and their teachers’ and supervisors’ ratings of their leadership effectiveness can serve as powerful motivators for change.

Finally, in contrast to some other leadership measures, VALED has undergone extensive psychometric development and testing. We conducted a sorting study to investigate content validity and a pilot study where we addressed ceiling effects, and cognitive interviews to refine wording. We also conducted a known group study that showed the tool’s ability to reliably distinguish principals, test-retest reliability, convergent-divergent validity, and principal value-added to student achievement. As part of this testing, we identified several key properties of VALED. The measure—  

• Works well in a variety of settings and circumstances
• Is construct valid
• Is reliable
• Is feasible for widespread use
• Provides accurate and useful reporting of results
• Is unbiased
• Yields a diagnostic profile for summative and formative purposes
• Can be used to measure progress over time in the development of leadership
• Predicts important outcomes
• Is part of a comprehensive assessment of the effectiveness of a leader's behaviors

What is the influence of VALED on education leadership research and practice?

VALED is used in schools and districts across the US and internationally for both formative and evaluative purposes to support school leadership development. For example, Baltimore City Public Schools uses VALED as a component of their School Leader Evaluations. VALED has also spurred studies on principal evaluation, including the association between evaluation, feedback and important school outcomes, the implementation of principal evaluation, and its uses to support principal growth and development. In addition, it provides a reliable and valid instrument for scholars to use in their studies as a measure of leadership effectiveness.

Andy Porter is professor emeritus of education at the Pennsylvania State University. He has published widely on psychometrics, student assessment, education indicators, and research on teaching.

Ellen Goldring is Patricia and Rodes Hart Chair, professor of education and leadership at Vanderbilt University. Her research interests focus on the intersection of education policy and school improvement with emphases on education leadership.

Joseph Murphy is an emeritus professor of education and the former Frank W. Mayborn Chair of Education at Peabody College, Vanderbilt University. He has published widely on school improvement, with special emphasis on leadership and policy and has been led national efforts to develop leadership standards. 

Produced by Katina Stapleton (Katina.Stapleton@ed.gov), program officer for NCER’s education leadership portfolio.

This year, Inside IES Research is publishing a series of blogs showcasing a diverse group of IES-funded education researchers and fellows that are making significant contributions to education research, policy, and practice. In recognition of Asian American and Pacific Islander Heritage Month, in this interview blog we asked Ben Le, an IES Predoctoral Fellow at New York University and a team member of the IES-funded R&D Center on the Success of English Learners (CSEL), to discuss his career journey and research interests.

How have your background and experiences shaped your scholarship and career in studying diversity, equity, and inclusion in education?

My research interests center around how race/ethnicity and language intersect to create unique privileges and discrimination. I hope my research can explore different ways we can support racially and linguistically marginalized students in schools, allowing them to bring their complete selves into the classroom and to help them thrive without having to give up their familial and communal languages.

Growing up in the United States as a Vietnamese-Mexican man has motivated me to look for new ways that we can conceptualize barriers for linguistically and racially marginalized students. While English learners (ELs) are currently the primary focus of my research, I’d like to recognize that I have never been classified as an EL.

I have been fortunate enough to be part of the IES-funded NYU Predoctoral Interdisciplinary Research Training (IES-PIRT) program, which has provided me the opportunity to  further explore and better understand racialized ELs’ access and opportunity in the classroom. My hope is that my IES-PIRT training will prepare me to work closely with local communities and organizations to enact change in our school systems. Ideally, we can build systems that truly support linguistically and racially marginalized students while offering them both access and opportunity that prepares them for life after school.

Can you tell us about your current IES-funded project?

As part of the CSEL R&D Center work, I am using a quantitative intersectional lens to highlight the importance of race/ethnicity for the diverse group of ELs in New York City public schools. I am particularly interested in how patterns of high school and college outcomes for current and former ELs vary based on race/ethnicity and gender. Focusing on 6-year graduation rates, I disaggregated my sample by race/ethnicity, gender, and ever-EL status (whether the student has ever been classified as an EL) to compare the probabilities across these subgroups and look for differential probabilities of being an ever-EL and a specific race/ethnicity. I focused on the two largest racial/ethnic groups of ELs in New York City, Asian Pacific Islander (PI) and Latine. For example, I compared the probability to graduate within 4 years between never-EL Asian/ (PI) young women to ever-EL Latino young men.

Interestingly, results, which were presented at the 2022 American Educational Research Association Annual Meeting, show that student probabilities for 6-year graduation are primarily organized by race/ethnicity, with Asian/PI students outperforming Latine students. Additionally, young women tend to outperform young men of their same racial/ethnic group, and in general, ever-EL status seems to matter even more for young men than young women. But these patterns do not explain away the racial/ethnic disparities seen in this New York City data. While ever-EL status matters, on aggregate, the ever-EL and never-EL differences primarily exist within racial/ethnic and gender subgroups. For example, never-EL Asian/PI young men outperform ever-EL Asian/PI young men, but ever-EL Asian/PI young men still outperform never-EL Latina young women.

Through my research, I hope to highlight the diversity and nuance within this ever-EL population, not to argue that ever-EL status does not matter. Instead, these findings have only motivated me to continue centering race/ethnicity and gender in future analyses for ELs.

What do you see as the greatest research needs to improve the relevance of education research for diverse communities of students and families?

From my perspective, we need to center the voices and concerns of these communities, families, and students in our data collection and analysis. I think it is essential to be involved with the families and meet them where they are to find effective solutions that benefit the communities we strive to serve. We need to make sure we are uplifting underserved families’ voices instead of talking over them. Relatedly, we need data and data collection to reflect the nuances and intricacies we are trying to discuss. Hopefully, future data collection can more accurately reflect the identities of the students we study. For example, I hope we can move away from collecting data as “male/female” and have a more expansive understanding of gender identities and not reify the gender binary.

What advice would you give to graduate students from underrepresented, minoritized groups that are pursuing a career in education research?

My first piece of advice would be to remember your own lived experiences and try to remind yourself that you do deserve to be in your graduate program. It’s easy to feel imposter syndrome—I think a lot of us do. Historically, academia and these programs were not made for us, and sadly, there is still a lot of work to be done, so that we don’t need to change to fit into these spaces. Still, these institutions and research fields benefit from our voices and perspectives. Remembering that these programs need us and that our experiences matter may be easier said than done, but I find it helpful to surround myself with fellow critical scholars and peers both within and outside of academia.

Secondly, finding community and support from peers and mentors has been absolutely crucial for my research and mental health. Doctoral programs aren’t easy; you are constantly being challenged intellectually and then you have to put your ideas and work out to be judged and critiqued. Being able to lean on friends and mentors for emotional support and to challenge and refine your research ideas is key to having a good and productive experience. I am super fortunate at NYU, through my sociology of education program and the IES-PIRT program, to have found such a caring community and supportive mentors, while also being pushed and challenged to pursue better and more critical work.

Produced by Helyn Kim (Helyn.Kim@ed.gov), program officer for the English Learners portfolio, NCER.

Mental Health Awareness Month was established in May of 1949 to increase awareness of the importance of mental health and wellness in Americans’ lives and to celebrate improvements in mental health treatment. Nearly 75 years later as we honor mental health awareness in this country, we are confronted with an alarming rise in mental health needs among America’s youth that the U.S. Surgeon General advises could be helped in part with school-based prevention and intervention supports.

One approach to school-based mental health is being tested in an IES-funded efficacy replication study called School Adolescent Mood Project: Efficacy of IPT-AST in Schools. This project is being led by Dr. Jami Young, associate chair of research in the Department of Child and Adolescent Psychiatry and Behavioral Sciences, faculty member in the PolicyLab at the Children’s Hospital of Philadelphia (CHOP), and professor of psychiatry at the University of Pennsylvania Perelman School of Medicine. Dr. Young was interviewed by Rebecca Sun, IES Intern, to help us learn more about conditions on the ground in her study and how telehealth-delivered Interpersonal Psychotherapy-Adolescent Skills Training (IPT-AST) is working in these schools.

Tell us about the schools you are working in and the different mental health needs you are seeing. How do you think the pandemic influenced what you are seeing now? 

In the School Adolescent Mood Project (SAM project), we have been working closely with 16 schools in Pennsylvania and New Jersey that serve 9^th and 10^th grade students. Over the past three years, what we are seeing in these 16 schools is very different than what we are used to seeing in our past two decades of school-based depression prevention work. Compared to our earlier studies (see this one for example), in the current SAM project, the rate of teens with elevated scores on our depression screening has more than doubled, as has the percentage of students with scores indicating more significant depression.

The increases we have seen in the SAM project mirror increases in depression and anxiety that have been reported by others during the pandemic, including the most recent findings reported by the CDC. Our still unpublished screening data from the SAM project suggest that there are a substantial number of adolescents who are experiencing mental health difficulties and would benefit from services in schools and/or the community. There is much work that needs to be done, including increases in funding and workforce expansion, to grow and sustain school-based services to support these students. Additionally, establishing connections between schools and mental health agencies is essential, so students who need more intensive services can receive timely care.

What do the youth in your study say are their biggest concerns? 

We are seeing many adolescents who are reporting significant issues with depression and anxiety. Unfortunately, we have also seen an increase in adolescents who report suicidal thoughts and behaviors. The pandemic was difficult for many adolescents as they experienced school closures, loss of loved ones to COVID-19, and feelings of loneliness and social isolation. Even with the return to in-person learning and afterschool activities in the 2021-2022 school year that allowed many students to regain a sense of normalcy, other adolescents found the return to school to be a challenging adjustment. In the SAM project, we saw an increase in the number of adolescents who reported serious thoughts of wanting to kill themselves or who had made a suicide attempt from the 2020-2021 school year when most schools were virtual or hybrid to the 2021-2022 school year when schools were back to in person learning. Unfortunately, we have continued to see an increase in the percentage of adolescents reporting thoughts of wanting to kill themselves this academic year.      

Our school partners have also noted a significant increase in students with suicide-related concerns. One school counselor shared, “I think we have done more suicide risk assessments this year than the past 5 years combined.” This points to a critical need to fund and implement evidence-based screening, assessment, and prevention initiatives for youth suicide.

What have you learned so far about the feasibility and effectiveness of providing the IPT-AST intervention in a telehealth format?

We needed to think creatively about how to ensure that students received the support that they needed and to consider how we could promote continuity of care if schools needed to close again. We made the decision to deliver IPT-AST through telehealth. We thought the COVID pandemic was an important opportunity to capitalize on the recent growth and innovations in digital health and to study the acceptability and efficacy of IPT-AST when delivered through telehealth.

We’ve interviewed a subset of adolescents who participated in our IPT-AST groups to get their feedback about the program, including how they felt about IPT-AST being delivered through telehealth. We are still analyzing these interviews and looking at our other data on acceptability and feasibility, including attendance and satisfaction data. Anecdotally, we have noticed a change in the acceptability of telehealth over the course of our study. In our first year, providing our group online meant that students could receive services and interact with their peers even when school was closed. As we have moved further from school closures, more adolescents have expressed a desire to return to in-person groups. Several adolescents have suggested we consider hybrid models that incorporate both in-person and virtual sessions. One positive benefit of running IPT-AST groups online is that we have been able to include students from different schools in the same group. If we find positive benefits of telehealth-delivered IPT-AST, it will be interesting to see whether some schools and districts prefer to continue this approach as it will enable providers to deliver these services outside of the school day.

Do you think the pandemic has eased the stigma surrounding seeking mental health?

We do not measure stigma specifically in our study but hope that all the attention around mental health has been beneficial. As May is Mental Health Awareness Month, we think it is important to emphasize that mental health is health. If we can continue to spread this message, stigma around mental health will continue to decrease. We are hopeful that the increased focus on youth mental health, decreases in stigma, and proposed policy solutions to increase access to mental services will mean that more adolescents will be able to get the care they need.

This blog was produced by Emily Doolittle (Emily.Doolittle@ed.gov), NCER team lead for social behavioral research.

In honor of Mental Health Awareness Month, NCSER is featuring an IES-funded study on student behavioral supports and interventions that best address the mental health needs of students. Positive Behavioral Interventions and Supports (PBIS) and school mental health (SMH) are both evidence-based interventions that provide student mental health support independent of one another. For this blog, we interviewed Dr. Brandon Schultz, principal investigator of a current study investigating the integration of both PBIS and SMH into a comprehensive school intervention. In the interview below, he discusses the differences between PBIS and SMH, how this research contributes to equity and inclusion in the classroom, and his research journey.

Your study is comparing schools that integrate PBIS and SMH into the enhanced version of the Interconnected Systems Framework (ISF) to schools that implement these as separate, parallel systems. Can you describe PBIS and SMH, and explain the key differences between the integrated framework and parallel systems?

PBIS is a tiered prevention system that addresses student behavioral needs. It provides universal support (Tier 1) to all students, including clear schoolwide behavioral expectations and a rewards system for desired behaviors. For students who do not respond to these efforts, Tier 2 provides targeted help through classroom-level or small group interventions, such as teacher consultation or student mentoring/counseling. For students who need intensive support, Tier 3 provides specialized one-to-one behavioral services. SMH, in contrast, focuses specifically on mental illnesses (for example, anxiety, trauma, depression) and, in some cases, involves community-based therapists working contractually with schools. Typically, PBIS and SMH function separately as co-located services, but there is a growing recognition that student needs are best met when these efforts are meaningfully integrated. Integration, however, is challenging because it requires educators to rethink their teaming and progress monitoring practices and include different stakeholders in critical decision-making processes. This study tests innovations to the ISF model, designed by my co-PI, Dr. Mark Weist (University of South Carolina), to meet the challenges of integrating these systems in two diverse school districts.

How did you become interested in this area of research?

My previous research was mostly focused on school-based interventions for students with ADHD, but it became clear that without systems-level change, interventions meant to help students with ADHD are unlikely to be implemented or sustained effectively, no matter how well they are designed. So, I became interested in understanding school systems and identifying the elements, processes, and resources that are critical for student support services of all kinds. 

How does your research contribute to equity and inclusion in education?

Part of my current study is focused the degree to which innovations to the ISF model can reduce racial inequities in school disciplinary actions. Research shows that Black students receive higher rates of exclusionary punishments (for example, suspensions and expulsions) than their White counterparts, even after controlling for the type of infraction. The modified ISF model aims to reduce the overall need for exclusionary punishments, especially among students of color. By improving team functioning, ISF allows educators to identify systemic problems that lead to racial inequities in disciplinary referrals and to generate new strategies to address student needs in a fair and equitable manner. With this model, we anticipate increased support for students of color that obviates disciplinary referrals. We are working with school districts now to examine disciplinary data before, during, and after the implementation of the enhanced ISF. Our hope is to identify strategies that close race-related gaps and share the lessons learned broadly.

Have you encountered any challenges in studying this integrated framework in elementary schools?

Yes, absolutely. Systems-level change in general is difficult, as it requires change agents to overcome structural inertia rooted in local norms, routines, and expectations. Those challenges have been exacerbated by the COVID-19 pandemic and preexisting trends in childhood mental illnesses. 

During the pandemic, student progress in mathematics and reading have dramatically declined. Meeting these academic needs, a priority for teachers, can divert attention away from student mental health needs. For example, all teachers in one of our states are required to take a year-long online course in reading instruction, partly to address student learning loss. Although commendable, this requirement creates a significant burden for teachers that can leave little room for other concerns. 

Preexisting mental health trends demonstrate that mental illness was increasing sharply among school-age children; by 2018, nearly 15% of all K-12 students experienced a psychiatric condition each year. Then, with the onset of the pandemic, indicators of childhood mental illness (for example, emergency room visits for suicidal behavior) spiked. Childhood anxiety and depression doubled worldwide from pre-pandemic estimates, and it is unclear whether those rates will return to baseline.

Together, these events have created real challenges, not just for our research, but for student support services in general.

What is currently the greatest area of need in studying school-based systems that support student mental health, particularly for those students with or at risk for emotional and behavioral disorders?

Perhaps the greatest area of need for supporting students with emotional and behavioral disorders is understaffing in critical school mental health positions. There is a significant shortage of school psychologists, counselors, social workers, and nurses nationwide. In North Carolina, the current ratio of school psychologists to students is 1:2,527, five times higher than recommended. This understaffing hinders schools’ ability to provide high-quality services and complicates efforts to test and refine innovative practices because field-based practitioners are unable to collaborate on research efforts. Researchers have had to hire individuals to fulfill critical roles, such as behavioral consultants, that might otherwise have been assigned to district-employed staff. Trained personnel then exit the school district when the research project ends and that skillset is lost. We hope that states prioritize the hiring of school mental health practitioners in the coming years to ensure optimal student support services and that university-school research collaborations can reliably lead to sustainable innovations.

NCSER looks forward to seeing the results of this efficacy trial and will continue to fund research aimed at supporting the mental health and social-emotional-behavioral needs of students with or at risk for disabilities.

This blog was authored by Isabelle Saillard, student volunteer for NCSER and undergraduate at the University of Virginia.