Inside IES Research

As education accountability policies continue to hold school leaders responsible for the success of their schools, it is crucial to assess and develop leadership throughout the school year. In honor of the IES 20^th Anniversary, we are highlighting NCER’s investment in leadership measures. This guest blog discusses the Vanderbilt Assessment of Leadership in Education (VALED). The VALED team was led by Andy Porter and included Ellen Goldring, Joseph Murphy and Steve Elliott, all at Vanderbilt University at the time. Other important contributors to the work are Xiu Cravens, Morgan Polikoff, Beth Minor Covay, and Henry May. The VALED was initially developed with funding from the Wallace Foundation and then further developed and validated with funding from IES.

What motivated your team to develop VALED?

There is currently widespread agreement that school principals have a major impact on schools and student achievement. However, at the time we developed VALED, we noticed that there were limited research-based instruments to measure principal leadership effectiveness aligned to both licensure standards and rooted in the evidence base. Prior to the VALED, principal leadership evaluation focused primarily on managerial tasks. However, we believed that principal leadership centered on improving teaching and learning, school culture, and community and parent engagement (often called learning-centered leadership) is at the core of leadership effectiveness.

What does VALED measure?

The VALED is a multi-rater assessment of learning-centered leadership behaviors. The principal, his/her supervisor, and teachers in the school complete it, which is why VALED is sometimes referred to as a 360 assessment or multi-source feedback.

VALED measures six core components and six key processes that define learning-centered leadership. The core components are high standards for student learning, rigorous curriculum, quality instruction, culture of learning and professional behavior, connections to external communities, and performance accountability. The key processes are planning, implementing, supporting, communicating, monitoring, and advocating.

How is the VALED different from other school leadership assessments?

The VALED is unique because it focuses on school leadership behaviors aligned to school improvement and school effectiveness, incorporates feedback and input from those who collaborate closely with the principal, includes a self- assessment, acknowledges the distributed work of leadership in a school, and has strong psychometric properties. We think there are several elements that contribute to the uniqueness of the instrument.

First, VALED is based on what we have learned from scholarship and academic research rather than less robust frameworks such as personal opinions and or unrepresentative samples. The VALED was crafted from concepts identified as important in that knowledge and understanding. The VALED model is based upon knowledge about connections between leadership and learning and provides a good deal of the required support for the accuracy, viability, and stability of the instrument.

Second, principals rarely receive data-based feedback, even though feedback is essential for growth and improvement. The rationale behind multi-source or 360-degree feedback is that information regarding leadership efficacy resides within the shared experiences of teachers and supervisors, collaborating with the principal, rather than from any one source alone. Data that pinpoint gaps between principal’s own self-assessment, and their teachers’ and supervisors’ ratings of their leadership effectiveness can serve as powerful motivators for change.

Finally, in contrast to some other leadership measures, VALED has undergone extensive psychometric development and testing. We conducted a sorting study to investigate content validity and a pilot study where we addressed ceiling effects, and cognitive interviews to refine wording. We also conducted a known group study that showed the tool’s ability to reliably distinguish principals, test-retest reliability, convergent-divergent validity, and principal value-added to student achievement. As part of this testing, we identified several key properties of VALED. The measure—  

• Works well in a variety of settings and circumstances
• Is construct valid
• Is reliable
• Is feasible for widespread use
• Provides accurate and useful reporting of results
• Is unbiased
• Yields a diagnostic profile for summative and formative purposes
• Can be used to measure progress over time in the development of leadership
• Predicts important outcomes
• Is part of a comprehensive assessment of the effectiveness of a leader's behaviors

What is the influence of VALED on education leadership research and practice?

VALED is used in schools and districts across the US and internationally for both formative and evaluative purposes to support school leadership development. For example, Baltimore City Public Schools uses VALED as a component of their School Leader Evaluations. VALED has also spurred studies on principal evaluation, including the association between evaluation, feedback and important school outcomes, the implementation of principal evaluation, and its uses to support principal growth and development. In addition, it provides a reliable and valid instrument for scholars to use in their studies as a measure of leadership effectiveness.

Andy Porter is professor emeritus of education at the Pennsylvania State University. He has published widely on psychometrics, student assessment, education indicators, and research on teaching.

Ellen Goldring is Patricia and Rodes Hart Chair, professor of education and leadership at Vanderbilt University. Her research interests focus on the intersection of education policy and school improvement with emphases on education leadership.

Joseph Murphy is an emeritus professor of education and the former Frank W. Mayborn Chair of Education at Peabody College, Vanderbilt University. He has published widely on school improvement, with special emphasis on leadership and policy and has been led national efforts to develop leadership standards. 

Produced by Katina Stapleton (Katina.Stapleton@ed.gov), program officer for NCER’s education leadership portfolio.

This year, Inside IES Research is publishing a series of blogs showcasing a diverse group of IES-funded education researchers and fellows that are making significant contributions to education research, policy, and practice. In recognition of Asian American and Pacific Islander Heritage Month, in this interview blog we asked Ben Le, an IES Predoctoral Fellow at New York University and a team member of the IES-funded R&D Center on the Success of English Learners (CSEL), to discuss his career journey and research interests.

How have your background and experiences shaped your scholarship and career in studying diversity, equity, and inclusion in education?

My research interests center around how race/ethnicity and language intersect to create unique privileges and discrimination. I hope my research can explore different ways we can support racially and linguistically marginalized students in schools, allowing them to bring their complete selves into the classroom and to help them thrive without having to give up their familial and communal languages.

Growing up in the United States as a Vietnamese-Mexican man has motivated me to look for new ways that we can conceptualize barriers for linguistically and racially marginalized students. While English learners (ELs) are currently the primary focus of my research, I’d like to recognize that I have never been classified as an EL.

I have been fortunate enough to be part of the IES-funded NYU Predoctoral Interdisciplinary Research Training (IES-PIRT) program, which has provided me the opportunity to  further explore and better understand racialized ELs’ access and opportunity in the classroom. My hope is that my IES-PIRT training will prepare me to work closely with local communities and organizations to enact change in our school systems. Ideally, we can build systems that truly support linguistically and racially marginalized students while offering them both access and opportunity that prepares them for life after school.

Can you tell us about your current IES-funded project?

As part of the CSEL R&D Center work, I am using a quantitative intersectional lens to highlight the importance of race/ethnicity for the diverse group of ELs in New York City public schools. I am particularly interested in how patterns of high school and college outcomes for current and former ELs vary based on race/ethnicity and gender. Focusing on 6-year graduation rates, I disaggregated my sample by race/ethnicity, gender, and ever-EL status (whether the student has ever been classified as an EL) to compare the probabilities across these subgroups and look for differential probabilities of being an ever-EL and a specific race/ethnicity. I focused on the two largest racial/ethnic groups of ELs in New York City, Asian Pacific Islander (PI) and Latine. For example, I compared the probability to graduate within 4 years between never-EL Asian/ (PI) young women to ever-EL Latino young men.

Interestingly, results, which were presented at the 2022 American Educational Research Association Annual Meeting, show that student probabilities for 6-year graduation are primarily organized by race/ethnicity, with Asian/PI students outperforming Latine students. Additionally, young women tend to outperform young men of their same racial/ethnic group, and in general, ever-EL status seems to matter even more for young men than young women. But these patterns do not explain away the racial/ethnic disparities seen in this New York City data. While ever-EL status matters, on aggregate, the ever-EL and never-EL differences primarily exist within racial/ethnic and gender subgroups. For example, never-EL Asian/PI young men outperform ever-EL Asian/PI young men, but ever-EL Asian/PI young men still outperform never-EL Latina young women.

Through my research, I hope to highlight the diversity and nuance within this ever-EL population, not to argue that ever-EL status does not matter. Instead, these findings have only motivated me to continue centering race/ethnicity and gender in future analyses for ELs.

What do you see as the greatest research needs to improve the relevance of education research for diverse communities of students and families?

From my perspective, we need to center the voices and concerns of these communities, families, and students in our data collection and analysis. I think it is essential to be involved with the families and meet them where they are to find effective solutions that benefit the communities we strive to serve. We need to make sure we are uplifting underserved families’ voices instead of talking over them. Relatedly, we need data and data collection to reflect the nuances and intricacies we are trying to discuss. Hopefully, future data collection can more accurately reflect the identities of the students we study. For example, I hope we can move away from collecting data as “male/female” and have a more expansive understanding of gender identities and not reify the gender binary.

What advice would you give to graduate students from underrepresented, minoritized groups that are pursuing a career in education research?

My first piece of advice would be to remember your own lived experiences and try to remind yourself that you do deserve to be in your graduate program. It’s easy to feel imposter syndrome—I think a lot of us do. Historically, academia and these programs were not made for us, and sadly, there is still a lot of work to be done, so that we don’t need to change to fit into these spaces. Still, these institutions and research fields benefit from our voices and perspectives. Remembering that these programs need us and that our experiences matter may be easier said than done, but I find it helpful to surround myself with fellow critical scholars and peers both within and outside of academia.

Secondly, finding community and support from peers and mentors has been absolutely crucial for my research and mental health. Doctoral programs aren’t easy; you are constantly being challenged intellectually and then you have to put your ideas and work out to be judged and critiqued. Being able to lean on friends and mentors for emotional support and to challenge and refine your research ideas is key to having a good and productive experience. I am super fortunate at NYU, through my sociology of education program and the IES-PIRT program, to have found such a caring community and supportive mentors, while also being pushed and challenged to pursue better and more critical work.

Produced by Helyn Kim (Helyn.Kim@ed.gov), program officer for the English Learners portfolio, NCER.

Mental Health Awareness Month was established in May of 1949 to increase awareness of the importance of mental health and wellness in Americans’ lives and to celebrate improvements in mental health treatment. Nearly 75 years later as we honor mental health awareness in this country, we are confronted with an alarming rise in mental health needs among America’s youth that the U.S. Surgeon General advises could be helped in part with school-based prevention and intervention supports.

One approach to school-based mental health is being tested in an IES-funded efficacy replication study called School Adolescent Mood Project: Efficacy of IPT-AST in Schools. This project is being led by Dr. Jami Young, associate chair of research in the Department of Child and Adolescent Psychiatry and Behavioral Sciences, faculty member in the PolicyLab at the Children’s Hospital of Philadelphia (CHOP), and professor of psychiatry at the University of Pennsylvania Perelman School of Medicine. Dr. Young was interviewed by Rebecca Sun, IES Intern, to help us learn more about conditions on the ground in her study and how telehealth-delivered Interpersonal Psychotherapy-Adolescent Skills Training (IPT-AST) is working in these schools.

Tell us about the schools you are working in and the different mental health needs you are seeing. How do you think the pandemic influenced what you are seeing now? 

In the School Adolescent Mood Project (SAM project), we have been working closely with 16 schools in Pennsylvania and New Jersey that serve 9^th and 10^th grade students. Over the past three years, what we are seeing in these 16 schools is very different than what we are used to seeing in our past two decades of school-based depression prevention work. Compared to our earlier studies (see this one for example), in the current SAM project, the rate of teens with elevated scores on our depression screening has more than doubled, as has the percentage of students with scores indicating more significant depression.

The increases we have seen in the SAM project mirror increases in depression and anxiety that have been reported by others during the pandemic, including the most recent findings reported by the CDC. Our still unpublished screening data from the SAM project suggest that there are a substantial number of adolescents who are experiencing mental health difficulties and would benefit from services in schools and/or the community. There is much work that needs to be done, including increases in funding and workforce expansion, to grow and sustain school-based services to support these students. Additionally, establishing connections between schools and mental health agencies is essential, so students who need more intensive services can receive timely care.

What do the youth in your study say are their biggest concerns? 

We are seeing many adolescents who are reporting significant issues with depression and anxiety. Unfortunately, we have also seen an increase in adolescents who report suicidal thoughts and behaviors. The pandemic was difficult for many adolescents as they experienced school closures, loss of loved ones to COVID-19, and feelings of loneliness and social isolation. Even with the return to in-person learning and afterschool activities in the 2021-2022 school year that allowed many students to regain a sense of normalcy, other adolescents found the return to school to be a challenging adjustment. In the SAM project, we saw an increase in the number of adolescents who reported serious thoughts of wanting to kill themselves or who had made a suicide attempt from the 2020-2021 school year when most schools were virtual or hybrid to the 2021-2022 school year when schools were back to in person learning. Unfortunately, we have continued to see an increase in the percentage of adolescents reporting thoughts of wanting to kill themselves this academic year.      

Our school partners have also noted a significant increase in students with suicide-related concerns. One school counselor shared, “I think we have done more suicide risk assessments this year than the past 5 years combined.” This points to a critical need to fund and implement evidence-based screening, assessment, and prevention initiatives for youth suicide.

What have you learned so far about the feasibility and effectiveness of providing the IPT-AST intervention in a telehealth format?

We needed to think creatively about how to ensure that students received the support that they needed and to consider how we could promote continuity of care if schools needed to close again. We made the decision to deliver IPT-AST through telehealth. We thought the COVID pandemic was an important opportunity to capitalize on the recent growth and innovations in digital health and to study the acceptability and efficacy of IPT-AST when delivered through telehealth.

We’ve interviewed a subset of adolescents who participated in our IPT-AST groups to get their feedback about the program, including how they felt about IPT-AST being delivered through telehealth. We are still analyzing these interviews and looking at our other data on acceptability and feasibility, including attendance and satisfaction data. Anecdotally, we have noticed a change in the acceptability of telehealth over the course of our study. In our first year, providing our group online meant that students could receive services and interact with their peers even when school was closed. As we have moved further from school closures, more adolescents have expressed a desire to return to in-person groups. Several adolescents have suggested we consider hybrid models that incorporate both in-person and virtual sessions. One positive benefit of running IPT-AST groups online is that we have been able to include students from different schools in the same group. If we find positive benefits of telehealth-delivered IPT-AST, it will be interesting to see whether some schools and districts prefer to continue this approach as it will enable providers to deliver these services outside of the school day.

Do you think the pandemic has eased the stigma surrounding seeking mental health?

We do not measure stigma specifically in our study but hope that all the attention around mental health has been beneficial. As May is Mental Health Awareness Month, we think it is important to emphasize that mental health is health. If we can continue to spread this message, stigma around mental health will continue to decrease. We are hopeful that the increased focus on youth mental health, decreases in stigma, and proposed policy solutions to increase access to mental services will mean that more adolescents will be able to get the care they need.

This blog was produced by Emily Doolittle (Emily.Doolittle@ed.gov), NCER team lead for social behavioral research.

This year, Inside IES Research is publishing a series of blogs showcasing a diverse group of IES-funded education researchers and fellows that are making significant contributions to education research, policy, and practice. In recognition of Asian American and Pacific Islander Heritage Month, in this interview blog we asked Dr. Xin Wei, a senior quantitative researcher at Digital Promise to discuss her career journey. Dr. Wei’s current IES-funded study uses statistical and machine-learning techniques to understand the test-taking behavior of National Assessment of Educational Progress (NAEP) grade 8 learners with and without disabilities.

How did you become interested in a career in education research?

As a child, I aspired to become a teacher, and in college I decided to pursue a degree in child development. During my senior year of college, I worked as a research assistant on a project studying statistical and psychometric methods used to analyze learning differences among children. This experience sparked my interest in education research and revealed the potential for statistical analysis to inform and enhance teaching practices.

Graduate studies at the University of Wisconsin-Madison and Stanford University helped me gain a deeper understanding of quantitative methods in education research. Through applying and improving quantitative methods, I discovered how national and state longitudinal datasets can help us understand the learning, social, and emotional needs of students with disabilities and which policy interventions can help us achieve better outcomes. This opportunity helped me understand the challenges students with disabilities face in the education system and deepened my appreciation for secondary data analysis and its power to inform intervention research.

Currently, my research focuses on analyzing log/process data to understand how digital learning and assessments can facilitate student learning, accurately measure progress, and improve outcomes for students with disabilities. Through this work, I am committed to advancing the education research field at the intersection of special education, learning analytics, and psychometrics.

What has been the biggest challenge you have encountered, and how did you overcome the challenge?

When I came to the United States to pursue a graduate degree at the age of 23, I faced a host of challenges that forced me out of my comfort zone. Navigating a new culture and adapting to academic expectations and research demands was overwhelming. Additionally, understanding U.S. K-12 education policies and practices was no easy feat. However, I was fortunate enough to have incredible mentors, professors, peers, and colleagues who provided me with guidance, support, and patience when I needed it most. These individuals played a crucial role in helping me grow as a researcher.

The most important lesson I learned from the challenges I faced was the value of continuous learning and growth in my career. These experiences have strengthened my commitment to making a positive impact in education and helping others who may be facing similar obstacles.

How can the broader education research community better support the careers and scholarship of researchers from underrepresented groups?

The student population in the United States is diverse, and it is essential that the education research community reflects that diversity by including scholars who bring unique perspectives and experiences.

One way to do this is by actively seeking out and valuing diverse voices in research, teaching, and leadership positions. This includes promoting diversity in conference panels, as well as actively recruiting and hiring researchers from underrepresented groups. By creating a culture of inclusivity, the education research community can better support the careers and scholarship of researchers from underrepresented groups.

Another way to better support the careers and scholarship of researchers from underrepresented groups is through mentoring programs, summer internships, and postdoc positions. These opportunities can provide valuable professional development and collaboration opportunities. In addition, research grants specifically targeted toward underrepresented groups can also help support their work and advance their careers. It is essential to widely advertise these opportunities and make them accessible to ensure that all researchers have an equal chance to participate.

In your area of research, what do you see as the greatest research needs or recommendations to address diversity, equity, and inclusion and to improve the relevance of education research for diverse communities of students and families?

To address diversity, equity, and inclusion in education research, it is crucial to adopt an asset-based approach when working with neurodiverse students. By shifting the focus from deficits to strengths, we can recognize and leverage their unique abilities, promoting more equitable educational practices. Additionally, targeted support should be provided to address the specific challenges underserved students face, ensuring inclusive learning environments. For instance, my research findings indicate that students with autism exhibit strengths in visuospatial reasoning and are drawn to STEM fields. However, autistic students may benefit from extra support to develop perseverance and improve their weaker areas (such as word problems) in math.

To address this gap, I am currently analyzing the NAEP process/log, performance, and survey data to study the impact of digital tools (such as text-to-speech) on student performance. This line of research is crucial and should be expanded to gather new insights on inclusive and accessible learning possibilities as technologies continue to develop.

In addition, research efforts should extend beyond traditional methods and incorporate the analysis of multimodal data. By considering a range of data sources, including behavior log/process data, speech, facial expressions, and eye-tracking data, we can gain deeper insights into how students interact with digital learning and assessments. This comprehensive approach enables us to capture nuanced aspects of their experiences and informs the design and implementation of effective educational interventions and digital learning platforms.

What advice would you give to emerging scholars from underrepresented, minoritized groups that are pursuing a career in education research?

First and foremost, seek out a great mentor and research team. Having someone to guide and support you in the field can be tremendously beneficial to your career. Look for someone who shares your research interests, is supportive of your goals, and is committed to helping you succeed. Learning from others in your team is a great way to improve your skills and knowledge.

Second, don’t be afraid of change. The greatest opportunities often require stepping out of your comfort zone and exploring new research areas or methodologies. Be open to feedback and new perspectives that can help you grow as a researcher.

Third, be brave! It is important to recognize that your unique experiences and perspectives are valuable assets to the research community. Do not be afraid to share your ideas and contributions with others. Being proactive about your work can be a great way to build your network and collaborate with other researchers in the field.

Lastly, know that you have the potential to lead a research team yourself. Keep working hard, stay focused on your goals, and do not be afraid to take on leadership roles when the opportunities arise. Pursuing this career as an emerging scholar from an underrepresented or minoritized group can be challenging but also incredibly rewarding, and you can make a meaningful impact in the field and inspire others to follow in your footsteps.

Dr. Xin Wei is currently a senior quantitative researcher at Digital Promise. Prior to joining Digital Promise, she held the position of principal research scientist at SRI International for a duration of 15 years. She specializes in using applied experimental design, statistical and machine-learning techniques to evaluate and improve instruction, interventions, assessments, and policies. In addition to her current IES study, Dr. Wei has designed and directed statistical analysis of more than 26 grants funded by federal agencies.

Produced by NCER program officer Wai Chow (Wai-Ying.Chow@ed.gov) and Virtual Student Federal Service intern Audrey Im.

NCER student volunteer, Rachael Higham, has long been interested in writing instruction. She currently works as a remedial language tutor for high school students with disabilities, and she began her graduate studies with a focus on postsecondary writing instruction. To learn more about the current science and research on writing, Rachael interviewed Dr. Charles MacArthur about his research-based postsecondary writing curriculum, Supporting Strategic Writers (SSW), which he and his team developed and evaluated through IES grants. The goal of SSW is to foster metacognitive self-evaluation through the use of strategic learning and genre-based pedagogy to help improve writing skills and self-confidence.

Take a minute to answer this question: Do you remember how you were taught to write a paper in high school or in college?

Maybe you remember the five-paragraph essay, MLA formatting, or the RACE strategy, but were you ever taught specific strategies for planning and evaluating your papers?

While I was interviewing Dr. MacArthur about his recently completed IES project, he posed a similar question to me. He asked me how I navigated writing in college and if a teacher had ever explicitly taught me how to write. I realized that while I had some explicit teaching in text structure in high school, by the time I reached college, I relied heavily on feedback to inform my future writing. The idea that students learn from revising is a common view in writing education. However, this view does not always consider students who struggle with writing and who may need more explicit instruction, even in college.

As a teacher of high school students with learning disabilities, I often find that by the time many of my students reach my classroom, they feel defeated by the writing process. Writing is something that has become a source of fear and dread for them. My goal with each student is to find and develop strategies that bolster their writing skills and change writing from something that seems unattainable to something that they can do independently. I was excited to talk to Dr. MacArthur and learn more about the research that he and his have been doing. Below are his responses to the questions I posed.

What are the key components of the SSW curriculum?

The emphasis of SSW is to enable students to take control of their own learning through rhetorical analysis of genre. To do that, students are taught explicit strategies and cognitive procedures based on what good writers do. This is reinforced with metacognitive strategies that help students become aware of why they are using specific writing strategies and procedures and recognize how and when to transfer them to other classes. SSW places emphasis on genre-based strategies not only in the text but also in the planning and evaluation phases.

The heart of strategy instruction in SSW is the “think-aloud,” which is when instructors share, in real time, the thoughts that they are experiencing as they’re writing or editing a text to show how they are figuring things out. Instructors need to show—not just explain—how to write. What we writing instructors are teaching is invisible, so the think aloud makes the process visible to students. It also lets students see that writing is hard even for their teacher. Teachers can get stuck and need to work through it based on the strategies that are being taught.

What is the number one thing that you would tell a developmental or first-year writing teacher?

Teaching strategies to students on planning and evaluating their work helps improve writing. There have been hundreds of studies from K-12 (see these meta-analyses as examples 1, 2, 3) that show how strategy instruction works to improve writing. This experimental study of SSW adds to that literature and shows that strategic instruction with genre pedagogy can work in the postsecondary developmental writing environment.

What type of future research would you like to see done with the SSW curriculum?

There is a wealth of valuable research that could be done in the future. Future research could delve into how to build on the developmental course’s gain in subsequent courses. For example, it would be interesting to look at the transition between developmental writing courses and first-year composition in terms of pedagogical integration.

Another area of transfer is between compositions courses and disciplinary writing in postsecondary settings. For example, how could postsecondary institutions improve writing across the curriculum? How could strategy instruction similar to SSW work in this setting?

Additionally, strategy instruction started in special education, but it was found to be useful throughout the entire K-12 population. Similarly, SSW was found to be successful in developmental writing classrooms. It would be great to see the effects of SSW in first year composition classes.

You can find publications from this project and the earlier SSW project in ERIC here and here respectively. The What Works Clearinghouse also reviewed an earlier evaluation of the SSW here.

This blog was written by Rachael Higham, a graduate intern through the Virtual Student Federal Service Internships program, and facilitated by Dr. Meredith Larson (Meredith.Larson@ed.gov), a research analyst and program officer at NCER.