Inside IES Research

Notes from NCER & NCSER

Developing the Vanderbilt Assessment of Leadership in Education (VALED)

As education accountability policies continue to hold school leaders responsible for the success of their schools, it is crucial to assess and develop leadership throughout the school year. In honor of the IES 20th Anniversary, we are highlighting NCER’s investment in leadership measures. This guest blog discusses the Vanderbilt Assessment of Leadership in Education (VALED). The VALED team was led by Andy Porter and included Ellen Goldring, Joseph Murphy and Steve Elliott, all at Vanderbilt University at the time. Other important contributors to the work are Xiu Cravens, Morgan Polikoff, Beth Minor Covay, and Henry May. The VALED was initially developed with funding from the Wallace Foundation and then further developed and validated with funding from IES.

What motivated your team to develop VALED?

There is currently widespread agreement that school principals have a major impact on schools and student achievement. However, at the time we developed VALED, we noticed that there were limited research-based instruments to measure principal leadership effectiveness aligned to both licensure standards and rooted in the evidence base. Prior to the VALED, principal leadership evaluation focused primarily on managerial tasks. However, we believed that principal leadership centered on improving teaching and learning, school culture, and community and parent engagement (often called learning-centered leadership) is at the core of leadership effectiveness.

What does VALED measure?

The VALED is a multi-rater assessment of learning-centered leadership behaviors. The principal, his/her supervisor, and teachers in the school complete it, which is why VALED is sometimes referred to as a 360 assessment or multi-source feedback.

VALED measures six core components and six key processes that define learning-centered leadership. The core components are high standards for student learning, rigorous curriculum, quality instruction, culture of learning and professional behavior, connections to external communities, and performance accountability. The key processes are planning, implementing, supporting, communicating, monitoring, and advocating.

How is the VALED different from other school leadership assessments?

The VALED is unique because it focuses on school leadership behaviors aligned to school improvement and school effectiveness, incorporates feedback and input from those who collaborate closely with the principal, includes a self- assessment, acknowledges the distributed work of leadership in a school, and has strong psychometric properties. We think there are several elements that contribute to the uniqueness of the instrument.

First, VALED is based on what we have learned from scholarship and academic research rather than less robust frameworks such as personal opinions and or unrepresentative samples. The VALED was crafted from concepts identified as important in that knowledge and understanding. The VALED model is based upon knowledge about connections between leadership and learning and provides a good deal of the required support for the accuracy, viability, and stability of the instrument.

Second, principals rarely receive data-based feedback, even though feedback is essential for growth and improvement. The rationale behind multi-source or 360-degree feedback is that information regarding leadership efficacy resides within the shared experiences of teachers and supervisors, collaborating with the principal, rather than from any one source alone. Data that pinpoint gaps between principal’s own self-assessment, and their teachers’ and supervisors’ ratings of their leadership effectiveness can serve as powerful motivators for change.

Finally, in contrast to some other leadership measures, VALED has undergone extensive psychometric development and testing. We conducted a sorting study to investigate content validity and a pilot study where we addressed ceiling effects, and cognitive interviews to refine wording. We also conducted a known group study that showed the tool’s ability to reliably distinguish principals, test-retest reliability, convergent-divergent validity, and principal value-added to student achievement. As part of this testing, we identified several key properties of VALED. The measure—  

  • Works well in a variety of settings and circumstances
  • Is construct valid
  • Is reliable
  • Is feasible for widespread use
  • Provides accurate and useful reporting of results
  • Is unbiased
  • Yields a diagnostic profile for summative and formative purposes
  • Can be used to measure progress over time in the development of leadership
  • Predicts important outcomes
  • Is part of a comprehensive assessment of the effectiveness of a leader's behaviors

What is the influence of VALED on education leadership research and practice?

VALED is used in schools and districts across the US and internationally for both formative and evaluative purposes to support school leadership development. For example, Baltimore City Public Schools uses VALED as a component of their School Leader Evaluations. VALED has also spurred studies on principal evaluation, including the association between evaluation, feedback and important school outcomes, the implementation of principal evaluation, and its uses to support principal growth and development. In addition, it provides a reliable and valid instrument for scholars to use in their studies as a measure of leadership effectiveness.


Andy Porter is professor emeritus of education at the Pennsylvania State University. He has published widely on psychometrics, student assessment, education indicators, and research on teaching.

Ellen Goldring is Patricia and Rodes Hart Chair, professor of education and leadership at Vanderbilt University. Her research interests focus on the intersection of education policy and school improvement with emphases on education leadership.

Joseph Murphy is an emeritus professor of education and the former Frank W. Mayborn Chair of Education at Peabody College, Vanderbilt University. He has published widely on school improvement, with special emphasis on leadership and policy and has been led national efforts to develop leadership standards. 

Produced by Katina Stapleton (Katina.Stapleton@ed.gov), program officer for NCER’s education leadership portfolio.

 
 
 

The Comprehensive Assessment of Leadership for Learning: How We Can Support School Leaders to Improve Learning for All Students

As educational accountability policies continue to hold school leaders responsible for the success of their schools, it is crucial to assess and develop leadership throughout the school year. In honor of School Principals’ Day and the IES 20th Anniversary, we are highlighting NCER’s investment in formative leadership measures. In this guest blog, researchers Rich Halverson and Carolyn Kelley from the University of Wisconsin-Madison and Mark Blitz from the Wisconsin Center for Education Products and Services discuss the development and evolution of their IES-funded Comprehensive Assessment of Leadership for Learning (CALL).

What is CALL?

CALL is a survey tool based on a distributed leadership model that emphasizes the work of leaders rather than their positions or identities. In 2008, we led a team of researchers at the University of Wisconsin-Madison to identify the key leadership tasks necessary for school improvement, regardless of who made the tasks happen. The CALL survey invites each educator in a school to assess the degree to which these core tasks are conducted, then aggregates these responses to provide a school-level portrait of the state of leadership practice in their school.

How was CALL developed?

Our CALL team relied on over 30 years of research on leadership for school improvement to name about 100 key tasks in five domains of practice. The team then worked over a year with expert educators and leaders to articulate these tasks into survey items phrased in language that teachers would readily understand as describing the work that happens every day in their schools. We designed each item to assess the presence and quality of leadership practices, policies, and programs known to improve school quality and student learning. We validated the survey with qualitative and quantitative analyses of survey content, structure, and reliability.

What inspired you to develop CALL?

We believed a measure like CALL is necessary in the era of data-driven decision-making. Educators are inundated by accountability and contextual data about their schools, but they are left on their own for data to help them understand how to develop and implement the strategies, policies, and programs that support student success. Traditional school data systems leave a hole where feedback matters most for educators–at the practice-level where the work of leaders and educators unfolds. That is the hole that CALL is designed to fill.

How is the CALL different from other leadership surveys?

Traditional surveys include items that invite educators to rate their leaders on important tasks using Likert scale measures. The results of these surveys produce scores that allow leaders to be rated and compared. But, as a school leader, it is hard to know what to do with a 3.5 score on an item like “My principal is an effective instructional leader.” CALL items are designed differently. Each CALL item response represents a distinct level of practice, so respondents can learn about optimum practices simply by taking the survey. If the collected responses by educators in your school averaged a “2” on one of the items, the description of the next level practice (“3”) clearly articulates an improvement goal.

In addition, our online CALL reporting tools provide formative feedback by allowing users to compare item and domain scores between academic departments and grade levels, as well as across schools. The reports name specific areas of strength and improvement, and also suggest research-driven strategies and resources leaders can use to improve specific aspects of leadership.

How did CALL transition into a commercial measure?

The CALL project provides a model of how IES-funded research can have broad impact in schools around the country. We are thrilled that CALL developed into the rare educational survey that was embraced by the people who tested it as well as the research community. Many of our development partners asked about whether they could continue with CALL as the survey took on new life as a commercial product after our grant ended.

The Wisconsin Center for Education Products and Services (WCEPS) provided us with the business services and the support to bring CALL to market. CALL became a WCEPS partner in 2014 and has since developed into a successful leadership and school improvement resource. Under the leadership of WCEPS’s Mark Blitz, the CALL model became a framework to build successful collaborations with learning and research organizations across the country.

Leading professional learning groups such as WestEd, WIDA, the Southern Regional Education Board, and the Georgia Leadership Institute for School Improvement worked with Mark and the WCEPS team to build customized CALL-based formative feedback systems for their clients. Research partners at East Carolina University, Teachers College, and the University of Illinois at Chicago used CALL to collect baseline data on leadership practices for school improvement and principal preparation projects. CALL has also developed customized versions of the survey to support leadership for personalized learning (CALL PL) and virtual learning (Long Distance CALL). These partnerships have provided opportunities for hundreds of schools and thousands of educators to experience the CALL model of formative feedback to improve teaching and learning in schools.

What’s the next step for CALL?

In 2021, the CALL project entered a new era of leadership for equity. With the support of the Wallace Foundation, we created CALL for Equity Centered Leadership (CALL-ECL) to provide school districts with feedback on the leadership practices that create more equitable schools. CALL-ECL is part of a $100 million+ Wallace Foundation initiative to transform how districts across the country develop partnerships to prepare and support a new generation of equity-centered leaders. According to Wallace Research Director Bronwyn Bevan, “The foundation is excited about CALL-ECL because it will help leaders identify the organizational routines that sustain inequality and replace them with routines that help all students thrive.”

Our $8 million, six-year CALL-ECL project will document the development of these new preparation and support program, and will create a new CALL survey as an information tool to describe and assess equity-centered leadership practices. We believe that by 2027, CALL-ECL will be able to share the practices of equity-centered leadership developed through the Wallace initiatives with districts and schools around the world. Our hope is that CALL-ECL will give school leaders and leadership teams the data they need to continually evolve toward better opportunities and outcomes for all young people.


Richard Halverson is the Kellner Family Chair of Urban Education and Professor of Educational Leadership and Policy Analysis in the UW-Madison School of Education. He is also a co-director of the Comprehensive Assessment of Leadership for Learning and leads the Wallace Foundation Equity-Centered Leadership Pipeline research project.

 

Carolyn Kelley is a distinguished professor in the Department of Educational Leadership and Policy Analysis. Dr. Kelley’s research focuses on strategic human resources management in schools, including teacher compensation, principal and teacher evaluation, and leadership development.

 

Mark Blitz is the project director of the Comprehensive Assessment of Leadership for Learning (CALL) at the Wisconsin Center for Education Products & Services.

 

 

This blog was produced by Katina Stapleton (Katina.Stapleton@ed.gov), program officer for NCER’s education leadership portfolio.

 
 
 

NASA to Kick Off Its Latest National Student Challenge at the 2021 ED Games Expo on June 1

The 8th Annual ED Games Expo will occur next week from June 1 to 5. The free event is all virtual, open to the public, and will showcase game-changing innovations in education technology developed through more than 40 programs at the Department of Education (ED) and across the federal government.

 

NASA National Student Challenge Event at the Ed Expo

One of many noteworthy Expo events will occur on Tuesday, June 1, from 6 to 8 PM Eastern when NASA’s Flight Opportunities program will introduce a new national student challenge. Educators can register to attend this LIVE event on June 1 here. The NASA TechRise Student Challenge will invite teams of sixth- to 12th-grade students to submit ideas for climate or remote sensing experiments to fly on a high-altitude balloon, and space exploration experiments to fly aboard a suborbital rocket.

 

 

NASA developed the NASA TechRise Student Challenge to enable students to have a deeper understanding of Earth’s atmosphere, space exploration, coding and electronics, and a broader understanding of the value of test data. The challenge will also provide students with the opportunity to engage with NASA and technology communities and expose them to careers in science, technology, and space exploration fields.

The challenge will begin accepting applications in August for student teams affiliated with U.S. public, private, and charter schools, including U.S. territories.  The winning teams each will receive $1,500 to build their payloads, as well as an assigned spot on a NASA-sponsored commercial suborbital flight. Balloon flights will offer more than four hours of flight time, while suborbital rockets will provide around three minutes of test time in microgravity conditions. The Flight Opportunities program, based at NASA’s Armstrong Flight Research Center, a part of Space Technology Mission Directorate, is leading the NASA TechRise Student Challenge. The challenge is being administered by California-based Future Engineers, which developed its platform with awards in 2016 and 2017 from the ED/IES SBIR program. Future Engineers’ platform has also been employed to manage past educational and NASA challenges, including the Name the Mars Rover student challenge. 

During the June 1 event, NASA experts will provide information to educators on the official competition. Teachers are invited to join a NASA TechRise Educator Summer Workshop, which will dive into the basics of electronics, coding, and designing for flight. The first workshop will be on Wednesday, July 28, 2021 and repeated on Wednesday, August 11, 2021. For challenge details and to pre-register for the competition, please visit the contest website.

 

More ED Games Expo Events to Engage Students in Hands-On Projects and Challenges

In addition to the NASA event, five more virtual events featuring government programs that engage students in project-based learning will occur on Monday, June 1 between 12:30PM to 6PM Eastern. Topics include students building and flying satellites, programs for museums, local communities and military facilities to engage students in experiential and real-world learning, and a program to inspire students to be inventors and entrepreneurs. See the Expo Agenda here for lineup of events and the ED Games Expo Playlists Page for video trailers by participating developers. 

 

We look forward to "seeing you" at the virtual ED Games Expo starting on June 1!


Edward Metz (Edward.Metz@ed.gov) is the Program Manager for the Small Business Innovation Research program at the US Department of Education’s Institute of Education Sciences.

 

Principals as Instructional Leaders and Managers—Not an “Either-Or”

Recently Morgaen Donaldson (University of Connecticut) and Madeline Mavrogordato (Michigan State), Peter Youngs (University of Virginia), and Shaun Dougherty (University of Connecticut) presented early results from their IES-funded study on principal evaluation policies at the AERA national conference. We asked the team to share their preliminary findings.

What is the purpose of your study?

There is widespread agreement among researchers, policymakers, and practitioners that principals play a critical role in providing high-quality education to students. The role of principals may grow even larger under the 2015 Every Student Succeeds Act, which grants districts and states more flexibility regarding how to promote effective principal leadership. However, we know remarkably little about what school districts can do to improve principals’ leadership practices.

Given the importance of principals and the relative dearth of research on how to improve their leadership, we have been studying the extent to which principal evaluation systems focus on learning-centered leadership, one promising conception of leadership, in 22 districts in Connecticut, Michigan, and Tennessee. We are examining associations between the types of leadership emphasized in principal evaluation policies, the leadership practices that principals implement in their schools, and student performance.  

What is the major focus of principal evaluation policies?

To date, through document analysis we have found that district principal evaluation policies heavily emphasize instructional leadership, which focuses on teaching and learning issues, and de-emphasize managerial leadership, which concentrates on administrative tasks such as budgeting and overseeing school facilities. Similarly, in interviews and surveys superintendents and principals report that their evaluation systems focus on instructional leadership. For example, one Michigan superintendent said, “the emphasis on education right now [is] to take the principals away from being a manager to being an instructional leader.”

How are administrators actually interpreting the policies?

Further investigation revealed a more nuanced relationship between written district policies and administrators’ interpretations of these policies, however. We found no relationship between written policies’ emphasis on instructional leadership and principals’ survey responses regarding whether their district focused on this type of leadership. Principals’ perceptions of their district’s focus on managerial leadership was related to the emphasis of this type of leadership in the policies, however. Thus, when districts placed a higher emphasis on managerial leadership in their written evaluation policies, principals reported that they perceived a stronger emphasis on this type of leadership.

Moreover, we found that holding constant the written policy’s emphasis on managerial leadership, there was an inverse relationship between the written policies’ emphasis on instructional leadership and the principals’ perceived policy emphasis on managerial leadership. Thus, the greater the written emphasis on instruction, the less principals perceived that their policy emphasized management.

In addition, interview data reveal that although superintendents state that they emphasize instructional leadership they in fact weigh managerial leadership quite heavily. In superintendents’ framing, a principal’s competence in managerial leadership enabled him or her to practice instructional leadership. Superintendents asserted that when principals addressed managerial concerns, they could progress to exercising instructional leadership. If principals were unable to address managerial issues, superintendents reported that they moved rapidly to intervene and potentially remove principals.

What are the next steps for your IES research project?

These preliminary findings add to a growing body of evidence suggesting a complex interplay between managerial and instructional leadership. They also reflect a longstanding tension between the two dominant conceptions of principal leadership among practitioners.  We plan to further examine the multifaceted relationship between instructional and managerial leadership as we continue our work on this project. We are currently surveying teachers about the types of leadership principals exercise in their schools and conducting a second round of interviews with superintendents to understand their perspectives in greater depth. In the next stage of the project, we will examine associations between the types of leadership emphasized in principal evaluation policies, the leadership practices that principals implement in their schools, and student performance in grades 3-8. 

Katina Stapleton, NCER Program Officer, oversees the project described in this blog post, and provided a framework for their responses.

A Renewed Focus on Education Leadership

Education leaders, such as school principals, have been the focus of education research for decades. While research suggests that there are substantial (albeit mostly indirect) relationships between school leadership and student achievement, there is still much to learn. More information is needed about how the knowledge, skills, abilities, and actions of school leaders are impacting student outcomes and the best ways to prepare and support school leaders.

Since 2004, the National Center for Education Research (NCER) within the Institute of Education Sciences (IES) has only funded 15 research studies on education leadership through several education research grant programs. A technical working group met last year and recommended that NCER strengthen its portfolio of research on this important topic.

In response, IES is revamping its Education Leadership topic as a separate part of its Education Research grants program. By doing so, we hope to provide a greater focus on education leadership research and increase understanding of how school leadership can improve achievement and opportunities for students.

Through the leadership topic, IES is offering research opportunities on programs, policies, and practices that support leaders in K-12 education systems at the school, district, or state level and ultimately lead to improved student outcomes.  While IES invites all applications that meet the topic requirements, we are especially interested in

  • exploratory research on the specific competencies and behaviors needed by leaders to support at-risk or high-risk students and improve student outcomes in challenging educational settings, such as persistently low-performing schools and high-poverty schools and districts;
  • exploratory research on the relationship between student education outcomes and district policies regarding the identification and selection of education leaders, assignment of leaders to specific schools, leadership turnover, and the distribution of leadership roles and responsibilities among multiple individuals within a school;
  • evaluations of leadership interventions that have the potential to improve student outcomes; and
  • the validation of existing leadership measures and the development and validation of new leadership measures for the purpose of research, formative assessment, and accountability.

Education leadership researchers who have strong partnerships with school districts and/or state education agencies should also consider applying to the Researcher-Practitioner Partnerships in Education Research topic within the Partnerships and Collaborations Focused on Problems of Practice or Policy grants program.  This topic allows researchers to carry out initial research that is of importance to district and/or state partners and then develop a plan for future research.

For more information on funding opportunities for research on education leadership contact Katina Stapleton (Katina.Stapleton@ed.gov) or Corinne Alfeld (Corinne.Alfeld@ed.gov) or visit the IES website. If you missed the deadline to submit an optional Letter of Intent for the Education Leadership topic, please email it to Katina or Corinne directly.

A One-Stop Shop for Leadership Research

We have consolidated all education research grants related to education leaders under our Education Leadership research portfolio. The focus of these studies varies and includes the development and validation of leadership measures (such as the Vanderbilt Assessment of Leadership in Education (VAL-ED) and the exploration of potential relationship(s) between malleable factors (such as principals’ skills and behaviors) and student outcomes. In addition to these studies, NCER has funded research on education leadership through its R&D centers on school choice, scaling up effective schools, and analyzing longitudinal education data

Written by Katina Stapleton, Education Research Analyst, NCER

PHOTO: Principals meet with the U.S. Department of Education leadership in 2015 (courtesy of U.S. Department of Education)