Large-scale surveys can provide state and district education leaders with a deeper understanding of how recommendations are enacted and resources are used. However, survey data are only actionable if the survey asks the right questions and yields sufficient responses.

Recently, REL Northwest helped the Alaska Department of Education and Early Development (AK DEED) and its partners develop and administer a statewide school-level survey to better understand how schools and districts implement trauma-engaged practices and policies across the state.

To ensure survey responses reflected perspectives across a wide variety of roles, from teachers to counselors to school leaders, AK DEED asked each school to convene a team of staff familiar with the school's use of trauma-engaged practices (e.g., teachers, school counselors, administrators). Teams gathered for an hour to complete the survey. Despite the complexity of the survey, nearly 60% of public K-12 schools across Alaska responded.

Below, we walk through each step of the survey process, summarize Alaska's approach, and provide the strategies and lessons learned that may be useful for similar survey efforts by other state education agencies or school districts.

The Need for a Trauma-Engaged Schools Implementation Survey
Children that experience trauma are at a higher risk for poor educational, social, health, and economic outcomes. However, providing children with positive and supportive experiences helps to counteract the impact of adverse childhood experiences (Bethell et al., 2014; Blodgett & Lanigan, 2018; Duke, 2020; Flaherty et al., 2013; Hughes et al., 2017; Hunt et al., 2017).

Schools can play a vital role in facilitating positive and supportive experiences for students. In Alaska, two in three children are exposed to traumatic experiences (Alaska Department of Health and Social Services, 2015). In 2019, Alaska released Transforming Schools: A Framework for Trauma-Engaged Practice in Alaska to better equip educators to aid students experiencing trauma. Since its release, the Transforming Schools Framework and suite of accompanying resources have been disseminated to districts and schools statewide. However, little was known about how the framework was used in schools across Alaska.

To ensure schools had the resources needed to use a trauma-engaged approach, the Alaska Department of Education and Early Development (AK DEED) and its partners – the Association of Alaska School Boards, Alaska Department of Health, Alaska Mental Health Trust Authority, and the University of Alaska Anchorage Center for Human Development – wanted to create a survey to better understand how schools and districts are implementing the Transforming Schools framework.

"Many Alaskan schools report that supporting students' mental health, especially those who are coping with trauma and adversity is essential to making schools a place to learn and thrive. This survey and subsequent support will help to focus on what trauma-engaged practices districts are using and ultimately plan for how best to educate and care for students." Patrick Sidmore, program coordinator in the Alaska School Health and Safety Team at AK DEED, explained.

The Transforming Schools framework encourages educators to co-create, reflect, and respect cultural expertise as they develop and refine a trauma-engaged approach. Thus, REL Northwest sought to ensure every step of the survey development and administration process was purposefully designed to elevate co-creation, collaboration, reflection, and flexibility.

Survey development
REL Northwest facilitated a working group to develop the trauma-engaged practices survey over the course of five meetings that spanned six months. The working group included highly engaged individuals representing state agencies, universities, associations, and school districts. After an iterative process, the resulting survey reflected collective priorities for understanding how schools and districts implement trauma-engaged approaches across Alaska.

Lessons Learned:

• Dedicate sufficient time. When designing a survey, allow enough time for both independent reflection and group dialogue to collaboratively refine the instrument.
• Include diverse voices. To ensure the survey resonates across roles and sites, involve individuals with unique expertise who are well-positioned to gather feedback from their colleagues. Including multiple perspectives from the beginning improves both quality and cross-organization buy-in.  

Survey administration
Informed by coaching from REL Northwest, Alaska deployed a coordinated communications strategy to promote the survey. The strategy included messaging from multiple individuals in varied roles at AK DEED, as well as messaging from partner organizations and agencies. Throughout the survey administration process, REL Northwest supported the survey team to check response rates weekly, allowing purposeful time to think through gaps in responses and craft outreach plans to target those who hadn't responded. REL Northwest also developed guidance for AK DEED outlining the steps to regularly check response rates.

Although AK DEED began the survey administration process with a plan in place, REL Northwest encouraged agency staff to maintain a flexible mindset and pivot as needed to achieve a high response rate. For example, during the survey administration process, coaching from REL Northwest informed AK DEED's decision to add additional outreach strategies that included broad messaging from multiple leaders in the state and individualized outreach from AK DEED staff well-acquainted with schools across the state. AK DEED also added school-level cash incentives for survey completion and modified how survey links were distributed to make it easier for recipients to respond (e.g., using a general link that could be easily shared via personal correspondence in addition to individualized survey links distributed through the SurveyMonkey platform).

Lessons Learned:

• Use varied outreach strategies. Create email blasts, personalized emails, newsletters, and use one-on-one conversations to bolster response rates. Coordinate joint and complementary messaging from multiple agencies or organizations that are viewed as leaders in the state to reiterate the importance of completing the survey.
• Check and reflect on response rates. Regularly review response rates throughout the survey administration process to identify potential issues and develop strategies to address those issues. Regular response rate checks may help identify problematic contact information that suggests the need for additional encouragement from state leaders or identify a more personalized outreach strategy for specific districts or schools. 
• Be flexible. Our best-laid plans may not reflect the best approach. Don't be shy about pivoting and adjusting the plan if you're not getting the response rates you anticipated. Consider changes to how outreach occurs, extending the survey administration window, or how data is collected.

Survey completion
Each school principal was asked to convene a team of diverse voices and provide an hour for collaborative survey completion. Although AK DEED requested that a school team, rather than an individual, complete the survey, the exact number of team members and roles were left to the principal's discretion. This flexible team approach with dedicated time for the survey allowed participants to reflect and engage in discussion to provide survey responses that reflected the school's perspective rather than one individual's perceptions.

Lessons Learned:

• Allow flexibility in survey respondents. Empower school leaders to ensure the survey completion team reflects the culture of each school. Staffing structures, as well as responsibilities and perspectives of staff, vary across schools.
• Encourage dedicated time for survey completion. Completing a survey that reflects schoolwide practices and policies takes time. To promote reflective and collaborative survey completion, encourage district and school leaders to carve out time and provide incentives to recognize the time investment.

The trauma-engaged schools implementation survey example highlights how applying principles of co-creation, collaboration, cultural respect, and flexibility can lead to a survey tool that reflects multiple perspectives and priorities, a high response rate, and valuable data representing multiple perspectives via a single school-level survey.

With data collection complete for the trauma-engaged schools' implementation survey, REL Northwest's Alaska Trauma-Engaged Schools Partnership will explore these data to provide information on which trauma-engaged strategies schools use, whether factors like school size and location influence the use of trauma-engaged strategies, and how the use of these strategies is related to student and staff outcomes (e.g., attendance, school climate).

Findings from this study will aid AK DEED and its partners in refining the state's trauma-engaged services and resources.

Additional RESOURCES

• Read about the Alaska Trauma-Engaged Schools Partnership with REL Northwest.
• The Educator's Guide to Questionnaire Development guide provides step-by-step guidance on how to create survey instruments. 
• The Program Evaluation Toolkit offers helpful information on evaluation design, sampling, data collection, and more.
• The Workshop on Survey Methods: Facilitator's guide and resources resource for those leading workshops on survey methods.

Please subscribe to our newsletter, reach out via email, follow us on Twitter, and visit our Website to learn more about REL Northwest.