Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: The Usability of Social-emotional and Behavioral Interventions: Links to Implementation and Translation to Youth Outcomes (USABILITY)
Center: NCER Year: 2023
Principal Investigator: Lyon, Aaron Awardee: University of Washington, Seattle
Program: Social, Emotional, and Behavioral Context for Teaching and Learning      [Program Details]
Award Period: 3 years (09/01/2023 – 08/31/2026) Award Amount: $1,699,995
Type: Exploration Award Number: R305A230391

Co-Principal Investigator: Bruns, Eric

Purpose: In this project, the researchers will identify what makes social, emotional, and behavioral interventions (SEBIs) usable in elementary and middle schools. An array of evidence-based SEBIs are now available at the universal (Tier 1) level of multi-tiered systems of supports (MTSS) frameworks, but most attempts to implement and sustain Tier 1 SEBIs are unsuccessful. Research indicates that the design of the programs and practices themselves is a critical determinant of adoption and use and that, if these do not match the context, schools may struggle to implement the SEBI. And yet, the extent and impact of usability problems among SEBIs remains largely unexamined in research. Unlike academically focused interventions in schools, SEBIs are more likely to be viewed as misaligned with the core educational mission of schools, creating additional implementation barriers and a heightened need for usability. A key objective of this project will be to identify a set of principles that can be used to guide researchers as they develop or refine SEBIs to optimize their usability.

Project Activities: The research team will conduct three studies. In study 1, teachers and students will participate in systematic, scenario-driven, task-based usability testing to identify the number, type, and severity of usability problems across 10 SEBIs. In study 2, they will survey school administrators and staff and collect academic administrative data to determine how specific aspects of usability are associated with targeted outcomes. In the final study, they will convene a 2-day expert summit where a broad set of SEBI developers, stakeholders, and expert advisors from across the country will convene to prioritize identified usability issues and develop SEBI redesign recommendations for common problems that are most associated with problematic implementation and student outcomes.

Products: Project results will yield a set of outputs designed to inform and enhance the ability to implement Tier 1 SEBIs. These include

  1. a set of comprehensive and systematic usability data for leading evidence-based, Tier 1 SEBIs
  2. information about which specific aspects of SEBI usability facilitate or inhibit desired student and implementation outcomes
  3. an array of materials to provide guidance to the field, including a revised theory of change for how SEBI usability drives outcomes, guidance on how to assess SEBI usability, and a SEBI usability practice guide, which will include a matrix of usability problems and design solutions for use by intervention developers working in formal education settings

Structured Abstract

Setting: The research will take place in racially and socioeconomically diverse school districts in Washington state. The researchers will also collaborate with SEBI developers and purveyors across a range of geographically diverse schools nationwide and will host a culminating summit in Washington state.

Sample: Participants include teachers (n = 125) and students (n = 125) from approximately 40 schools in Washington state (study 1) and administrators and key school staff (n = 250) from approximately 50 districts across the country (study 2). The USABILITY Learning Community will include a broad set of SEBI developers, stakeholders, and expert advisors from across the country (n = 22; study 3). Samples for Studies 1 and 2 will be aligned using a common recruitment survey assessing demographics, experience, and attitudes.

Factors: This research team will examine the interrelationships among aspects of SEBI usability; implementation outcomes such as adoption; school outcomes such as climate; and student social, emotional, behavioral, and academic outcomes. In addition, the USABILITY Learning Community will develop strategies for redesigning and developing usable SEBIs based upon the SEBI usability problems and strategies identified in studies 1 and 2.

Research Design and Methods: In Study 1 (usability assessment), researchers will identify the number, type, and severity of usability problems and overall ratings of usability for 10 SEBIs. Teachers and students will be given a random subset of SEBIs to review using a structured user testing method. Study 2 (linking usability to implementation outcomes and student outcomes) will use survey methods and existing academic data to determine the adoption, use, and impact of each intervention. Study 3 (stakeholder-driven redesign solutions) will engage the USABILITY Learning Community in a systematic process to develop intervention redesign recommendations for usability problems that have been most associated with problematic implementation and student outcomes.

Control Condition: Due to the nature of the design, this project does not include a control condition.

Key Measures: Key measures will include the Intervention Usability Scaleand a qualitative user testing process to identify the number, type, and severity of usability problems for each SEBI. The researchers will also survey schools around the country to gather information about implementation strategies, implementation outcomes, implementation climate, and student outcomes associated with each of the 10 SEBIs.

Data Analytic Strategy: For study 1, the researchers will determine the number, type, and severity of usability problems across 10 SEBIs using descriptive statistics and qualitative content analysis. They will analyze study 2 data to determine the associations between specific aspects of usability and targeted outcomes using generalized mixed effects models using restricted maximum likelihood estimation with random effects for school staff and school, including covariates to control for possible confounding heterogeneity. They will formulate SEBI redesign recommendations using qualitative content analysis and the Framework for Reporting Adaptations and Modification to Evidence-based Interventions.

Related IES Projects: The Organizational Assessment for Strategic Implementation in Schools (OASIS): A Measurement Suite to Support Educator Adoption of Evidence-Based Practices (R305A160114), Helping Educational Leadership Mobilize Evidence (HELM): An Organizational Intervention to Promote Strategic Implementation in Schools (R305A200023), A Research Institute for Implementation Science in Education (RIISE) to Address the "Last Mile" in Education (R305B210004)