Project Activities
The research team will conduct three studies. In study 1, teachers and students will participate in systematic, scenario-driven, task-based usability testing to identify the number, type, and severity of usability problems across 10 SEBIs. In study 2, they will survey school administrators and staff and collect academic administrative data to determine how specific aspects of usability are associated with targeted outcomes. In the final study, they will convene a 2-day expert summit where a broad set of SEBI developers, stakeholders, and expert advisors from across the country will convene to prioritize identified usability issues and develop SEBI redesign recommendations for common problems that are most associated with problematic implementation and student outcomes.
Structured Abstract
Setting
The research will take place in racially and socioeconomically diverse school districts in Washington state. The researchers will also collaborate with SEBI developers and purveyors across a range of geographically diverse schools nationwide and will host a culminating summit in Washington state.
Sample
Participants include teachers (n = 125) and students (n = 125) from approximately 40 schools in Washington state (study 1) and administrators and key school staff (n = 250) from approximately 50 districts across the country (study 2). The USABILITY Learning Community will include a broad set of SEBI developers, stakeholders, and expert advisors from across the country (n = 22; study 3). Samples for Studies 1 and 2 will be aligned using a common recruitment survey assessing demographics, experience, and attitudes.
This research team will examine the interrelationships among aspects of SEBI usability; implementation outcomes such as adoption; school outcomes such as climate; and student social, emotional, behavioral, and academic outcomes. In addition, the USABILITY Learning Community will develop strategies for redesigning and developing usable SEBIs based upon the SEBI usability problems and strategies identified in studies 1 and 2.
Research design and methods
In Study 1 (usability assessment), researchers will identify the number, type, and severity of usability problems and overall ratings of usability for 10 SEBIs. Teachers and students will be given a random subset of SEBIs to review using a structured user testing method. Study 2 (linking usability to implementation outcomes and student outcomes) will use survey methods and existing academic data to determine the adoption, use, and impact of each intervention. Study 3 (stakeholder-driven redesign solutions) will engage the USABILITY Learning Community in a systematic process to develop intervention redesign recommendations for usability problems that have been most associated with problematic implementation and student outcomes.
Control condition
Due to the nature of the design, this project does not include a control condition.
Key measures
Key measures will include the Intervention Usability Scaleand a qualitative user testing process to identify the number, type, and severity of usability problems for each SEBI. The researchers will also survey schools around the country to gather information about implementation strategies, implementation outcomes, implementation climate, and student outcomes associated with each of the 10 SEBIs.
Data analytic strategy
For study 1, the researchers will determine the number, type, and severity of usability problems across 10 SEBIs using descriptive statistics and qualitative content analysis. They will analyze study 2 data to determine the associations between specific aspects of usability and targeted outcomes using generalized mixed effects models using restricted maximum likelihood estimation with random effects for school staff and school, including covariates to control for possible confounding heterogeneity. They will formulate SEBI redesign recommendations using qualitative content analysis and the Framework for Reporting Adaptations and Modification to Evidence-based Interventions.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Project results will yield a set of outputs designed to inform and enhance the ability to implement Tier 1 SEBIs. These include
- a set of comprehensive and systematic usability data for leading evidence-based, Tier 1 SEBIs
- information about which specific aspects of SEBI usability facilitate or inhibit desired student and implementation outcomes
- an array of materials to provide guidance to the field, including a revised theory of change for how SEBI usability drives outcomes, guidance on how to assess SEBI usability, and a SEBI usability practice guide, which will include a matrix of usability problems and design solutions – for use by intervention developers working in formal education settings
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
