Co-Principal Investigator: Clayton R. Cook (University of Minnesota)

Purpose: In this project, researchers will develop a suite of measures called the Organizational Assessment for Strategic Implementation in Schools (OASIS). OASIS is designed for teachers and will provide school and district leaders with information regarding how best to support and sustain the use of evidence-based practices to prevent emotional and behavioral problems among students. While research has shown that these are practices teachers and schools can use to reduce emotional and behavioral problems for students, teachers and schools are not necessarily using these practices in their classrooms. Researchers will develop four measures that will assess aspects of a school culture, including attitudes, school climate, and school leadership characteristics. Teachers will complete the assessments and school and district leaders can use data from OASIS to make decisions about how best to devote resources to carrying out effective practices in the classroom.

Project Activities: The research team will revise measures that are used to assess features of health organizations that support the use of evidence-based health practices so that they address the features of schools. Researchers will revise and develop items and then will use a series of focus groups, expert review, and field testing to further revise and refine the items. The testing will occur in schools that use one of two evidence-based programs to aim to reduce the emotional and behavioral problems of elementary school students.

Products: The products of this project include a fully developed and validated suite of assessments designed to assess characteristics of schools to support and sustain evidence-based practices in schools, and peer reviewed publications.

Structured Abstract

Setting: This project will take place in elementary schools in Washington.

Sample: Participants in this study will be approximately 700 teachers and other school personnel who work in elementary schools.

Assessment: This computer-based suite of assessments is designed to be completed by teachers and to provide school and district personnel with information regarding barriers to the implementation of evidence-based practices and to guide decision-making regarding the distribution of resources to support implementation of these practices. The assessments are being revised from the Exploration, Preparation, Implementation, Sustainment (EPIS) measures which were developed by members of the research team to examine implementation-related organizational factors in the health field. The final OASIS assessments will cover four core constructs: (1) strategic implementation climate, which encompasses staff norms and expectations about implementing new programs and practices; (2) strategic implementation leadership, which involves specific leadership behaviors that support or inhibit successful implementation; (3) implementation citizenship behavior, which are behaviors that go beyond standard core job responsibilities and may mediate the influence of climate and leadership on implementation success; and (4) attitudes toward evidence-based practices, which encompasses the attitudes of teachers and which may mediated the influence of climate and leadership on implementation. The assessment also includes cut scores indicating when the likelihood of high-quality implementation is substantially reduced.

Research Design and Methods: This project will include five studies over the three years: three assessment development studies, a large-scale validation study, and a pilot usability study. In Year 1, the team will carry out three assessment development studies: (1) focus group studies with teachers and school personnel to adapt and add items to form a preliminary item pool; (2) expert review of the tools by researchers and practitioners in order to examine conceptual representativeness and appropriateness of the measures across a range of education settings; and (3) field testing and refinement, which involves two small-scale pilots to estimate completion time, troubleshoot technical problems, and revise the item pool. In Year 2, the researchers will examine construct validity by administering the measures to a large sample of teachers from schools implementing one of two universal evidence-based prevention programs: School-Wide Positive Behavioral Interventions and Supports (SWPBIS) or Promoting Alternative Thinking Strategies (PATHS). By administering OASIS in these schools, researchers will be able to determine associations between OASIS and successful implementation of SWPBIS or PATHS. In Year 3, the researchers will conduct a usability study that will involve qualitative interviews with administrators and other stakeholders to maximize utility and usability of the OASIS system and refine the OASIS reports.

Control Condition: Due to the nature of the research design, there is not a control condition

Key Measures: In order to examine the validity of OASIS, researchers will use measures of classroom behavioral outcomes, student academic outcomes such as standardized tests and attendance, and related organizational measures. These organization measures include the Organizational Health Inventory, the Principal Instructional Management Rating Scale, the Organizational Citizenship Behavior Scale, and the Public School Teacher Questionnaire. Researchers will also examine the associations between OASIS and the successful implementation of SWPBIS and PATHS by using each program's fidelity measure and the Primary Intervention Rating Scale.

Data Analytic Strategy: Researchers will code focus group and interview data using directed content analysis. The research team will analyze validity by examining response distributions, factor analysis using latent variable modeling, and item response theory. Researchers will use multilevel latent variable models to examine the multi-level factor structure of the OASIS items and the convergent and discriminant validity.

Publications

Journal article, monograph, or newsletter

Lyon, A.R., Cook, C.R., Brown, E.C., Locke, J., Davis, C., Ehrhart, M., and Aarons, G.A. (2018). Assessing Organizational Implementation Context in the Education Sector: Confirmatory Factor Analysis of Measures of Implementation Leadership, Climate, and Citizenship. Implementation Science, 13(1), 5.

Lyon, A.R., Whitaker, K., Locke, J., Cook, C.R., King, K.M., Duong, M., ... and Aarons, G.A. (2018). The Impact of Inter-Organizational Alignment (IOA) on Implementation Outcomes: Evaluating Unique and Shared Organizational Influences in Education Sector Mental Health. Implementation Science, 13(1), 24.