Co-Principal Investigators: Aloe, Ariel M.; Domitrovich, Celene E.; Davidson, Laura

Purpose: The purpose of this project is to create linked forms of the student self-report Social-Emotional Competency Assessment (SECA) across five school districts (located in the east, south, and west), grade levels (3rd through 12th), and subpopulations (gender, race-ethnicity, SES). Doing so tailors forms to align with local practice/standards (including by grade level), to be culturally sustaining for student subpopulations, and to be of manageable length while precise. The proposed strategy builds on a successful IES Researcher-Practitioner Partnership (R305H130012) which created the Washoe County School District (WCSD) versions of the SECA. Beyond downloads of the freely available items, in just a few years, these versions have been formally requested by 28 school districts, 17 evaluators, 4 state education agencies, 9 nonprofits, and 9 international organizations. The current project aims to support such uses by making it easier for districts to tailor forms to reflect local practices and standards, to equitably assess students from diverse subpopulations, and to efficiently achieve needed precision of estimation.

Project Activities: In year one, researchers, practitioners, and local co-creation networks will collaborate to adapt items. The focal assessed construct encompasses students' intrapersonal and interpersonal knowledge and skills to support students' academic outcomes (standardized test scores, GPAs, graduation). A core group of common items plus locally adapted unique items will be placed into forms specific to districts, grade levels, and subpopulations, using prior publications and new analyses of existing SECA datasets, review of district social-emotional learning (SEL) practices, and completion of data forums, focus groups, individual reflections, and field testing. In the first year, researchers will also create a secure open-license online platform to administer the forms and to produce linked scores. Districts will test the online platform in year one and conduct student surveys with it in years two and three. To inform iterative improvement between survey administrations, researchers will create summary sheets of item and form statistics (e.g., item discriminations and difficulties; model-level and item-level measurement invariance), refine algorithms to produce linked scores from the online platform, and write code for dynamic reporting of average SECA scores and associations of SECA scores with students' academic outcomes across student subgroups. Throughout, local co-creation networks and a national project-wide advisory network will inform iterative item improvement. Across project years, aggregate results (protecting confidentiality) will be shared with education researchers and practitioners, including through the online platform's public landing pages. The online platform will be strategically designed for sustained use by districts after the project ends.

Products: This project creates two focal products: (a) the tailored SECA forms for each of the five districts, grade levels, and student subpopulations, and (b) a secure open-license online platform that these and other districts can use for continued item improvement. The online platform will include an item bank, evidence base, step-by-step guidebooks, and algorithms. The item bank not only includes items used in the project but is also designed for expansion so that districts can share continuously improved items. The team will collaborate on products for practitioner and scholarly audiences, such as blogs, newsletters, practice-oriented journals, and peer-reviewed research journals, and may also present at conferences attended by practitioners and researchers.

Structured Abstract

Setting: Data collection will take place in five school districts in and around Washoe County, NV, Arlington, TX, and Chatham, Harwich, Martha's Vineyard, and Taunton, MA.

Sample: This project will include students from five districts that enroll from approximately 2,000 to 67,000 students. These students attend 2 to 54 elementary schools, 1 to 10 middle and junior high schools, and 1 to 9 high schools. The students reflect diverse backgrounds from 8 to 41% Latinx, 38 to 78% non-Latinx White, 2 to 21% Black, and 6 to 12% other race-ethnicities. From 14 to 34% receive free and reduced lunch, from 4 to 25% are English language learners, and from 9 to 22% have an individualized education plan.

Assessment: The SECA is a student self-report measure of social-emotional competencies originally designed for administration by schools to students in grades 5 to 12 and to align with the five domains of the Collaborative for Academic, Social, and Emotional Learning (CASEL 5; self-awareness, self-management, social awareness, relationship skills, responsible decision making) as expressed by WCSD SEL standards and practices. The project builds on a 138-item bank as well as a 40-item form, expanding the item bank to include grades 3 and 4 and to create forms tailored to additional districts and student subgroups. The project focuses on aspects of the CASEL 5 that proximally relate to students' academic achievement, recasting the conceptual framework around broadband intrapersonal/interpersonal domains of knowledge and skill identified in the prior IES-funded partnership. The framework also incorporates concepts and items from educational psychology (e.g., motivated learning strategies) and from equity models, including from CASEL, to support culturally sustaining measurement across subpopulations.

Research Design and Methods: The project uses both quantitative and qualitative methods. In year one, psychometric analyses of previously collected SECA data will complement synthesis of existing evidence about SECA items. These results will inform year one qualitative data forums, focus groups, individual reflections, and field testing. In years two and three, districts will use the online platform to administer adapted SECA forms, request dynamically generated reports, and engage with local stakeholders to interpret results and iteratively improve items and forms.

Control Condition: Due to the nature of the study, there is no control condition.

Key Measures: Student academic outcomes include GPAs, standardized test scores, academic risk status, and graduation status. Student characteristics include grade level, gender, race-ethnicity, English language learner, free and reduced lunch, and individualized education plan status.

Data Analytic Strategy: To analyze transcripts from focus groups, the project uses iterative thematic analysis. To analyze the SECA responses, the project uses: (a) factor analytic models to test for dimensionality, (b) factor analytic and item response theory models to test for measurement invariance and to link scores across forms, (c) analysis of covariance models of student subgroup averages, and (d) regression associations with academic outcomes. Results will be presented in ways that are meaningful to practitioners and other stakeholders, and local co-creation and national project-wide advisory networks will continuously inform the interpretation of results and next-step decision making.

Cost Analysis: A cost analysis will estimate the cost to districts of using the final platform.

Related IES Projects: Creating a Monitoring System for School Districts to Promote Academic, Social, and Emotional Learning: A Researcher-Practitioner Partnership (R305H130012)