Co-Principal Investigator: Greer, Fred; Shi, Dexin

Purpose: The purpose of this study is to evaluate the forms for a social-emotional progress monitoring system, Flex System for Progress Monitoring (Flex PM), as part of conducting Response to Intervention (RtI) in the preschool environment. The Flex PM is part of the existing Behavioral Assessment System for Children, 3rd Edition (BASC-3), a widely used social-emotional system of measurement that includes a screener, observations forms, and other rating scales. RtI is a systematic process of prevention and early intervention using multiple tiers of intervention to target children at different levels of risk. To use RtI efficiently and effectively, there must be a method to systematically collect data on the impact of an intervention on individual children in order to make decisions about maintaining or adjusting their interventions. At the Tier 2 level (classroom or small group intervention), progress monitoring data is needed to determine whether a child is responding appropriately to the intervention or if the child needs individualized intervention (Tier 3). The Flex PM was recently developed to address the need for progress monitoring of social-emotional development in preschoolers; however, there is currently limited data on the its psychometric properties. The current project will investigate the measurement properties of the progress monitoring system, including the assessment forms' sensitivity to change, reliability, and validity for preschoolers who are and are not at risk. In addition, the project will examine stakeholders' perceptions of conducting progress monitoring of social-emotional risk in preschoolers, which has important implications for implementing RtI in a preschool program.

Project Activities: The research team will analyze the existing norming sample dataset used to develop and initially validate the Flex PM to examine measurement properties of the standard assessment forms. In addition, the team will collect prospective data to examine assessment characteristics, measurement properties, and psychometric soundness of the Flex PM for preschoolers determined to be at risk for social-emotional or behavior problems and receiving Tier 2 intervention in the context of an RtI framework. The team will also collect data from school-based stakeholders, including teachers and administrators, to understand the type of assessments that are typically used to monitor social-emotional risk (before implementation of the Flex PM) and their perspectives on progress monitoring and RtI.

Products: The products of this project will include more in-depth knowledge of the characteristics of the Flex PM standard forms within the context of RtI and stakeholders' perspectives and social validity of the assessment. The project will result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders such as practitioners and policymakers.

Structured Abstract

Setting: The research will take place in public pre-Kindergarten (pre-K) programs across school districts in South Carolina that are implementing social-emotional RtI, including rural and suburban areas. In addition, the research team will also utilize previously collected norming data on preschoolers across the United States.

Sample: The first sample includes teacher ratings of 418 preschoolers that were collected in a nationally representative norming dataset used to develop the Flex PM. The second sample will include newly collected data on teachers' ratings of preschool-aged children in public pre-K programs. The research team will screen all preschoolers as part of Tier 1 (approximately 1,000 children) across four school districts to identify children who are at risk for social-emotional or behavioral problems. From this screening, the evaluation sample will include 750 children who are at risk and involved in Tier 2 intervention and 500 children who are not at risk. In addition, the research team will collect data about progress monitoring and Tier 2 practices, as well as feedback on the Flex PM, from approximately 120 stakeholders (including teachers, teaching assistants, school-based psychologists, behavior coaches, and school administrators) who are likely to use the Flex PM forms.

Assessment: The Flex PM – Teacher Rater-Preschool is a set of forms used by preschool teachers to monitor the progress of children participating in Tier 2 intervention for behavioral or social-emotional risk. There are four standard Flex PM – Teacher Rater-Preschool forms that cover three general domains within social-emotional risk. These include Adaptive Skills (Developmental Social Disorders form), Internalizing Problems (Internalizing Problems form), and Externalizing Problems (Attention Deficit/Hyperactivity Disorder form and Disruptive Behaviors form). The Flex PM standard forms are brief assessments, consisting of 8 to 14 items per domain. The forms use items from the BASC-3 TRS-P and the Behavioral and Emotional Screening System. The Flex PM forms are designed to be used for preschoolers participating in either Tier 2 or Tier 3 intervention, though this research will focus on its use as part of Tier 2 programs only.

Research Design and Methods: This project will analyze an existing dataset, the national norming data used to develop the Flex PM, as well as collect prospective data from new teachers and preschoolers. In Year 1, the research team will analyze data from the existing norming sample to examine characteristics of the Flex PM. They will also collect data on the prospective sample. The new data collection will begin with universal screening for all the preschool children in participating schools. The children who are flagged by the screener as at-risk for social-emotional or behavior problems and in need of Tier 2 intervention will be chosen to participate in the study, along with a group of children not at risk who are matched on key demographic factors (such as sex and race/ethnicity) within each school. Before beginning measurement with the Flex PM, data will be collected from school-based stakeholders about what they are currently using as Tier 2 assessments. During the school year, the at-risk children will receive the relevant Tier 2 intervention and progress monitoring measures will be collected for both groups of children. At the end of the school year, student and teacher outcomes will be measured, including stakeholder views of the process. The same process for the prospective sample will occur for different classrooms to evaluate items for at-risk and not at-risk samples of preschoolers and to determine the psychometric soundness of the assessment forms in Years 2 and 3 respectively. In Year 4, social validity data will be collected from stakeholders to examine their perspectives on the Flex PM, progress monitoring, and RtI. Stakeholders (teachers, school psychologists, administrators) will be interviewed to examine their use of the system and usefulness for measuring intervention success.

Control Condition: Although there is no control condition, researchers will examine differences in how the assessment functions for at-risk preschoolers compared to children not at risk and not participating in Tier 2 intervention.

Key Measures: The primary measures for the current study are derived from the overall BASC-3 family of assessments, including the progress monitoring system being evaluated in this project – Flex PM. The Behavioral and Emotional Screening System for preschool teachers, the screening assessment used within the BASC-3 measurement system, will be used to identify which children in Tier 1 may be at risk and in need of Tier 2 intervention and as an outcome measure. To determine the validity of the Flex PM, the research team will use preschool office discipline referrals, a checklist of kindergarten readiness skills, the Pediatric Symptoms Checklist (behavioral screener), BASC-3 Student Observation System (teacher ratings of child behaviors during observation), and the Achenbach System of Empirically Based Assessment( ASEBA; teacher report of problem behaviors). Child and teacher demographic information will be provided by each school's attendance coordinator. Teachers will complete social validity scales to determine the Flex PM's acceptability, feasibility, and usability. Finally, teachers, administrators, and other stakeholders (including behavior coaches and school psychologists) will be interviewed at multiple time points to determine their current Tier 2 measures (prior to implementing the Flex PM) as well as their views towards progress monitoring and use of the Flex PM.

Data Analytic Strategy: For the existing normative sample, item characteristics of the Flex PM standard forms will be examined using confirmatory factor analysis (CFA). Using this same sample, the quality of the forms will be examined to determine consistency (model-based reliability) and validity (variance accounted) associated with the CFA structure. Qualitative data on current practices will be examined using a content analysis to categorize current intervention practices in preschool settings. The prospective sample will be used to examine the capability of the Flex PM to identify differences between at-risk preschoolers and those not at risk. Sensitivity to identify differences will be measured by calculating five indices: odds ratios, standardized mean difference effect size, dependent and independent t-tests, interaction effects from a factorial analysis of variance, and the relative change index. In addition to these traditional tests, invariance testing will incorporate latent scale characteristics across groups and examine differential item functioning. For examining the capability of the Flex PM to identify change with at-risk children before and after Tier 2 intervention, the research team will determine reliability through Cronbach's alpha at each time point and temporal stability by correlating pre- and post-treatment scores. Concurrent validity will be examined by comparing the Flex PM to other forms of progress monitoring, including office discipline referrals, direct observation, and screeners. Criterion validity will be evaluated through the association between Flex PM and the ASEBA. Change in latent factor means scores from baseline to post-intervention will be examined to establish the stability of the form over administrations. Social validity will be examined through correlations between the scores on the questionnaire and the Flex PM scores. Data from open-ended questions and interviews about views on progress monitoring and RtI will be coded and interpreted to examine stakeholders' perceptions of progress monitoring as part of an RtI system.

Related Projects: Validation of the Behavioral and Emotional Screening System for Early Identification for Social-Emotional and Behavioral Problems in Preschoolers (R324A100104); A Psychometric Investigation of Universal Screening for Social Emotional Development in Preschool Using Parent and Teacher Informants (R305A150152)