The research activities include: a) validation of DBR for use in screening; b) validation of DBR for progress monitoring; and c) examination of foundational psychometric properties. With regard to screening assessment, the researchers will establish appropriate cut-points for current and predictive student risk in both elementary and middle school student samples located in districts across three states. Concurrently, the team will examine traditional psychometric indicators (e.g., construct validity, criterion-related validity, reliability) along with other forms of information relevant to score interpretation and use (e.g., social and educational consequences, relevance, and utility).
Setting
The project will take place in elementary and middle schools in Connecticut, North Carolina, and New York.
Sample
Approximately 2,000 elementary and middle school students and their teachers across 120 classrooms will participate.
The study will be conducted in four phases: validation for use in screening, validation for use in progress monitoring, examination of foundational psychometric properties, and unified validation in a problem-solving model (e.g., utility). Data from DBR and criterion measures will be collected from teachers three times per year to allow for descriptive analyses of DBR data trends over time. A stratified random subsample of students in the screening group will be tracked over four years (assessed two times per year) to determine whether DBR measures are predictive of long-term behavioral challenges. Single-case methodology will be used to investigate the impact of DBR use on teacher problem-solving behavior and track DBR sensitivity to change. In addition, multi-trait, multi-method analyses will be used to determine how much of the variation in obtained DBR scores is attributable to the traits (behaviors) relative to the method. A portion of teachers in this phase will also participate in a verbal protocol analysis study where they will "think aloud" as they watch videotapes of students and rate student behavior.
Control condition
There is no control condition.
Key measures
Key measures include: DBR scales; behavior rating scales including the Social Skills Improvement System (SSIS) and Behavior Assessment System for Children-2, Behavioral and Emotional Screening System (BESS) and Teacher Report Scale (BASC-2 TRS); systematic direct observation; student record review; Curriculum-Based Measurement–Reading (CBM–R); the research team developed Problem-Solving-Behavior (P-S-B) Checklist; and the Usage Rating Profile (a measure of perceived utility).
Data analytic strategy
A variety of analytic methods will be used to assess the DBR. Receiver operating characteristic (ROC) analysis will be used to set DBR cut scores. Consistency and consensus estimates and hierarchical linear modeling (HLM) will be used to examine trends over time. Multi-trait, multi-method analyses will be conducted using confirmatory factor analysis. For progress monitoring, descriptive data will be summarized and formative data will be examined using visual analysis strategies. Qualitative data collected through the "think aloud" study will be reviewed and coded for key topics of interest to the study (e.g., teacher ability to differentiate between engagement and disruptive behavior).
Products: Products of this project include a validated DBR tool for screening and progress monitoring, and published reports on the psychometric indicators and information relevant to score interpretation and use.
Journal article, monograph, or newsletter
Johnson, A.H., Miller, F., Chafouleas, S.M., and Kooken, J. (2011). Direct Behavior Rating: Updates and Current Research Directions. The Connecticut School Psychologist, 17(3): 5-6.
Johnson, A.H., Miller, F.G., Chafouleas, S.M., Riley-Tillman, T.C., Fabiano, G.A., and Welsh, M.E. (2016). Evaluating the Technical Adequacy of DBR-SIS in Tri-Annual Behavioral Screening: A Multisite Investigation. Journal of School Psychology, 54: 39-57. doi:10.1016/j.jsp.2015.10.001
Kooken, J., Welsh, M.E., McCoach, D.B., Miller, F.G., Chafouleas, S.M., Riley-Tillman, T.C., and Fabiano, G. (in press). Test Order in Teacher-Rated Behavior Assessments: Is Counterbalancing Necessary? Psychological Assessment. doi:10.1037/pas0000314
Miller, F. G., Crovello, N., and Chafouleas, S. M. (in press). (2017). Progress Monitoring the Effects of Daily Report Cards Across Elementary and Secondary Settings Using Direct Behavior Rating- Single Item Scales . Assessment for Effective Intervention. doi:10.1177/1534508417691019
Miller, F.G., Chafouleas, S.M., Riley-Tillman, T.C., and Fabiano, G.A. (2014). Teacher Perceptions of the Usability of School-Based Behavior Assessments. Behavioral Disorders, 39(4): 201-210.
Miller, F.G., Cohen, D., Chafouleas, S.M., Riley-Tillman, T.C., Welsh, M.E., and Fabiano, G.A. (2015). A Comparison of Measures to Screen for Social, Emotional, and Behavioral Risk. School Psychology Quarterly, 30(2): 184-196. doi:10.1037/spq0000085
Miller, F.G., Patwa, S.S., and Chafouleas, S.M. (2014). Using Direct Behavior Rating - Single Item Scales to Assess Student Behavior Within Multi-Tiered Systems of Support. Journal of Special Education Leadership, 27(2): 76-85. Full text
Miller, F.G., Riley-Tillman, T.C., Chafouleas, S.M., and Schardt, A.A. (2016). Direct Behavior Rating Instrumentation: Evaluating the Impact of Scale Formats. Assessment for Effective Intervention. doi:10.1177/1534508416658007
Riley-Tillman, T. C., Sims, W. & Cohen, D. (in press). Daily Report Cards and Direct Behavior. Ratings as an Intervention/Assessment Package. Assessment for Effective Intervention.
