Project Activities
In the assessment development phase, the research team will develop and refine the pool of DBR items for the multi-item scales. Next, in the assessment evaluation phase, the team will evaluate the reliability, validity, and treatment sensitivity of the DBR scales. In the technology development phase, the team will develop mobile-enabled, web-based tools for rating and summarizing student behaviors using the newly developed DBR scales, and refine the system based on user feedback to maximize feasibility, procedural integrity, and instrument utility.
Structured Abstract
Sample
Overall, approximately 475 teachers in Grades K-3 and their students (including those with and without behavior problems) will participate in this study.
Research design and methods
In year 1, an initial pool of items will be identified based largely on a national teacher survey of common referral concerns. This pool of items will be expanded, refined, and verified based on the following four methods: (1) a review of extant measures, (2) analysis of large datasets of daily report card target behaviors, (3) focus groups with teachers, parents, and school administrators (Consumer Advisory Panel), and (4) feedback from a panel of experts (Scientific Advisory Panel). In years 2â4, Generalizability (G) and Decision (D) studies will be used to generate information to further refine the multi-item scales (e.g. internal consistency reliability, inter-rater reliability, temporal stability). In addition, a series of multitrait-multimethod studies will investigate the convergent and discriminant validity and temporal stability of the scales. Treatment sensitivity of the multi-item scales will be investigated through a series of multiple-baseline single-case design intervention studies. A series of parallel G and D studies will compare the dependability of the single-item compared to multi-item scales. In years 3 and 4, to develop mobile-enabled, web-based tools, the team will use focus groups with teachers and a second panel of experts (Consumer Advisory Panel) to solicit feedback regarding acceptability, usability, utility, and perceived barriers to use.
Control condition
Due to the nature of the research design, there is no control condition.
Key measures
Key measures are the DBR scales to be developed and behavior rating scales for determining validity and utility. The behavior rating scales for determining validity include the Integrated Screening and Intervention System Teacher Rating Form, Daily Report Card, the Social Skills Improvement System (SSIS), the Teacher Report Form (TRF Externalizing and Internalizing Scales), and the Academic Competence Evaluation Scale (ACES).
Data analytic strategy
Multiple statistical and measurement techniques will be used to develop and assess the DBR, including exploratory factor analysis, Generalizability studies, and Decision studies. Visual analysis and non-overlap effect size techniques will be used to evaluate the multiple-baseline single-case design studies. Multitrait-multimethod analyses will be conducted using an analysis of variance approach. Focus group data will be transcribed and content analysis used to identify emergent themes.
People and institutions involved
IES program contact(s)
Products and publications
Products: The products of this project will include validated web-based DBR scales and tools for progress monitoring social behavior for students in Grades K-3. They will also include peer-reviewed publications and presentations.
Supplemental information
Co-Principal Investigator: Amy Briesch (Northeastern University) and Julie Sarno Owens (Ohio University)
Setting: The research will take place in elementary schools in Massachusetts and Ohio.
Assessment: Direct Behavior Rating (DBR) scales combine the strength of a rating scale (e.g., efficient data recording) and the benefits of systematic direct observation (e.g., data recording that occurs at the time and place of the behavior). In brief, a DBR is completed by a rater (e.g., teacher) who quantifies perception of a well-defined behavior of a student through rating that behavior in close proximity to the time and place of behavior observation. The DBRs developed in this grant will include multi-item DBR scales measuring academic enablers (study skills, interpersonal skills, motivation, and academic engagement) and externalizing behaviors (disruptive behavior and oppositional behavior) for use in the formative assessment of student behavior in Grades Kâ3.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.