Design Comparable Incidence Rate Ratio (IRR) Effect Sizes for Count Data in Single-Case Experimental Designs

To investigate the performance of the proposed BC-IRR method, a comprehensive Monte Carlo simulation study will be conducted. Count data based on multiple baseline design will be generated. Different series length, number of cases, baseline trend, treatment effect, between-case variance, and degree of overdispersion will be considered in the data generation. A BC-IRR will be estimated for each generated dataset using the generalized linear mixed model (GLMM) and the generalized estimating equation (GEE) approach, and the statistical inferences will be conducted using a Wald test. The performance of the estimators and their statistical inferences will be evaluated based on bias, mean squared error (MSE), coverage rate of the 95% confidence interval, and empirical Type I error rate. To develop the IRR benchmarks, the project team will adopt the "across-studies comparison" approach in which a distribution of empirical effect sizes is established through meta-analysis. The work will focus on two common outcome domains in SCEDs: reading and behavior. For the behavioral outcome domain, the analysis will further distinguish between behaviors with positive and negative valence. In total, three empirical distributions of IRR effect sizes will be established and each will consist of 200 effect size estimates. For each study, the project team will estimate a within-case IRR and a between-case IRR effect size for the immediate treatment effect adjusting for baseline trend. After obtaining the distributions of the effect size estimates, ranges of effect size values will be obtained according to the quantiles (e.g., bottom 25%, 25–50%, 50–75%, and top 25%) to serve as the empirical benchmarks.

User Testing: The developed R package and web-based calculator will be tried out by the advisory board and graduate students recruited from the Co-PI's SCED courses. The advisory board will inform necessary improvements regarding the functionality, utility, and overall user-friendliness of these tools. Students will actively use the web-based calculator throughout the semester. Their feedback will be gathered through a multi-faceted approach such as surveys, focus groups, or individual interviews. Any issues related to functionality, clarity, or missing information reported by the students will be investigated further to identify the underlying causes. Based on the feedback, the initial versions of the R package and web-based calculator will undergo revisions and updates as needed. Any improvements made to the R package will be seamlessly integrated into the web-based calculator to ensure consistency and coherence.