Co-Principal Investigator: Nahum-Shani, Inbal

The purpose of this project is to develop, implement, evaluate, and continually refine a training program in the development of adaptive interventions (AIs) in education and the use of sequential, multiple assignment, randomized trials (SMARTs) for optimizing adaptive interventions. AIs use a sequence of individually tailored decision rules that specify whether, how, and when to alter the dosage (duration, frequency, or amount), type, or delivery of interventions to students. These interventions seek to address the individual and changing needs of students as they progress through an intervention and are particularly relevant for students with disabilities who may need more intensive supports to demonstrate improvement. Despite the critical role AIs play in various domains of education, experimental research aiming to systematically optimize AIs in education is still in its infancy. SMARTs are experimental designs that enable scientists to address multiple scientific questions for optimizing a high-quality AI, but because SMARTs are relatively new, most educational researchers have not been exposed to them as part of their formal training. While research on AIs and SMART methods has grown significantly in the past few years, there is currently no comprehensive training in AIs and SMARTs. This project attempts to fill this gap through the development, implementation, and evaluation of a training program that consists of (1) freely-available, web resources, (2) an in-person training institute for education researchers, and (3) follow-up mentoring with trainees and their teams.

During this project, the training team will engage in the following activities:

1. Establish a website with training materials and related resources;
2. Develop the content of the training institute to cover a variety of topics, including an introduction to AIs, experimental designs for evaluating AIs, and techniques for analyzing data from SMARTs;
3. Recruit and select institute trainees;
4. Engage trainees in accessing foundational materials on the website;
5. Hold two training institutes during the summers of 2019 and 2020;
6. Providing online training materials and related resources (e.g., video-taped institute sessions);
7. Meet with trainees and their team in one follow-up videoconference; and
8. Evaluate the project website, training materials, training institute, and follow-up mentoring annually.

Through this approach, the training team intends to improve the knowledge and skills needed for the development of effective AIs in education, including knowledge and skills pertaining to the design and conduct of SMARTs. The specific learning objectives include an understanding of (a) the nature and utility of AIs in educational settings, (b) various experimental designs for evaluating and optimizing AIs, (c) how to apply data analytic strategies for common primary and secondary aims in a SMART, (d) the utility and application of SMARTs in pilot studies, and (e) various ways in which SMARTs optimize AIs in education and other fields.

The training team will evaluate specific aspects of the training program (i.e., project website, training institute, mentoring) annually to inform continuous improvement. To evaluate the website, they will track the total number and number of unique visits. To evaluate the effectiveness of the training institute, trainees will complete a questionnaire to rate various aspects of the training and make suggestions for improvement. They will complete a similar questionnaire following the videoconference designed to promote ongoing learning.

Trainee outcomes will also be evaluated to determine the extent to which they apply the knowledge and skills learned as part of the training program. Trainees will complete a survey once per year, for up to 2 years following training completion to determine the extent to which the training program facilitated scholarly activities (e.g., research studies, grant submissions, peer-reviewed publications, presentations, teaching activities) related to AIs. The training team will also track the total number of IES grant submissions that focus on AIs and, whenever possible, this data will be linked to trainees as an objective measure of grant submissions. The training team will also calculate the total training cost per person and the cost-effectiveness.