The RCT Institute for Established Researchers provided training on the design, execution, interpretation, and reporting of randomized controlled trials (RCTs) in education. The goal of the project was to increase the number of researchers with extensive training in the design and execution of RCTs and the number of researchers who carry out RCTs in the field of education research.

The project planned to hold three summer training institutes with one held in 2017, 2018, and 2019. Each Institute was to include 9 days of instruction. Instruction included 22 units of in-class instruction, each of which last 2 to 2½ hours, and 13 units of group work, each of which is scheduled to be about 2 hours but may go longer. In-class sessions addressed a particular topic on the design, planning, analysis, and interpretation of RCTs in education using lecture, discussion, in class exercises, and additional readings. For the group work, about six training participants worked together as a research team designing a RCT that could be submitted as part of a grant proposal to IES under the Efficacy project type. The groups received support from both the Institute instructional staff, and post-doctoral fellows and advanced doctoral students in statistics and the quantitative social sciences with experience with randomized trials (these pre and postdocs also gained experience as methodological consultants). Each group presented their proposal and received a critique from the instructors and other participants.

The project team advertised the Institutes through 1) an IES Newsflash, 2) emails to SREE members, and 3) emails to all past participants urging them to inform their colleagues about the institute. Applicants submitted a curriculum vita and a personal statement indicating how the Institute would advance their career. Participants were required to have three primary qualifications: (1) Education background, (2) Statistical background (including graduate courses covering analysis of variance and regression) and computing background (including ability to use SPSS, SAS, STATA, or R to manipulate variables and run simple analyses), and (3) Likelihood of immediate use of training. The program co-directors evaluated the candidates' vita and personal statement on each of the three criteria using a three-point scale and candidates were selected based on the scores from two raters. If any co-director had a conflict of interest in evaluating any candidate, a senior member of the instructional staff carried out the rating instead.

The project team completed the 2018 and 2019 Institutes and trained 58 participants out of the 104 applicants. The third Institute could not be held in 2020 nor 2021 because of the disruptions caused by COVID-19. The project team estimated the cost per participant at about $8,926. In place of the third Institute, the project team prepared a working paper on how randomized controlled trials could be adjusted for the disruptions caused by COVID-19.

The project team intends to release a set of instructional units to be available online at the Northwestern University IPR STEPP Center website (https://steppcenter.northwestern.edu/index.html) for a course roughly equivalent to the RCT Institute. The units will include videos of lectures from the Institutes; study guides that include specific learning objectives for the unit, pointers to specific portions of the videos, pointers to specific parts of the reference materials (journal articles, reports, or book chapters), exercises for the user, and answers to those exercises; and references.

Key Outcomes:

The project team requested feedback from the 58 Institute participants.

• A 10-item survey (Survey of the Use of Training from the Summer RCT Institute) was provided to participants at the end of the Institute and again months to one year after the end of the Institute.
• Thirty-eight participants responded to the follow up survey (61% from academia, 32% from a research firm or non-profit and the rest from a SEA/LEA or federal office).
• 90% reported using what they learned in the Institute in their research
• 57% reporting using what they learned at the Institute in their teaching
• 95% reported using what they learned at the Institute in discussions with their colleagues and 79% reported using what they learned in evaluating manuscripts
• 63% reported using what they learned in the preparation of grant proposals
• 78% reported using what they learned while working on actual trials, 37% on presentations or papers about trials, and 95% plan to work on randomized trials in the future
• 95% reported they would recommend the Institute to their colleagues

Related IES Projects: A Summer RCT Institute for Established Researchers (R305B200029); A Summer RCT Training Institute for Established Researchers (R305B130023); RCT Training Institute (R305U110001); Proposal for an RCT Training Institute (R305U080001)

Products and Publications

ERIC Citations: Find available citations in ERIC for this award here.

Project Website: https://steppcenter.northwestern.edu/education-training/professional-development-institutes.html

Select Publications

Hedges, L. and Tipton, E. (2020) Addressing the Challenges to Educational Research Posed by COVID-19. Northwestern Institute for Policy Studies working paper WP029-47. (available at https://www.ipr.northwestern.edu/our-work/working-papers/2020/wp-20-47.html and at https://steppcenter.northwestern.edu/research/rcts-during-covid-19.html)