Project Activities
The project planned to hold three summer training institutes with one held in 2017, 2018, and 2019. Each Institute was to include 9 days of instruction. Instruction included 22 units of in-class instruction, each of which last 2 to 2½ hours, and 13 units of group work, each of which is scheduled to be about 2 hours but may go longer. In-class sessions addressed a particular topic on the design, planning, analysis, and interpretation of RCTs in education using lecture, discussion, in class exercises, and additional readings. For the group work, about six training participants worked together as a research team designing a RCT that could be submitted as part of a grant proposal to IES under the Efficacy project type. The groups received support from both the Institute instructional staff, and post-doctoral fellows and advanced doctoral students in statistics and the quantitative social sciences with experience with randomized trials (these pre and postdocs also gained experience as methodological consultants). Each group presented their proposal and received a critique from the instructors and other participants.
Key outcomes
The project team completed the 2018 and 2019 Institutes and trained 58 participants out of the 104 applicants. The third Institute could not be held in 2020 nor 2021 because of the disruptions caused by COVID-19. The project team estimated the cost per participant at about $8,926. In place of the third Institute, the project team prepared a working paper on how randomized controlled trials could be adjusted for the disruptions caused by COVID-19.
The project team advertised the Institutes through 1) an IES Newsflash, 2) emails to SREE members, and 3) emails to all past participants urging them to inform their colleagues about the institute. Applicants submitted a curriculum vita and a personal statement indicating how the Institute would advance their career. Participants were required to have three primary qualifications: (1) Education background, (2) Statistical background (including graduate courses covering analysis of variance and regression) and computing background (including ability to use SPSS, SAS, STATA, or R to manipulate variables and run simple analyses), and (3) Likelihood of immediate use of training. The program co-directors evaluated the candidates' vita and personal statement on each of the three criteria using a three-point scale and candidates were selected based on the scores from two raters. If any co-director had a conflict of interest in evaluating any candidate, a senior member of the instructional staff carried out the rating instead.
The project team requested feedback from the 58 Institute participants.
- A 10-item survey (Survey of the Use of Training from the Summer RCT Institute) was provided to participants at the end of the Institute and again months to one year after the end of the Institute.
- Thirty-eight participants responded to the follow up survey (61% from academia, 32% from a research firm or non-profit and the rest from a SEA/LEA or federal office).
- 90% reported using what they learned in the Institute in their research
- 57% reporting using what they learned at the Institute in their teaching
- 95% reported using what they learned at the Institute in discussions with their colleagues and 79% reported using what they learned in evaluating manuscripts
- 63% reported using what they learned in the preparation of grant proposals
- 78% reported using what they learned while working on actual trials, 37% on presentations or papers about trials, and 95% plan to work on randomized trials in the future
- 95% reported they would recommend the Institute to their colleagues
People and institutions involved
IES program contact(s)
Products and publications
Products: The project team intends to release a set of instructional units to be available online at the Northwestern University IPR STEPP Center website (https://steppcenter.northwestern.edu/index.html) for a course roughly equivalent to the RCT Institute. The units will include videos of lectures from the Institutes; study guides that include specific learning objectives for the unit, pointers to specific portions of the videos, pointers to specific parts of the reference materials (journal articles, reports, or book chapters), exercises for the user, and answers to those exercises; and references.
ERIC Citations: Find available citations in ERIC for this award here.
Publications:
Chan, W. and Hedges, L. (2022) Pooling Interactions into Error Terms in Multisite Experiments. Journal of Educational and Behavioral Statistics, 47(6): 639-665
Hedges, L. and Tipton, E. (2020) Addressing the Challenges to Educational Research Posed by COVID-19. Northwestern Institute for Policy Studies working paper WP029-47. (available at https://www.ipr.northwestern.edu/our-work/working-papers/2020/wp-20-47.html and at https://steppcenter.northwestern.edu/research/rcts-during-covid-19.html)
Project website:
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
