Summer Research Training Institute: Cluster-Randomized Trials: July 6–17, 2008
| Sunday, July 6 Monday, July 7 Tuesday, July 8 Wednesday, July 9 Thursday, July 10 Friday, July 11 Monday, July 14 Tuesday, July 15 Wednesday, July 16 Thursday, July 17 | |
| Sunday, July 6, 2008 | |
| Sunday afternoon | Arrival at Hotel Orrington, 1710 Orrington Ave, Evanston |
| 6:00 | Meet Patti Ferguson in Lobby; walk to Allen Center |
| 6:30 | Dinner |
| Monday, July 7, 2008 | |
| Section 1: Planning the Evaluation | |
| Session 2 | Specifying the conceptual and operational models; formulating precise questions |
| 8:00 10:00 | This session covers: (1) developing the rationale for the importance of the intervention, including deciding if an intervention is ready for an RCT (randomized controlled trial); (2) determining and justifying the type of study (development, efficacy, or scale-up), including a review of what is already known in the area, relevant pilot data and preliminary studies; (3) specifying the theory of change underlying the intervention, including a conceptual model specifying key cause-effect constructs and their linkages and an operational model of the processes and activities that affect the outcomes; and (4) framing the question precisely so that a trial can be designed to provide an answer that will be useful. |
| 10:00 10:30 | Break |
| Session 3 | Describing and quantifying outcomes |
| 10:30 12:30 | This session covers considerations for identifying relevant variables and selecting tests and measures in education trials, including (1) reliability, validity, sensitivity, and relevance of measures; (2) specifying proximal (mediating) and distal outcomes/variables; (3) alignment and overalignment of measures with the intervention; (4) continuity across ages/grades for follow-up measures; (5) developmental appropriateness of measures; (6) feasibility of use; (7) respondent burden; (8) efficiencyminimizing overlap among measures; (9) attention to possible unexpected as well as expected outcomes; (10) issues associated with correlated measures, creating composite measures; and (11) measurement issues associated with special populations. |
| Instructor: | Mark Lipsey |
| 12:30 1:30 | Lunch |
| Session 4 | Assessment of treatment implementation/assessment of control condition |
| 1:30 3:15 | This session covers strategies used to assess instruction, process, and treatment fidelity, including systematic observation, logs and diaries, questionnaires, and interviews. It includes discussion of (1) the concepts of implementation versus fidelity; (2) the importance of clear specification of the intervention as basis for assessing fidelity/implementation; and (3) measuring the relevant experience in the control group. |
| Instructor: | David Cordray |
| 3:15 3:45 | Break |
| Session 5 | Introducing the Group Activity Assignment |
| 3:45 5:30 | The Group Activity Assignment is designed to provide attendees with an opportunity to gain experience in applying the concepts and strategies that are presented in the various technical sessions to a specific RCT. With the assistance of the instructors, the activity will mimic the process of developing a feasible and technically sound RCT. Each group will formulate an intervention topic, decide on the type of RCT that is most appropriate, articulate the theory of action, and specify intervention and control contrasts. As the sessions progress, the group will have an opportunity to apply the measurement, sampling, design and analysis issues that have been discussed in the technical sessions. Continuous feedback and resources (e.g., extant data, normative conventions, and decision frameworks) will be made available to assist in the development of the overall RCT design. Toward the end of the Training Institute, the final design from each group will be presented to the full group. |
| Instructor: | David Cordray |
| 5:30 6:30 | Dinner |
| Top | |
| Tuesday, July 8, 2008 | |
| Session 6 | Basic experimental design with special considerations for education studies |
| 8:00 10:00 | Sessions 6 to 8 will cover the logic of randomized experiments and their advantages for making causal inferences and a review of the basics of experimental design and analysis of variance focusing on the two most widely used designs: the hierarchical design and the (generalized) randomized blocks design. Issues that arise because of the hierarchical structure of populations in education (students nested in classes nested in schools) are discussed. Additional topics include: (1) issues of which units to randomize (classrooms, schools, or individual students); (2) how to do the randomization; (3) multiple levels of randomization (students to teachers, then teachers to conditions) and what the implications are for design and inference; (4) randomization within schools/classrooms (contamination concerns) versus randomization between schools/classrooms (power issues); (5) nature and role of blocking and covariates (including covariate and blocking considerations at student, classroom, and school levels); (6) crossovers and attrition after randomization; (7) handling multiple cohorts of treatment and control groups; and (8) how hypothesized aptitude x treatment interactions are included in design. |
| Instructor: | Larry Hedges |
| 10:00 10:30 | Break |
| Session 7 | Basic experimental design with special considerations for education studies |
| 10:30 12:30 | Continuation of Session 6. |
| Instructor: | Larry Hedges |
| 12:30 1:30 | Lunch |
| Session 8 | Basic experimental design with special considerations for education studies |
| 1:30 3:15 | Continuation of Session 7. |
| Instructor: | Larry Hedges |
| 3:15 3:30 | Break |
| 3:30 5:30 | Group Projects |
| 5:30 6:30 | Dinner |
| Session 9 | IES Grant Opportunities |
| 7:00 8:00 | Discussion of IES grant opportunities with Lynn Okagaki, US Commissioner of Education Research |
| Top | |
| Wednesday, July 9, 2008 | |
| Session 10 | Sample size and statistical power I |
| 8:00 10:00 | Sessions 10 and 11 will cover: (1) computing statistical power for cluster randomized trials; (2) the role of between-unit variance components and intraclass correlation; (3) planning sample sizes with adequate power; (4) the effect of blocking (matching) and covariates on power; (5) and how choice of analysis influences power. The sessions will also include: (6) discussion of effect size; (7) how to determine and justify the minimum effect size to detect; (8) designing around power considerations associated with the minimum detectable effect size; and (9) cost considerations. |
| Instructor: | Howard Bloom |
| 10:00 10:30 | Break |
| Session 11 | Sample size and statistical power II |
| 10:30 12:30 | Continuation of Session 10. |
| Instructor: | Howard Bloom |
| 12:30 1:30 | Lunch |
| Session 12 | Sampling and external validity |
| 1:30 3:15 | This session includes discussion of: (1) the nature of sampling in experiments; (2) that the concept of blocks in experiments is the same as clusters in sampling; (3) the logic of generalization from blocks (clusters) as fixed effects; (4) the logic of generalization from blocks as random effects; (5) considerations and procedures for sampling clusters (districts, schools, classrooms, instructional small groups) from a population and units within clusters (classrooms within schools, students within classrooms); (6) sample representativeness and sample diversity; (7) oversampling; (8) testing interactions between sample characteristics and treatment response to explore generalizability; and (9) sampling issues in multi-site studies. |
| Instructor: | Howard Bloom |
| 3:15 3:30 | Break |
| 3:30 5:30 | Group Projects |
| 5:30 6:30 | Dinner |
| Top | |
| Thursday, July 10, 2008 | |
| Session 13 | Statistical analysis overview I |
| 8:00 10:00 | This session will provide an introduction to hierarchical linear model analysis and how it is related to analysis of variance methods and regression approaches to analysis of the same designs. The session will include an (1) overview of multilevel models as a means to analyze data with dependencies due to repeated measures; (2) introduce the intra-class correlation (ICC) to describe the degree of dependency or correlation in repeated measures within each level of these models; and (3) describe how different types of multilevel models address these dependencies by modeling the ICC. There will be an (4) overview of two-level models such as individual growth curve models and models that account for nesting of children in classrooms; and (5) three-level models such as individual growth curve models that describe change over time of children nested in classrooms. |
| Instructor: | Larry Hedges |
| 10:00 10:30 | Break |
| Session 14 | Statistical analysis overview II (Computer Lab) |
| 10:30 12:30 | Hands-on practice using HLM and SAS -Demonstration of two-level models using HLM and SAS software packages -Demonstration of three-level models using HLM software |
| Instructor: | Larry Hedges /Spyros Konstantopoulos |
| 12:30 1:30 | Lunch |
| Session 15 | Computer lab session |
| 1:30 3:15 | Hands-on practice using HLM and SAS, continued Consultation time linked to the group project assignment |
| Instructor: | Larry Hedges /Spyros Konstantopoulos |
| 3:15 3:30 | Break |
| 3:30 5:30 | Group Projects |
| 5:30 6:30 | Dinner (Stained Glass restaurant) |
| Top | |
| Friday, July 11, 2008 | |
| Session 16 | Computer lab session |
| 8:00 10:00 | Hands-on practice using HLM and SAS, continued Introduction to Optimal Design for power computations Consultation time linked to the group project assignment |
| Instructor: | Larry Hedges/Jessaca Spybrook |
| 10:00 10:30 | Break |
| Session 17 | Computer lab session |
| 10:30 12:30 | Hands-on practice using HLM and SAS, continued Introduction to Optimal Design for power computations Consultation time linked to the group project assignment |
| Instructor: | Larry Hedges/Jessaca Spybrook |
| 12:30 1:30 | Lunch |
| 1:30 4:00 | Group Projects |
| 4:00 4:30 | Break |
| 4:30 6:00 | Networking Session |
| 6:30 | Dinner at Stained Glass, Evanston |
| Top | |
| Monday, July 14, 2008 | |
| Section 2: Implementing the Evaluation | |
| Session 19 | Dealing with missing data |
| 8:00 10:00 | Theory of analysis of datasets involving missing data and an introduction to software for handling missing data |
| Instructor: | John Graham |
| 10:00 10:30 | Break |
| Session 20 | Dealing with missing data |
| 10:30 12:30 | Continuation of session 20 |
| Instructor: | John Graham |
| 12:301:30 | Lunch |
| Session 21 | Implementation of randomized trials |
| 1:30 2:45 | Discussion of practical problems that arise in actually carrying out randomized trials in school settings |
| Instructor: | David Myers |
| 2:45 3:00 | Break |
| 3:00 5:00 | Group Projects |
| 5:30-6:30 | Dinner |
| Top | |
| Tuesday, July 15, 2008 | |
| Section 3: Data Analysis | |
| Session 22 | Alternatives to randomized trials I |
| 8:00 10:00 | Sessions 22 and 23 will provide an overview of the design alternatives that have the highest internal validity under favorable circumstances and may be considered when a randomized design is not feasible (regression discontinuity, nonrandomized comparison groups with statistical controls, and time series). The discussion of these designs will focus on their general character and logic, the circumstances in which they are applicable, and their relative advantages and disadvantages. |
| Instructor: | Mark Lipsey |
| 10:00 10:30 | Break |
| Session 23 | Alternatives to randomized trials II |
| 10:30 12:30 | Continuation of Session 22 |
| Instructor: | Mark Lipsey |
| 12:30 1:15 | Lunch |
| Session 24 | Group Projects |
| 1:30 3:15 | |
| Instructor: | David Cordray |
| 3:15 3:30 | Break |
| 3:30 5:30 | Group Projects |
| 5:30-6:30 | Dinner |
| Top | |
| Wednesday, July 16, 2008 | |
| Session 25 | Group Project Preparations |
| 8:00 10:00 | |
| 10:00 10:30 | Break |
| Session 26 | Group Project Presentations |
| 10:30 12:30 | One hour for each group presentation, followed by feedback and discussion |
| 12:30 1:30 | Lunch |
| Session 27 | Group Project Presentations |
| 1:30-2:30 | One hour for each group presentation, followed by feedback and discussion |
| 2:30 3:15 | Break |
| Session 28 | Group Project Presentations |
| 3:15 5:15 | One hour for each group presentation, followed by feedback and discussion |
| 5:30-6:30 | Dinner |
| Top | |
| Thursday, July 17, 2008 | |
| Session 29 | Group Project Presentations |
| 8:00 10:00 | One hour for each group presentation, followed by feedback and discussion |
| 10:00 10:30 | Break |
| Session 30 | Group Project Presentations |
| 10:30 12:30 | One hour for each group presentation, followed by feedback and discussion |
| 12:30 1:30 | Lunch |
| Session 31 | Course Evaluation and debrief |
| 1:30-3:30 | Discussion |
| 3:30 3:45 | Break |
| Session 31, continued | Course Evaluation and debrief |
| 3:45 5:00 | Completion of the evaluation forms |
| 6:30 8:00 | Graduation Dinner and Ceremony (Orrington Hotel) |
