A. Whether an evidence-based intervention will have a positive effect in your schools or classrooms may depend critically on your adhering closely to the details of its implementation.
The importance of adhering to the details of an evidence-based intervention when implementing it in your schools or classrooms is often not fully appreciated. Details of implementation can sometimes make a major difference in the intervention's effects, as the following examples illustrate.
Example. The Tennessee Class-Size Experiment - a large, multi-site randomized controlled trial involving 12,000 students - showed that a state program that significantly reduced class size for public school students in grades K-3 had positive effects on educational outcomes. For example, the average student in the small classes scored higher on the Stanford Achievement Test in reading and math than about 60 percent of the students in the regular-sized classes, and this effect diminished only slightly at the fifth-grade follow-up.23
Based largely on these results, in 1996 the state of California launched a much larger, state-wide class-size reduction effort for students in grades K-3. But to implement this effort, California schools hired 25,000 new K-3 teachers, many with low qualifications. Thus the proportion of fully-credentialed K-3 teachers fell in most California schools, with the largest drop (16 percent) occurring in the schools serving the lowest-income students. By contrast, all the teachers in the Tennessee study were fully qualified. This difference in implementation may account for the fact that, according to preliminary comparison-group data, class-size reduction in California may not be having as large an impact as in Tennessee.24
Example. Three well-designed randomized controlled trials have established the effectiveness of the Nurse-Family Partnership - a nurse visitation program provided to low-income, mostly single women during pregnancy and their children's infancy. One of these studies included a 15-year follow-up, which found that the program reduced the children's arrests, convictions, number of sexual partners, and alcohol use by 50-80 percent.25
Fidelity of implementation appears to be extremely important for this program. Specifically, one of the randomized controlled trials of the program showed that when the home visits are carried out by paraprofessionals rather than nurses - holding all other details the same - the program is only marginally effective. Furthermore, a number of other home visitation programs for low-income families, designed for different purposes and using different protocols, have been shown in randomized controlled trials to be ineffective.26
B. When implementing an evidence-based intervention, it may be important to collect outcome data to check whether its effects in your schools differ greatly from what the evidence predicts.
Collecting outcome data is important because it is always possible that slight differences in implementation or setting between your schools or classrooms and those in the studies could lead to substantially different outcomes. So, for example, if you implement an evidence-based reading program in a particular group of schools or classrooms, you may wish to identify a comparison group of schools or classrooms, roughly matched in reading skills and demographic characteristics, that is not using the program. Tracking reading test scores for the two groups over time, while perhaps not fully meeting the guidelines for "possible" evidence described above, may still give you a sense of whether the program is having effects that are markedly different from what the evidence predicts.
25 David L. Olds et. al., "Long-term Effects of Nurse Home Visitation on Children's Criminal and Antisocial Behavior: 15-Year Follow-up of a Randomized Controlled Trial," Journal of the American Medical Association, vol. 280, no. 14, October 14, 1998, pp. 1238-1244. David L. Olds et. al., "Long-term Effects of Home Visitation on Maternal Life Course and Child Abuse and Neglect: 15-Year Follow-up of a Randomized Trial," Journal of the American Medical Association, vol. 278, no. 8, pp. 637-643. David L. Olds et. al, "Home Visiting By Paraprofessionals and By Nurses: A Randomized, Controlled Trial," Pediatrics, vol. 110, no. 3, September 2002, pp. 486-496. Harriet Kitzman et. al., "Effect of Prenatal and Infancy Home Visitation by Nurses on Pregnancy Outcomes, Childhood Injuries, and Repeated Childbearing," Journal of the American Medical Association, vol. 278, no. 8, August 27, 1997, pp. 644-652.
26 For example, see Robert G. St. Pierre et. al., op. cit., no. 8; Karen McCurdy, "Can Home Visitation Enhance Maternal Social Support?" American Journal of Community Psychology, vol. 29, no. 1, 2001, pp. 97-112.