II. How to evaluate whether an intervention is backed by "strong" evidence of effectiveness.

This section discusses how to evaluate whether an intervention is backed by "strong" evidence that it will improve educational outcomes in your schools or classrooms. Specifically, it discusses both the quality and quantity of studies needed to establish such evidence.

A. Quality of evidence needed to establish "strong" evidence of effectiveness: Randomized controlled trials that are well-designed and implemented.

As discussed in section I, randomized controlled trials are a critical factor in establishing "strong" evidence of an intervention's effectiveness. Of course, such trials must also be well-designed and implemented in order to constitute strong evidence. Below is an outline of key items to look for when reviewing a randomized controlled trial of an educational intervention, to see whether the trial was well-designed and implemented. It is meant as a discussion of general principles, rather than as an exhaustive list of the features of such trials.

Key items to look for in the study's description of the intervention and the random assignment process

1. The study should clearly describe (i) the intervention, including who administered it, who received it, and what it cost; (ii) how the intervention differed from what the control group received; and (iii) the logic of how the intervention is supposed to affect outcomes.

Example. A randomized controlled trial of a one-on-one tutoring program for beginning readers should discuss such items as:

• who conducted the tutoring (e.g., certified teachers, paraprofessionals, or undergraduate volunteers);
• what training they received in how to tutor;
• what curriculum they used to tutor, and other key features of the tutoring sessions (e.g., daily 20-minute sessions over a period of six-months);
• the age, reading achievement levels, and other relevant characteristics of the tutored students and controls;
• the cost of the tutoring intervention per student;
• the reading instruction received by the students in the control group (e.g., the school's pre-existing reading program); and
• the logic by which tutoring is supposed to improve reading outcomes.

2. Be alert to any indication that the random assignment process may have been compromised.

For example, did any individuals randomly assigned to the control group subsequently cross over to the intervention group? Or did individuals unhappy with their prospective assignment to either the intervention or control group have an opportunity to delay their entry into the study until another opportunity arose for assignment to their preferred group? Such self-selection of individuals into their preferred groups undermines the random assignment process, and may well lead to inaccurate estimates of the intervention's effects.

Ideally, a study should describe the method of random assignment it used (e.g., coin toss or lottery), and what steps were taken to prevent undermining (e.g., asking an objective third party to administer the random assignment process). In reality, few studies - even well-designed trials - do this. But we recommend that you be alert to any indication that the random assignment process was compromised.

3. The study should provide data showing that there were no systematic differences between the intervention and control groups before the intervention.

As discussed above, the random assignment process ensures, to a high degree of confidence, that there are no systematic differences between the characteristics of the intervention and control groups prior to the intervention. However, in rare cases - particularly in smaller trials - random assignment might by chance produce intervention and control groups that differ systematically in various characteristics (e.g., academic achievement levels, socioeconomic status, ethnic mix). Such differences could lead to inaccurate results. Thus, the study should provide data showing that, before the intervention, the intervention and control groups did not differ systematically in the vast majority of measured characteristics (allowing that, by chance, there might have been some minor differences).

Key items to look for in the study's collection of outcome data

4. The study should use outcome measures that are "valid" - - i.e., that accurately measure the true outcomes that the intervention is designed to affect. Specifically:

• To test academic achievement outcomes (e.g., reading/math skills), a study should use tests whose ability to accurately measure true skill levels is well-established (for example, the Woodcock-Johnson Psychoeducational Battery, the Stanford Achievement Test, etc.).
• Wherever possible, a study should use objective, "real-world" measures of the outcomes that the intervention is designed to affect (e.g., for a delinquency prevention program, the students' official suspensions from school).
• If outcomes are measured through interviews or observation, the interviewers/observers preferably should be kept unaware of who is in the intervention and control groups.

  Such "blinding" of the interviewers/observers, where possible, helps protect against the possibility that any bias they may have (e.g., as proponents of the intervention) could influence their outcome measurements. Blinding would be appropriate, for example, in a study of a violence prevention program for elementary school students, where an outcome measure is the incidence of hitting on the playground as detected by an adult observer.

• When study participants are asked to "self-report" outcomes, their reports should, if possible, be corroborated by independent and/orobjective measures.

  For instance, when participants in a substance-abuse or violence prevention program are asked to self-report their drug or tobacco use or criminal behavior, they tend to under-report such undesirable behaviors. In somecases, this may lead to inaccurate study results, depending on whether the intervention and control groups under-report by different amounts.

  Thus, studies that use such self-reported outcomes should, if possible, corroborate them with other measures (e.g., saliva thiocyanate tests for smoking, official arrest data, third-party observations).

5. The percent of study participants that the study has lost track of when collecting outcome data should be small, and should not differ between the intervention and control groups.

A general guideline is that the study should lose track of fewer than 25 percent of the individuals originally randomized - the fewer lost, the better. This is sometimes referred to as the requirement for "low attrition." (Studies that choose to follow only a representative subsample of the randomized individuals should lose track of less than 25 percent of the subsample.)

Furthermore, the percentage of subjects lost track of should be approximately the same for the intervention and the control groups. This is because differential losses between the two groups can create systematic differences between the two groups, and thereby lead to inaccurate estimates of the intervention's effect. This is sometimes referred to as the requirement for "no differential attrition."

6. The study should collect and report outcome data even for those members of the intervention group who don't participate in or complete the intervention.

This is sometimes referred to as the study's use of an "intention-to-treat" approach, the importance of which is best illustrated with an example.

Example. Consider a randomized controlled trial of a school voucher program, in which students from disadvantaged backgrounds are randomly assigned to an intervention group - whose members are offered vouchers to attend private school - or to a control group that does not receive voucher offers. It's likely that some of the students in the intervention group will not accept their voucher offers and will choose instead to remain in their existing schools. Suppose that, as may well be the case, these students as a group are less motivated to succeed than their counterparts who accept the offer. If the trial then drops the students not accepting the offer from the intervention group, leaving the more motivated students, it would be create a systematic difference between the intervention and control groups - namely, motivation level. Thus the study may well over-estimate the voucher program's effect on educational success, erroneously attributing a superior outcome for the intervention group to the vouchers when in fact it was due to the difference in motivation.

Therefore, the study should collect outcome data for all of the individuals randomly assigned to the intervention group, whether they participated in the intervention or not, and should use all such data in estimating the intervention's effect. The study should also report on how many of the individuals assigned to the intervention group actually participated in the intervention.

7. The study should preferably obtain data on long-term outcomes of the intervention, so that you can judge whether the intervention's effects were sustained over time.

This is important because the effect of many interventions diminishes substantially within 2-3 years after the intervention ends. This has been demonstrated in randomized controlled trials in diverse areas such as early reading, school-based substance-abuse prevention, prevention of childhood depression, and welfare-to-work and employment. In most cases, it is the longer-term effect, rather than the immediate effect, that is of greatest practical and policy significance.

Key items to look for in the study's reporting of results

8. If the study claims that the intervention improves one or more outcomes, it should report (i) the size of the effect, and (ii) statistical tests showing the effect is unlikely to be due to chance.

Specifically, the study should report the size of the difference in outcomes between the intervention and control groups. It should also report the results of tests showing the difference is "statistically significant" at conventional levels -- generally the .05 level. Such a finding means that there is only a 1 in 20 probability that the difference could have occurred by chance if the intervention's true effect is zero.

a. In order to obtain such a finding of statistically significant effects, a study usually needs to have a relatively large sample size.

A rough rule of thumb is that a sample size of at least 300 students (150 in the intervention group and 150 in the control group) is needed to obtain a finding of statistical significance for an intervention that is modestly effective. If schools or classrooms, rather than individual students, are randomized, a minimum sample size of 50 to 60 schools or classrooms (25-30 in the intervention group and 25-30 in the control group) is needed to obtain such a finding. (This rule of thumb assumes that the researchers choose a sample of individuals or schools/classrooms that do not differ widely in initial achievement levels.).¹⁵ If an intervention is highly effective, smaller sample sizes than this may be able to generate a finding of statistical significance.

If the study seeks to examine the intervention's effect on particular subgroups within the overall sample (e.g., Hispanic students), larger sample sizes than those above may be needed to generate a finding of statistical significance for the subgroups.

In general, larger sample sizes are better than smaller sample sizes, because they provide greater confidence that any difference in outcomes between the intervention and control groups is due to the intervention rather than chance.

b. If the study randomizes groups (e.g., schools) rather than individuals, the sample size that the study uses in tests for statistical significance should be the number of groups rather than the number of individuals in those groups.

Occasionally, a study will erroneously use the number of individuals as its sample size, and thus generate false findings of statistical significance.

Example. If a study randomly assigns two schools to an intervention group and two schools to a control group, the sample size that the study should use in tests for statistical significance is just four, regardless of how many hundreds of students are in the schools. (And it is very unlikely that such a small study could obtain a finding of statistical significance.)

c. The study should preferably report the size of the intervention's effects in easily understandable, real-world terms (e.g., an improvement in reading skill by two grade levels, a 20 percent reduction in weekly use of illicit drugs, a 20 percent increase in high school graduation rates).

It is important for a study to report the size of the intervention's effects in this way, in addition to whether the effects are statistically significant, so that you (the reader) can judge their educational importance. For example, it is possible that a study with a large sample size could show effects that are statistically significant but so small that they have little practical or policy significance (e.g., a 2 point increase in SAT scores). Unfortunately, some studies report only whether the intervention's effects are statistically significant, and not their magnitude.

Some studies describe the size of the intervention's effects in "standardized effect sizes." ¹⁶ A full discussion of this concept is beyond the scope of this Guide. We merely comment that standardized effect sizes may not accurately convey the educational importance of an intervention, and, when used, should preferably be translated into understandable, real-world terms like those above.

9. A study's claim that the intervention's effect on a subgroup (e.g., Hispanic students) is different than its effect on the overall population in the study should be treated with caution.

Specifically, we recommend that you look for corroborating evidence of such subgroup effects in other studies before accepting them as valid.

This is because a study will sometimes show different effects for different subgroups just by chance, particularly when the researchers examine a large number of subgroups and/or the subgroups contain a small number of individuals. For example, even if an intervention's true effect is the same on all subgroups, we would expect a study's analysis of 20 subgroups to "demonstrate" a different effect on one of those subgroups just by chance (at conventional levels of statistical significance). Thus, studies that engage in a post-hoc search for different subgroup effects (as some do) will sometimes turn up spurious effects rather than legitimate ones.

Example. In a large randomized controlled trial of aspirin for the emergency treatment of heart attacks, aspirin was found to be highly effective, resulting in a 23 percent reduction in vascular deaths at the one-month follow-up. To illustrate the unreliability of subgroup analyses, these overall results were subdivided by the patients' astrological birth signs into 12 subgroups. Aspirin's effects were similar in most subgroups to those for the whole population. However, for two of the subgroups, Libra and Gemini, aspirin appeared to have no effect in reducing mortality. Clearly it would be wrong to conclude from this analysis that heart attack patients born under the astrological signs of Libra and Gemini do not benefit from aspirin. ¹⁷

10. The study should report the intervention's effects on all the outcomes that the study measured, not just those for which there is a positive effect.

This is because if a study measures a large number of outcomes, it may, by chance alone, find positive (and statistically-significant) effects on one or a few of those outcomes. Thus, the study should report the intervention's effects on all measured outcomes so that you can judge whether the positive effects are the exception or the pattern.

B. Quantity of evidence needed to establish "strong" evidence of effectiveness.

1. For reasons set out below,II. How to evaluate whether an intervention is backed by "strong" evidence of effectiveness. we believe "strong" evidence of effectiveness requires:

(i) that the intervention be demonstrated effective, through well-designed randomized controlled trials, in more than one site of implementation, and

(ii) that these sites be typical school or community settings, such as public school classrooms taught by regular teachers. Typical settings would not include, for example, specialized classrooms set up and taught by researchers for purposes of the study.

Such a demonstration of effectiveness may require more than one randomized controlled trial of the intervention, or one large trial with more than one implementation site.

2. In addition, the trials should demonstrate the intervention's effectiveness in school settings similar to yours, before you can be confident it will work in your schools and classrooms.

For example, if you are considering implementing an intervention in a large inner-city public school serving primarily minority students, you should look for randomized controlled trials demonstrating the intervention's effectiveness in similar settings. Randomized controlled trials demonstrating its effectiveness in a white, suburban population do not constitute strong evidence that it will work in your school.

3. Main reasons why a demonstration of effectiveness in more than one site is needed:

• A single finding of effectiveness can sometimes occur by chance alone. For example, even if all educational interventions tested in randomized controlled trials were ineffective, we would expect 1 in 20 of those trials to "demonstrate" effectiveness by chance alone at conventional levels of statistical significance. Requiring that an intervention be shown effective in two trials (or in two sites of one large trial) reduces the likelihood of such a false-positive result to 1 in 400.
• The results of a trial in any one site may be dependent on site-specific factors and thus may not be generalizable to other sites. It is possible, for instance, that an intervention may be highly effective in a school with an unusually talented individual managing the details of implementation, but would not be effective in another school with other individuals managing the detailed implementation.

  Example. Two multi-site randomized controlled trials of the Quantum Opportunity Program - a community-based program for disadvantaged high school students providing academic assistance, college and career planning, community service and work experiences, and other services - have found that the program's effects vary greatly among the various program sites. A few sites - including the original program site (Philadelphia) - produced sizeable effects on participants' academic and/or career outcomes, whereas many sites had little or no effect on the same outcomes.¹⁸ Thus, the program's effects appear to be highly dependent on site-specific factors, and it is not clear that its success can be widely replicated.

4. Pharmaceutical medicine provides an important precedent for the concept that "strong" evidence requires a showing of effectiveness in more than one instance.

Specifically, the Food and Drug Administration (FDA) usually requires that a new pharmaceutical drug or medical device be shown effective in more than one randomized controlled trial before the FDA will grant it a license to be marketed. The FDA's reasons for this policy are similar to those discussed above.¹⁹

¹⁵ Howard S. Bloom, "Sample Design for an Evaluation of the Reading First Program," an MDRC paper prepared for the U.S. Department of Education, March14, 2003. Robert E. Slavin, "Practical Research Designs for Randomized Evaluations of Large-Scale Educational Interventions: Seven Desiderata," paper presented at the annual meeting of the American Educational Research Association, Chicago, April, 2003.

¹⁶ The "standardized effect size" is calculated as the difference in the mean outcome between the treatment and control groups, divided by the pooled standard deviation.

¹⁷ Rory Collins and Stephen MacMahon, "Reliable Assessment of the Effects of Treatment on Mortality and Major Morbidity, I: Clinical Trials," The Lancet, vol. 357, February 3, 2001, p. 375.

¹⁸ Robinson G. Hollister, "The Growth of After-School Programs and Their Impact," paper commissioned by the Brookings Institution's Roundtable on Children, February 2003, at http://www.brook.edu/dybdocroot/views/papers/sawhill/20030225.pdf. Myles Maxfield, Allen Schirm, and Nuria Rodriguez-Planas, "The Quantum Opportunity Program Demonstration: Implementation and Short-Term Impacts," Mathematica Policy Research (no. 8279-093), August 2003.

¹⁹ Guidance for Industry: Providing Clinical Evidence of Effectiveness for Human Drugs and Biological Products, Food and Drug Administration, May 1998, pp. 2-5

Top