What works in afterschool programs: The impact of a reading intervention on student achievement in the Brockton Public Schools (phase II).
Fitzgerald, R., & Hartry, A. (2008). Berkeley, CA: MPR Associates, Inc. and the National Partnership for Quality Afterschool Learning at SEDL.
-
examining483Students, grades4-6
READ 180® Intervention Report - Adolescent Literacy
Review Details
Reviewed: November 2016
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for READ 180®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Stanford 10 Spelling subtest |
READ 180® vs. Business as usual |
1 Year |
Cohort 1;
|
630.82 |
625.88 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Stanford 10 Spelling subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 4;
|
619.81 |
613.45 |
No |
-- | ||
|
Stanford 10 Spelling subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 5;
|
637.20 |
634.14 |
No |
-- | ||
|
Stanford 10 Spelling subtest |
READ 180® vs. Business as usual |
2 Years |
Cohorts 1 & 2;
|
N/A |
N/A |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Stanford Achievement Test, Tenth Edition (Stanford 10) Reading Comprehension subtest |
READ 180® vs. Business as usual |
1 Year |
Cohort 1;
|
635.41 |
625.75 |
Yes |
|
|
|
|
Stanford 10 Vocabulary |
READ 180® vs. Business as usual |
1 Year |
Cohort 1;
|
639.11 |
630.68 |
No |
-- | ||
|
Stanford 10 Vocabulary |
READ 180® vs. Business as usual |
1 Year |
Cohort 2;
|
N/A |
N/A |
No |
-- | ||
|
Stanford Achievement Test, Tenth Edition (Stanford 10) Reading Comprehension subtest |
READ 180® vs. Business as usual |
1 Year |
Cohort 2;
|
N/A |
N/A |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Stanford Achievement Test, Tenth Edition (Stanford 10) Reading Comprehension subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 5;
|
644.34 |
627.20 |
Yes |
|
||
|
Stanford 10 Vocabulary |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 5;
|
651.04 |
634.91 |
Yes |
|
||
|
Stanford Achievement Test, Tenth Edition (Stanford 10) Reading Comprehension subtest |
READ 180® vs. Business as usual |
2 Years |
Cohorts 1 & 2;
|
N/A |
N/A |
No |
-- | ||
|
Stanford 10 Vocabulary |
READ 180® vs. Business as usual |
2 Years |
Cohorts 1 & 2;
|
N/A |
N/A |
No |
-- | ||
|
Stanford Achievement Test, Tenth Edition (Stanford 10) Reading Comprehension subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 4;
|
622.32 |
623.20 |
No |
-- | ||
|
Stanford 10 Vocabulary |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 4;
|
620.15 |
621.24 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Stanford 10 Total Reading Score |
READ 180® vs. Business as usual |
1 Year |
Cohort 2;
|
N/A |
N/A |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Stanford 10 Total Reading Score |
READ 180® vs. Business as usual |
2 Years |
Cohorts 1 & 2;
|
N/A |
N/A |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Dynamic Indicators of Basic Early Literacy Skills (DIBELS): Oral Reading Fluency subtest |
READ 180® vs. Business as usual |
1 Year |
Cohort 1;
|
106.27 |
103.73 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Dynamic Indicators of Basic Early Literacy Skills (DIBELS): Oral Reading Fluency subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 4;
|
105.21 |
101.13 |
No |
-- | ||
|
Dynamic Indicators of Basic Early Literacy Skills (DIBELS): Oral Reading Fluency subtest |
READ 180® vs. Business as usual |
23 Weeks |
Cohort 1, grade 5;
|
110.76 |
108.67 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 54%
Male: 46% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Massachusetts
-
Race Black 53% Other or unknown 5% White 31% -
Ethnicity Hispanic 11% Not Hispanic or Latino 89%
Study Details
Setting
The study included students in grades 4, 5, and 6 in four elementary schools in Brockton, Massachusetts.
Study sample
Brockton Public Schools identified four of its 16 elementary schools to participate in the study. Schools were chosen because they had a large number of students reading below grade level, they had adequate facilities, and afterschool programs already existed in the schools. Students who enrolled in the afterschool program at each of these four schools were randomly assigned within school- and grade-blocks to be in either a READ 180® classroom or a comparison classroom. The study took place over 2 school years (2006–07 and 2007–08). In each study year, 24 afterschool classrooms participated: 12 READ 180® classrooms and 12 comparison group classrooms. The sizes of these afterschool classes ranged from eight to 17 students. A total of 36 teachers participated in the study in Year 1, and 30 teachers participated in Year 2. There are three analytic samples of interest in this study: (1) Cohort 1, first year sample (297 students); (2) Cohort 2, first year sample (187 students); and (3) Cohorts 1 and 2, combined second year sample (294 students). Findings from the Cohort 1, first year sample are presented in Kim et al. (2011). Although findings from this sample were also presented in Fitzgerald and Hartry (2008), sample sizes and findings differed slightly between the two sources, and the WWC opted to use the most recent reference to use in this report. Findings from the Cohort 2, first year sample and the Cohorts 1 and 2, second year sample are presented in Fitzgerald and Hartry (2008). As reported in Kim et al. (2011), there were 155 students in the READ 180® group at baseline in the fall of 2007 (Cohort 1). Of these students, 67% were eligible for free or reduced-price lunch; 52% were female; and the average age of students was 10.6 years. At baseline in the fall of 2007, there were 157 students in the comparison group: 71% were eligible for free or reduced price lunch; 56% were female; and the average age of students was 10.6 years. Across both groups in Cohort 1, 28% of students were White, 54% of students were African-American, 12% were Hispanic, and 6% were other races or ethnicities. Across both groups, 36% of students were in grade 4, 44% of students were in grade 5, and 20% of students were in grade 6. Detailed information on the Year 2 sample, which is a combination of the Cohort 1, second year and Cohort 2, first year samples, is provided in Fitzgerald and Hartry (2008). The intervention group in year 2 included 152 students. Of these students, 49% were female; 92% were eligible for free or reduced-price lunch; 19% were in special education; 55% were African American, 32% were White, 7% were Hispanic, 5% were Asian American, and 2% were from other ethnic backgrounds. The comparison group in Year 2 also included 152 students. Of these students, 57% were female; 90% were eligible for free or reduced-price lunch; 18% were in special education; 43% were African American, 38% were White, 10% were Hispanic, 5% were Asian, and 5% were from other ethnic backgrounds.
Intervention Group
The study tested the READ 180® Enterprise intervention. Students in the intervention condition received the READ 180® structured reading program in an afterschool setting. Although the READ 180® program was implemented in an afterschool setting, the key program components were implemented, including the structuring of time to include whole-class instruction, as well as three rotations focused on (1) time using READ 180® software, (2) modeled and independent reading, and (3) small-group direct instruction. Because of the reduced 60-minute session length (relative to the standard READ 180® 90-minute session length), the program developer devised a schedule in which, on any given day, students would rotate through two rather than three of the small-group centers. Student workbooks (“rBooks®”) were also provided in keeping with the program design, and the intended class size of 15 or fewer students was generally maintained. In Year 1, READ 180® students received the program 4 days per week in 60-minute sessions for 23 weeks. In Year 2, three of the four study schools changed the schedule so that the program was implemented for only 2 days per week in 90-minute sessions. The fourth school provided the program 4 days per week and in 90-minute sessions.
Comparison Group
Students in the comparison condition attended Brockton Public Schools’ standard afterschool program, which generally includes 40 minutes of homework, 1 hour of another structured learning activity such as a math or reading, and the remainder of the time in physical exercise or recreation. Instructors could choose from 16 structured learning activities, including math games, reading, art projects, or science activities, or they could develop their own activities. In Year 1, comparison group students attended the regular afterschool program for 4 days each week. In Year 2, three of the four schools switched to a 2-day-per-week schedule for the regular afterschool program, while the fourth school retained the 4-day-per-week schedule.
Support for implementation
Scholastic, Inc., the publisher of READ 180®, provided professional development services to participating teachers. These services consisted of a full day of training prior to the launch of the READ 180® intervention, as well as a half-day of training after approximately 6 weeks of implementation. During the implementation period, a Scholastic trainer periodically met with all of the teachers implementing READ 180® to discuss challenges and identify solutions. All teachers also had access to an online professional development program, called RED, provided by Scholastic.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Kim, J. S., Capotosto, L., Hartry, A., & Fitzgerald, R. (2011). Can a mixed-method literacy intervention improve the reading achievement of low-performing elementary school students in an after-school program? Results from a randomized controlled trial of READ 180 Enterprise. Educational Evaluation and Policy Analysis, 33(2), 183–201.
-
Vaden-Kiernan, Michael; Jones, Debra Hughes; Rudo, Zena; Fitzgerald, Robert; Hartry, Ardice; Chambers, Bette; Smith, Dewi; Muller, Patricia; Moss, Marcey A. (2008). The National Partnership for Quality Afterschool Learning Randomized Controlled Trial Studies of Promising Afterschool Programs: Summary of Findings. Afterschool Research Brief. Issue No. 3. SEDL.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).