An evaluation of two contrasting approaches for improving reading achievement in a large urban district.
Skindrud, K., & Gersten, R. (2006). Elementary School Journal, 106(5), 389–407. Retrieved from: https://eric.ed.gov/?id=EJ750504
-
examining434Students, grades2-3
Practice Guide
Review Details
Reviewed: February 2023
- Practice Guide (findings for Open Court Reading and Success for All)
- Quasi-Experimental Design
- Meets WWC standards with reservations because it uses a quasi-experimental design in which the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Stanford Achievement Test, 9th Edition (SAT-9), Reading Subtest |
Open Court Reading and Success for All vs. Success for All® |
2 Years |
Cohort 1;
|
43.90 |
38.60 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
Study Details
Setting
The study was conducted in 12 schools in a large urban district in Northern California (Sacramento City Unified School District). The Schools implemented either Success for All (4 schools) or Open Court (8 schools) in the fall of 1997.
Study sample
The study does not report demographic characteristics for the analytic sample (only the analytic schools).
Intervention Group
Students in the treatment group received instruction using Open Court (OC). OC uses a systematic approach to teaching alphabetics, print knowledge, and phonemic awareness. In the current study, the district used the 1996 version of the curricula, Open Court Collections for Young Scholars. Two hours of daily whole-class reading instruction was followed by 30 minutes of small group instruction and/or independent work. For all grade 2 and 3 students in Title I elementary schools (including those in the study sample), SCUSD mandated a condensed selection of instructional content to "catch-up" students to OC content that they had not received because of beginning to use the curriculum after their initial primary school entry. Cohort 1 students, beginning to use OC in grade 2, received a condensed curriculum that included content from OC grades 1-3 over the course of the two year study. Students began using typical grade-appropriate OC materials in fall of 1999, after the conclusion of the study. During the second study year, the district used informal reading assessments every 6-8 weeks at the end of each unit.
Comparison Group
Students in the comparison group (for the purposes of this review), received instruction under the rubric of Success for All (SFA), which is a whole-school reading reform approach, requiring buy-in from 80% of teachers at the school. Under SFA, students are put into homogeneous groups, across classrooms and grades, based on reading skills, receiving 90 minutes of reading instruction in these groups daily, outside of their homerooms. SFA also proscribes additional writing instruction outside of these groups; all study schools were rated as doing so during both study years. The authors note that, because OC specifies a greater amount of reading instruction, that SFA schools in the study frequently included additional spelling and grammar, along with writing instruction, outside of the 90-minute reading block. A core reading curriculum is only proscribed in grade K-1, in grades 2-6 the school can choose their own reading curricula. The authors state that the materials and guidelines for instruction (Reading Roots for grade 1, and Reading Wings for grades 2-4), as well as the professional development, tutoring, and the SFA school facilitator and regional consultant oversight procedures, all followed those outlined by the developers of the curriculum.
Support for implementation
The authors state that schools in the evaluation sample achieved implementation typical of developer-supervised sites. For SFA, training and technical assistance are provided by SFA consultants from a regional SFA office. The SFA consultants also assessed implementation fidelity, rating it as a typical level of implementation when compared with national implementation averages. For OC, teachers received 4 days of basic grade-level training in year 1, followed by 4 days of advanced grade-level training in year 2. Each OC school received a reading coach (either FT or PT, depending on school size). Curriculum experts met monthly with reading coaches and administrators to refine instruction and supervision, and solve problems. Reading coaches collected implementation information but were prohibited from sharing the information with the study authors; the district-level reading coordinator indicated that although some schools had implementation problems at the beginning of the study, these were resolved by the second study year.
Success for All® Intervention Report - Beginning Reading
Review Details
Reviewed: March 2017
- Quasi-Experimental Design
- Meets WWC standards with reservations because it uses a quasi-experimental design in which the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Success for All®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
SAT-9: Language |
Success for All® vs. Open Court Reading© |
1 Year |
Grade: 3, lowest 25%, Cohort 2;
|
28.80 |
29.60 |
No |
-- | ||
|
SAT-9: Reading |
Success for All® vs. Open Court Reading© |
2 Years |
Grade: 3, Cohort 1;
|
38.60 |
43.90 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
SAT-9: Language |
Success for All® vs. Open Court Reading© |
1 Year |
Grade: 2, Cohort 1;
|
37.20 |
44.30 |
No |
-- | ||
|
SAT-9: Language |
Success for All® vs. Open Court Reading© |
1 Year |
Grade: 2, lowest 25%, Cohort 1;
|
22.15 |
29.80 |
Yes |
|
||
|
SAT-9: Reading |
Success for All® vs. Open Court Reading© |
1 Year |
Grade: 2/lowest 25%/Cohort 1;
|
25.80 |
33.60 |
No |
-- | ||
|
SAT-9: Language |
Success for All® vs. Open Court Reading© |
2 Years |
Grade: 3, lowest 25%, Cohort 1;
|
29.50 |
38.30 |
No |
-- | ||
|
SAT-9: Reading |
Success for All® vs. Open Court Reading© |
2 Years |
Grade: 3, lowest 25%, cohort 1;
|
25.40 |
34.60 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Other or unknown 100%
Study Details
Setting
The study was conducted in 12 schools in the Sacramento City Unified School District (SCUSD), a large urban district in northern California.
Study sample
Under California’s interpretation of Reading First, all 59 elementary schools in SCUSD were required to implement one of two models of reading instruction, SFA® or Open Court Reading©. In the fall of 1997, four schools implemented SFA®. A matched sample of Open Court Reading© schools were created by rank-ordering SCUSD schools by poverty level (measured by the percentage of students eligible for free or reduced-price meals and percentage of students on Aid to Families with Dependent Children), and selecting two comparison schools for each SFA® school—those ranked just above and just below each SFA® school. The study included two cohorts of students: students in Cohort 1 began using the reading programs in grade 2, while students in Cohort 2 started in grade 3. A total of 936 students in Cohort 1 and Cohort 2 participated in the study. The WWC based its effectiveness rating on findings from 531 students from the two analytic samples that were found to be equivalent at baseline: Cohort 1: 142 students in the SFA® group and 292 students in the comparison group—these students were followed from second to third grade; and Cohort 2: 36 students in the SFA® group and 61 students in the comparison group—these students were followed through third grade. The analytical sample for Cohort 2 includes only low-achieving students (that is, lowest 25% on a standardized test of reading achievement). Results for the full sample of Cohort 2 students are not included in this report because, based on information obtained from the authors, that sample of students was not equivalent on key characteristics at baseline.
Intervention Group
Students in the intervention group received reading instruction through SFA®. Students were put into homogeneous groups, across classrooms and grades, based on reading skills. They received 90 minutes of reading instruction daily, outside of their homerooms. SFA® also prescribes additional writing instruction outside of these groups. The SFA® training consultants monitored implementation fidelity and observed additional writing instruction in all study schools during both study years. The authors noted that teachers in SFA® schools frequently included additional spelling and grammar, along with writing instruction, outside of the 90-minute reading block. SFA® prescribes a core reading curriculum only in grades K–1; in grades 2–6, the schools can choose their own reading curricula. The authors state that the materials and guidelines for instruction (Reading Roots for grade 1 and Reading Wings for grades 2–4), as well as the professional development, tutoring, and the SFA® school facilitator and regional consultant oversight procedures, all followed those outlined by the developers of the curriculum.
Comparison Group
Students in the comparison group received reading instruction using Open Court Reading©, a systematic approach to teaching alphabetics, print knowledge, and phonemic awareness. For this study, the district used the 1996 version of the curriculum, Open Court Collections for Young Scholars. Two hours of daily whole-class reading instruction was followed by 30 minutes of small-group instruction and/or independent work. All study students received a condensed selection of instructional content to “catch-up” students to Open Court Reading© content that they had not received in prior years (since they began using the curriculum in either second or third grade).
Support for implementation
At SFA® schools, training and technical assistance were provided by SFA® consultants from a regional SFA® office. The SFA® consultants assessed implementation fidelity and rated it as a typical level of implementation when compared with national implementation averages. At Open Court Reading© schools, teachers received 4 days of basic grade-level training in Year 1, followed by 4 days of advanced grade-level training in Year 2. Each Open Court Reading© school received a reading coach (either full-time or part-time, depending on school size). Curriculum experts met monthly with reading coaches and administrators to refine instruction and supervision and to solve problems. Reading coaches collected implementation information but were prohibited from sharing the information with the study authors; the district-level reading coordinator indicated that although some schools had implementation problems at the beginning of the study, these were resolved by the second study year.
Open Court Reading© Intervention Report - Beginning Reading
Review Details
Reviewed: October 2014
- Quasi-Experimental Design
- Meets WWC standards with reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Open Court Reading©.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
SAT-9 |
Open Court Reading© vs. Success for All (SFA) |
Spring 1998 |
Grade 3;
|
43.90 |
38.60 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
40% English language learners -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
Study Details
Setting
The study was conducted in 12 schools in the Sacramento City Unified School District (SCUSD), a large urban district in northern California.
Study sample
Under California’s interpretation of Reading First, all 59 elementary schools in SCUSD were required to implement one of two models of reading instruction, Success for All (SFA)® or Open Court Reading©. In the fall of 1997, four schools implemented SFA®. A matched sample of Open Court Reading© schools were created by rank-ordering SCUSD schools by poverty level (measured by the percent of students eligible for free or reduced-price meals and percent of students on Aid to Families with Dependent Children), and selecting two comparison schools for each SFA® school—those ranked just above and just below each SFA® school. The study included two cohorts of students: students in Cohort 1 began using the reading programs in grade 2, while students in Cohort 2 began in grade 3. A total of 936 students in Cohort 1 and Cohort 2 participated in the study, including students continually enrolled at study schools from fall 1997 to spring 1999 who completed all study tests and did not repeat a grade. The WWC based its effectiveness rating on findings from 434 Cohort 1 students who participated in the study; 292 in the Open Court Reading© group and 142 in the comparison group—these students were followed from second to third grade. Results for the Cohort 2 students are not included in this report because, based on information obtained from the authors, that sample of students was not equivalent on key characteristics at baseline.
Intervention Group
Students in the intervention group received reading instruction using Open Court Reading©, a systematic approach to teaching alphabetics, print knowledge, and phonemic awareness. For this study, the district used the 1996 version of the curricula, Open Court Collections for Young Scholars. Two hours of daily whole-class reading instruction was followed by 30 minutes of small-group instruction and/or independent work. All study students received a condensed selection of instructional content to “catch-up” students to Open Court Reading© content that they had not received in prior years (since they began using the curriculum in either second or third grade).
Comparison Group
Students in the comparison group received reading instruction through SFA®. Students were put into homogeneous groups, across classrooms and grades, based on reading skills. They received 90 minutes of reading instruction daily, outside of their homerooms. SFA® also prescribes additional writing instruction outside of these groups. The SFA® training consultants monitored implementation fidelity and observed additional writing instruction in all study schools during both study years. The authors noted that teachers in SFA® schools frequently included additional spelling and grammar, along with writing instruction, outside of the 90-minute reading block. A core reading curriculum is only prescribed in grades K–1; in grades 2–6, the schools can choose their own reading curricula. The authors state that the materials and guidelines for instruction (Reading Roots for grade 1, and Reading Wings for grades 2–4), as well as the professional development, tutoring, and the SFA® school facilitator and regional consultant oversight procedures, all followed those outlined by the developers of the curriculum.
Outcome descriptions
The outcome measure was the Reading subtest from the SAT-9, administered in both spring 1998 and spring 1999. The authors converted all measures to normal curve equivalent scores. For a more detailed description of this outcome measure, see Appendix B. The Language subtest from the SAT-9 was reported by the authors; however, this outcome measure is not included in this report because it is not an eligible outcome under the Beginning Reading evidence review protocol. The intermediate findings (after 1 year of implementation) for second graders are reported in Appendix D.1.
Support for implementation
At Open Court Reading© schools, teachers received 4 days of basic grade-level training in year 1, followed by 4 days of advanced grade-level training in year 2. Each Open Court Reading© school received a reading coach (either full-time or part-time, depending on school size). Curriculum experts met monthly with reading coaches and administrators to refine instruction and supervision and to solve problems. Reading coaches collected implementation information but were prohibited from sharing the information with the study authors; the district-level reading coordinator indicated that although some schools had implementation problems at the beginning of the study, these were resolved by the second study year. At SFA® schools, training and technical assistance was provided by SFA® consultants from a regional SFA® office. The SFA® consultants assessed implementation fidelity and rated it as a typical level of implementation when compared with national implementation averages.
Open Court Reading© Intervention Report - Adolescent Literacy
Review Details
Reviewed: August 2012
- The study is ineligible for review because it does not use a sample within the age or grade range specified in the protocol.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Open Court Reading©.
Findings
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Study sample characteristics were not reported.An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).