Intervention: Success for All®

Intervention: Success for All®
August 2009

Appendices

Appendix A1.1 Study characteristics: Borman, Slavin, Cheung, Chamberlain, Madden, & Chambers, 2006 (randomized controlled trial)

Characteristic	Description
Study citation	Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N., & Chambers, B. (2006). Final reading outcomes of the national randomized field trial of Success for All^®. Retrieved from Success for All website: http://www.successforall.net/_images/pdfs/Third_Year_Results_06.doc
Participants	The study piloted the SFA^® program in fall 2001, when three schools were randomly assigned to the SFA^® and three schools to the comparison condition. In fall 2002, 35 new schools were recruited with 18 schools randomly assigned to implement SFA^® in grades K–2, and 17 schools randomly assigned to serve as comparisons.¹ The study presented findings after the intervention students completed one, two, and three years of the program. For the effectiveness ratings, the WWC focused on findings from the longitudinal sample, that is, schools and students who completed three years of the program.² After three years, 18 SFA^® schools with 707 students and 17 comparison schools with 718 students remained in the longitudinal sample.
Setting	The analysis sample included 35 elementary schools across 14 states located in rural and small towns in the South and urban areas of the Midwest.
Intervention	Intervention students received the SFA^® school reform program, including the SFA^® reading curriculum, tutoring for students’ quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. Intervention schools implemented SFA^® in grades K–2 and used their previously planned curriculum in grades 3–5. Some schools took a year to fully implement the program.
Comparison	Comparison schools continued using their regular, previously planned curriculum for grades K–2 (SFA^® was implemented in grades 3–5). Authors conducted observations at all schools and indicated that there was no evidence that when SFA^® was implemented in grades 3–5, students in grades K–2 were also exposed to SFA^®. All sample students were pretested with the Peabody Picture Vocabulary Test (PPVT) prior to SFA^® implementation, and schoolwide PPVT scores show equivalence between the program and comparison schools. Researchers also used information from the Common Core of Data (a database maintained by the National Center for Education Statistics) at several points over the course of the study to demonstrate the equivalence between the program and comparison schools on race/ethnicity, gender, English as a second language, special education, and free and reduced-price lunch. All equivalency tests were assessed at the school level, and no statistically significant differences were found.
Primary outcomes and measurement	Three subtests of the Woodcock Reading Mastery Test were administered during the period reflected in the intervention rating: Word Identification, Word Attack, and Passage Comprehension.³ (See Appendices A2.1–2.3 for more detailed descriptions of outcome measures.)
Staff/teacher training	SFA^® teachers received three days of training during the summer and approximately eight days of on-site follow-up during the first implementation year. Success for All Foundation trainers visited classrooms, met with groups of teachers, looked at data on children’s progress, and provided feedback to school staff on implementation quality and outcomes.
¹ The 17 additional comparison schools implemented SFA^® in grades 3–5, but students in grades K–2—the focus of this study and the WWC review—did not receive the intervention. ² The study provided analysis for two samples, the “longitudinal sample,” which included students who participated in the program for all three years, and the “in-mover sample,” which included the longitudinal sample plus students who transferred into the school. The WWC analysis focuses on the longitudinal sample. The WWC prioritized outcomes that reflected students’ exposure to the intervention for the longest period of time available. Findings reflecting students’ outcomes after shorter periods of implementation can be found in Appendices A4.1–A4.6. ³ One additional subtest of the Woodcock Reading Mastery Test (Letter Identification) was administered during an earlier time period and is presented as an additional finding in Appendix A4.1.

Top

Appendix A1.2 Study characteristics: Dianda & Flaherty, 1995 (quasi-experimental design)

Characteristic	Description
Study citation	Dianda, M., & Flaherty, J. (1995, April). Effects of Success for All^® on the reading achievement of first graders in California bilingual programs. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
Participants	This study involved seven elementary schools in California in which the majority of students were English language learners. Six schools remained by the third year of program implementation. Students were grouped into four language categories and received instruction in English, Spanish, or “Sheltered English.”¹ Only the English-speaking subsample was reviewed.² The report includes three cohorts of students who began participating in the study as kindergarteners in 1992 (99 intervention and 120 comparison students), 1993 (105 intervention and 62 comparison students), or 1994 (94 intervention and 59 comparison students), for a total of 539 participants. For the effectiveness rating, the WWC used data that reflected students’ exposure to the intervention for the longest period of time, which varied for the different cohorts and domains.³ Exact attrition rates are not known for this study; however, the post-attrition intervention and comparison samples were equivalent for the English-speaking subgroup. In the overall sample, the percentage of students eligible for free lunch varied from 70% to 98% in intervention schools, and from 47% to 80% in comparison schools. The percentages of minority students were between 50% and 70% for each study condition.
Setting	The analysis sample included seven elementary schools in California.
Intervention	Intervention students received the typical SFA^® curriculum, including the SFA^® reading curriculum, tutoring for students, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers.
Comparison	Comparison schools continued using their regular, previously planned curriculum. Each comparison school was matched with an SFA^® school in the same district with students who had similar demographics and pretest scores on the Peabody Picture Vocabulary Test measure.
Primary outcomes and measurement	Three subtests of the Woodcock Language Proficiency Battery were administered: Letter-Word Identification, Word Attack, and Passage Comprehension. The authors presented findings from each Woodcock subtest separately and also pooled findings from the Woodcock Letter-Word Identification subtests (see Appendices A2.1–A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	SFA^® teachers received three days of training during the summer and approximately eight days of on-site follow-up during the first implementation year. Success for All Foundation trainers visited classrooms, met with groups of teachers, looked at data on children’s progress, and provided feedback to school staff on implementation quality and outcomes. Specially trained certified teachers or qualified aides worked one-to-one with the students.
¹ English language learners participate in SFA^® in English alongside their English-dominant classmates during a common period in the morning. During the rest of the day, they receive sheltered content instruction or ESL instruction, depending on their level of English proficiency. ² The WWC Beginning Reading topic focuses only on students learning to read in English (see Beginning Reading Protocol). ³ Findings include outcomes after two years of exposure for the alphabetics and comprehension domains, and after two (1994 cohort), three (1993 cohort), and four (1992 cohort) years of exposure for the general reading domain. Findings reflecting students’ outcomes after shorter periods of implementation can be found in Appendix A4.3.

Top

Appendix A1.3 Study characteristics: Madden, Slavin, Karweit, Dolan, & Wasik, 1993 (quasi-experimental design)

Characteristic	Description
Study citation	Madden, N. A., Slavin, R. E., Karweit, N., Dolan, L., & Wasik, B. A. (1993). Success for All^®: Longitudinal effects of a restructuring program for inner-city elementary schools. American Educational Research Journal, 30(1), 123–148.
Participants	The study investigated the effects of two versions of the SFA^® program: full implementation and dropout prevention. Although these versions varied in their implementation of the whole-school reform model, the reading curricula are essentially the same at all schools, with each school receiving the same training, coaching support, and materials. Ratings presented in this report are not disaggregated by the variations in implementation of whole-school reforms. Within each comparison school, one-third of the students were randomly selected for testing purposes. The study focused on cohorts of students who started SFA^® in prekindergarten, kindergarten, and first grade and received the intervention for multiple years. To determine the effectiveness ratings, the WWC focused on the latest term results available. The third-year analytic sample included 671 students within five SFA^® schools and 671 students within five comparison schools spread across three grade levels.¹ African-American students constituted 97% to 100% of students in five intervention schools, with 83% to 98% of students qualified for free lunch. In comparison (Chapter 1) schools, at least 75% of students qualified for free lunch.
Setting	The analysis sample included 10 elementary schools in Baltimore, Maryland.
Intervention	Intervention students in the full implementation version received the typical SFA^® program, including the SFA^® reading curriculum, tutoring for students in grades 1–3, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers. Intervention students in the dropout prevention version had a reduced number of tutors and family support staff. Chapter 1 monies supported the dropout prevention program.
Comparison	The comparison condition included schools that implemented a traditional reading program built around Macmillan Connections basal series. Each comparison school was matched with an intervention school based on the percentage of students getting free or reduced-price lunch and historical achievement level. Students were then individually matched on a standardized test given by the school district. Pretest scores on WRMT Letter-Word Identification, Word Attack, and Durrell Oral Reading subtests served as covariates in analyses.
Primary outcomes and measurement	Two subtests of the Woodcock Language Proficiency Battery were administered: Letter-Word Identification and Word Attack. Additional measures included Durrell Analysis of Reading Difficulty Silent Reading and Oral Reading subtests and the California Achievement Test (CAT) Total Reading (see Appendices A2.1–A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	The teachers and tutors were regular certified teachers. They received detailed teacher’s manuals supplemented by two to three days of in-service at the beginning of the school year. For teachers of grades 1–3 and for reading tutors, these training sessions focused on the implementation of the reading program. Preschool and kindergarten teachers and aids were trained in the use of the thematic units, and other aspects of the preschool and kindergarten models. School facilitators also organized many information sessions to allow teachers to share problems and solutions, suggest changes, and discuss individual children.
¹ Additional findings for a subsample of low-achieving students can be found in Appendices A4.4–A4.6.

Top

Appendix A1.4 Study characteristics: Ross, Alberg, & McNelis, 1997 (quasi-experimental design)

Characteristic	Description
Study citation	Ross, S. M., Alberg, M., & McNelis, M. (1997). Evaluation of elementary school school-wide programs: Clover Park School District. Year 1: 1996–97. Memphis, TN: The University of Memphis, Center for Research in Education Policy.
Participants	The study compared whole-school improvement programs, Success for All^®, Accelerated Schools, and locally developed programs, in 19 schools. Schools were divided into four groups based on the similarity of several school characteristics, including enrollment, percentage of minority students, percentage of students eligible for free/reduced-price lunch, and initial academic performance. The WWC focused on only one group, “cluster 2A,” the third highest with respect to socioeconomic status, which included three SFA^® schools and three Accelerated Schools, with a total number of 252 first-grade students (148 students who attended SFA^® schools, 104 students who attended Accelerated Schools).¹ The study included data that reflected students’ outcomes after one year of program implementation. In the overall sample, the percentage of minority students in three intervention schools was between 47% and 63%. In the three comparison schools, the range was between 42% and 54%. The percentage of students eligible for free/reduced-price lunch varied from 63% to 66% in intervention schools, and from 66% to 71% in comparison schools.
Setting	The analysis sample included six elementary schools in Clover Park, Washington.
Intervention	Intervention students received the typical SFA^® program, including the SFA^® reading curriculum, tutoring for students in grades 1–3, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers.
Comparison	Accelerated Schools is a comprehensive school reform program that is designed to close the achievement gap between at-risk and not at-risk children. The program redesigns and integrates curricular, instructional, and organizational practices so that they provide enrichment for at-risk students.
Primary outcomes and measurement	Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The Durrell Analysis of Reading Difficulty Oral Reading subtest was also used (see Appendices A2.1–A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	No information on training for the specific teachers in this study was provided.
¹An additional group included one SFA^® school and three comparison schools (one school used Accelerated Schools design, and the other two used locally developed programs), but this comparison did not meet WWC evidence screens because the effect of SFA^® could not be separated from the effect of the school.

Top

Appendix A1.5 Study characteristics: Ross & Casey, 1998 (quasi-experimental design)

Characteristic	Description
Study citation	Ross, S. M., & Casey, J. (1998). Longitudinal study of student literacy achievement in different Title I school-wide programs in Fort Wayne Community Schools year 2: First grade results. Memphis, TN: The University of Memphis, Center for Research in Education Policy.
Participants	This study examines the effects of SFA^® in two Title I schools by comparing them with five other Title I schools that were implementing locally developed schoolwide programs.¹ The study did not report on the initial sample size, but 288 students in kindergarten (83 students in the SFA^® schools, 205 students at comparison schools) were included in the final analysis sample, and the post-attrition intervention and comparison samples were equivalent on the achievement pretest measure (PPVT). The study included data that reflected students’ outcomes after two years of program implementation.² School populations ranged between 31% and 50% minority enrollment; between 62% and 81% of students received free or reduced-price lunch.
Setting	The analysis sample included seven Title I elementary schools in Fort Wayne, Indiana.
Intervention	Intervention students received the typical SFA^® curriculum, including the Reading Roots reading curriculum in grade 1 and the Reading Wings reading curriculum in grade 2; one-to-one tutoring for the lowest-achieving students by certified teacher tutors, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers.
Comparison	The five comparison schools implemented locally developed schoolwide programs. The schools were comparable with SFA^® schools on pretest PPVT measures, socioeconomic status, and ethnicity. Four out of the five local school programs incorporated components of other branded programs, including Reading Recovery, Accelerated Reader, Four-Block, and STAR. These curricula place considerable emphasis on reading, use of basal readers, and multifaceted reading activities.
Primary outcomes and measurement	Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The study presented a combined measure of Word Identification and Word Attack. The Durrell Analysis of Reading Difficulty Oral Reading subtest was also used (see Appendices A2.1–A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	No information on training for the specific teachers was provided in this study.
¹The article reported on an additional intervention school that supplemented SFA^® with another branded intervention (Reading Recovery), but results from this portion of the study do not meet WWC evidence standards because the effect of SFA^® cannot be separated from the effect of Reading Recovery. ²Additional findings for a subsample of low-achieving students (that is, lowest 25% with respect to reading achievement) are reported in Appendices A4.1–A4.6.

Top

Appendix A1.6 Study characteristics: Ross, McNelis, Lewis, & Loomis, 1998 (quasi-experimental design)

Characteristic	Description
Study citation	Ross, S. M., McNelis, M., Lewis, T., & Loomis, S. (1998). Evaluation of Success for All^® programs: Little Rock school district year 1: 1997–1998. Memphis, TN: The University of Memphis, Center for Research in Education Policy.
Participants	This study involved 97 first-grade students with both pretest and posttest data in four schools. Two schools implemented the Success for All^® program (40 students), and two schools were selected as their matched comparison schools (47 students). The SFA^® schools and the comparison schools were similar in poverty level, achievement level, and enrollment. The study reported data on students’ outcomes after one year of program implementation.
Setting	The study took place in four elementary schools in Little Rock, Arkansas.
Intervention	Intervention students received the typical SFA^® program, including the SFA^® reading curriculum, tutoring for students in grades 1–3, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers.
Comparison	No information was provided on the nature of the comparison curriculum. The two comparison schools were matched to the SFA^® schools based on poverty level, achievement level, and enrollment. Pretest PPVT scores were used as a covariate to adjust for differences in students’ abilities.
Primary outcomes and measurement	Three subtests of the Woodcock Reading Mastery Test were administered: Word Identification, Word Attack, and Passage Comprehension. The Durrell Analysis of Reading Difficulty Oral Reading subtest was also used (see Appendices A2.1–A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	No information on training for the teachers in this study was provided.

Top

Appendix A1.7 Study characteristics: Smith, Ross, Faulks, Casey, Shapiro, & Johnson, 1993 (quasi-experimental design)

Characteristic	Description
Study citation	Smith, L. J., Ross, S. M., Faulks, A., Casey, J., Shapiro, M., & Johnson, B. (1993). 1991–1992 Ft. Wayne, Indiana SFA^® results. Memphis, TN: The University of Memphis, Center for Research in Education Policy.
Participants	This study involved approximately 286 students in kindergarten and first grade in four elementary schools in Fort Wayne, Indiana. Two schools implemented the SFA^® program. Two comparison schools were matched to the intervention schools based on poverty level, historical achievement level, and ethnicity; then pairs of students were matched on PPVT pretest scores. There were 74 kindergarteners and 69 first-grade students in the intervention group, and 74 kindergarteners and 69 first-grade students in the comparison group. Exact student attrition rates are not known for this study; however, the post-attrition intervention and comparison samples were equivalent on achievement pretest. School-level data—poverty level, achievement, and enrollment—were similar across all schools. The study included data on students’ outcomes after one year of program implementation.¹
Setting	The study took place in four elementary schools in Fort Wayne, Indiana.
Intervention	Intervention students received the typical SFA^® program, including the SFA^® reading curriculum, tutoring for students, quarterly assessments, family support teams for students’ parents, a facilitator who worked with school personnel, and training for all intervention teachers.
Comparison	Comparison schools continued using their regular, previously planned curriculum. No other information was provided on the comparison curriculum.
Primary outcomes and measurement	Four subtests of the Woodcock Reading Mastery Test were used: Letter Identification, Word Identification, Word Attack, and Passage Comprehension. Additional measures included the Peabody Picture Vocabulary Test and Durrell Analysis of Reading Difficulty Oral Reading subtest. The Merrill Language Screening Test and the Test of Language Development were also administered but have not been included in this review because they were outside the scope of the Beginning Reading review (see Appendices A2.1 – A2.3 for more detailed descriptions of outcome measures).
Staff/teacher training	Teachers in their first year of teaching SFA^® classes received three days of summer training and two to four additional in-service days during the school year. A school facilitator monitored and provided feedback throughout the year. Twice a year, trainers provided by the developer visited and observed teachers. After the first year, training was reinforced by regular in-services, an annual SFA^® conference, and implementation checks for the facilitators and trainers.
¹Additional findings for a low-achieving subset of students (lowest 25% with respect to reading achievement) are presented in Appendices A4.1–A4.6.

Top

Appendix A2.1 Outcome measures in the alphabetics domain by construct

Outcome measure	Description
*Letter knowledge*
Woodcock Reading Mastery Test (WRMT): Letter Identification subtest	The standardized test measures the number of letters that students are able to identify correctly (Smith et al., 1993).
*Phonics*
WRMT: Word Identification subtest	The Word Identification subtest is a test of decoding skills. The standardized test requires the child to read aloud isolated real words that range in frequency and difficulty (as cited in Borman et al., 2006; Ross & Casey, 1998; Ross, Alberg, & McNelis, 1997; Ross et al., 1998; Smith et al., 1993).
Woodcock Language Proficiency Battery (WLPB): Letter-Word Identification subtest	The Letter/Word Identification subtest is a standardized test that requires the child to read aloud isolated letters and real words that range in frequency and difficulty (as cited in Dianda & Flaherty, 1995; Madden et al., 1993).
WRMT and WLPB: Word Attack subtest	The standardized test measures phonemic decoding skills by asking students to read pseudowords. Students are aware that the words are not real (as cited in Borman et al., 2006; Dianda & Flaherty, 1995; Madden et al., 1993; Ross, Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993).

Top

Appendix A2.2 Outcome measures in the comprehension domain by construct

Outcome measure	Description
*Reading comprehension*
WRMT and WLPB: Passage Comprehension subtest	In this standardized test, comprehension is measured by having students fill in missing words in a short paragraph (as cited in Borman et al., 2006; Dianda & Flaherty, 1995; Ross, Alberg, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993).
Durrell Analysis of Reading Difficulty (DARD): Silent Reading Test	An individually administered, standardized diagnostic test that measures reading rate while students read passages silently and answer comprehension questions (as cited in Madden et al., 1993).
*Vocabulary development*
Peabody Picture Vocabulary Test (PPVT)	A standardized, receptive vocabulary test that asks students to choose which one of four pictures corresponds to a test word spoken aloud (as cited in Smith et al., 1993).

Top

Appendix A2.3 Outcome measures in the general reading domain by construct

Outcome measure	Description
California Achievement Test (CAT) Total Reading	A group-administered, standardized assessment battery composed of numerous reading and language-oriented subtests (as cited in Madden et al., 1993).
DARD Oral Reading Test	An individually administered, standardized diagnostic test that measures reading accuracy, reading rate, and oral reading comprehension (as cited in Madden et al., 1993; Ross, Albert, & McNelis, 1997; Ross & Casey, 1998; Ross et al., 1998; Smith et al., 1993).

Top

Appendix A3.1 Summary of findings for all domains¹

Domain
	Alphabetics		Comprehension
Outcome measure	Letter identification	Phonics	Reading comprehension	Vocabulary development	General reading achievement
*Meets evidence standards*
Borman et al., 2006	nr	+	+	nr	nr
*Meets evidence standards with reservations*
Dianda & Flaherty, 1995	nr	(+)	(+)	nr	(+)
Madden et al., 1993	nr	+	nr	nr	(+)
Ross, Alberg, & McNelis, 1997	nr	ind	ind	nr	ind
Ross & Casey, 1998	nr	ind	ind	nr	ind
Ross et al., 1998	nr	(+)	ind	nr	ind
Smith et al., 1993	(+)	(+)	(+)	ind	(+)
Rating of effectiveness	Positive		Mixed effects		Potentially positive
nr = no reported outcomes under this construct + = study finding was positive and statistically significant (+) = study finding was positive and substantively important, but not statistically significant ind = study finding was indeterminate, that is, neither substantively important nor statistically significant ¹This appendix reports findings considered for the effectiveness rating and the average improvement indices in each domain. More detailed information on findings for all measures within the domains and the constructs that factor into the domains can be found in Appendices A3.2–A3.4.

Top

Appendix A3.2 Summary of findings for alphabetics domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Borman et al., 2006 (randomized controlled trial)⁸ —Three years of intervention
WRMT: Word ID subtest⁹	Phonics	Kindergarten	35/1,425	462.96 (23.56)	457.41 (25.72)	5.55	0.22	Statistically significant	+9
WRMT: Word Attack subtest⁹	Phonics	Kindergarten	35/1,425	493.43 (16.45)	487.73 (17.64)	5.70	0.33	Statistically significant	+13
Madden et al., 1993 (quasi-experimental design)⁸,¹⁰ —Three years of intervention
WLPB: Letter-Word ID subtest	Phonics	Prekindergarten (Cohort 1)	10/492	18.53 (5.34)	15.91 (6.59)	2.62	0.44	ns	+17
WLPB: Word Attack subtest	Phonics	Prekindergarten (Cohort 1)	10/492	5.46 (4.11)	2.25 (3.55)	3.21	0.83	Statistically significant	+30
WLPB: Letter-Word ID subtest	Phonics	Kindergarten (Cohort 2)	10/440	25.09 (6.65)	21.54 (6.72)	3.55	0.53	ns	+20
WLPB: Word Attack subtest	Phonics	Kindergarten (Cohort 2)	10/440	8.63 (6.27)	5.21 (4.76)	3.42	0.61	ns	+23
WLPB: Letter-Word ID subtest	Phonics	Grade 1 (Cohort 3)	10/410	28.69 (6.72)	25.56 (6.19)	3.12	0.48	ns	+19
WLPB: Word Attack subtest	Phonics	Grade 1 (Cohort 3)	10/410	10.77 (6.94)	7.02 (5.49)	3.74	0.60	ns	+23
Dianda & Flaherty, 1995 (quasi-experimental design)⁸ —Two years of intervention
WLBP: Letter-Word ID subtest	Phonics	English-speaking kindergarten (1992 cohort)	7/219	nr	nr	na	0.34¹¹	ns	+13
WLBP: Word Attack subtest	Phonics	English-speaking kindergarten (1992 cohort)	7/219	nr	nr	na	0.26¹¹	ns	+10
Ross & Casey, 1998 (quasi-experimental design)⁸ —Two years of intervention
WRMT: Word ID subtest	Phonics	Kindergarten	7/288	32.14 (14.63)	31.30 (14.20)	0.84	0.06	ns	+2
WRMT: Word Attack subtest	Phonics	Kindergarten	7/288	12.25 (7.36)	10.40 (8.20)	1.85	0.23	ns	+9
Ross, Alberg, & McNelis, 1997 (quasi-experimental design)⁸ —One year of intervention
WRMT: Word ID subtest	Phonics	Grade 1	6/252	nr	nr	na	–0.01¹²	ns	0
WRMT: Word Attack subtest	Phonics	Grade 1	6/252	18.35	15.86	2.49 (8.89)¹³	0.28¹²	ns	+11
Ross et al., 1998 (quasi-experimental design)⁸ —One year of intervention
WRMT: Word ID subtest	Phonics	Grade 1	4/97	38.27	36.21	2.06 (12.31)¹⁴	0.17	ns	+7
WRMT: Word Attack subtest	Phonics	Grade 1	4/97	15.17	11.19	3.98 (8.89)¹⁴	0.44	ns	+17
Smith et al., 1993 (quasi-experimental design)⁸ —One year of intervention
WRMT: Word ID subtest	Phonics	Kindergarten (Cohort 1)	4/148	10.26 (9.82)	3.15 (4.95)	7.11	0.91	ns	+32
WRMT: Letter ID subtest	Letter knowledge	Kindergarten (Cohort 1)	4/148	32.43 (4.28)	29.36 (7.81)	3.07	0.48	ns	+19
WRMT: Letter ID subtest	Letter knowledge	Grade 1 (Cohort 2)	4/138	nr	nr	na	0.08¹¹	ns	+3
WRMT: Word ID subtest	Phonics	Grade 1 (Cohort 2)	4/138	35.04 (10.63)	28.00 (14.70)	7.04	0.55	ns	+21
WRMT: Word Attack subtest	Phonics	Grade 1 (Cohort 2)	4/138	12.60 (7.43)	7.90 (7.91)	4.70	0.61	ns	+23
Averages for alphabetics¹⁵
Borman et al., 2006—Three years of intervention							0.28	Statistically significant	+11
Madden et al., 1993—Three years of intervention							0.58	ns	+22
Dianda & Flaherty, 1995—Two years of intervention							0.30	ns	+12
Ross & Casey, 1998—Two years of intervention							0.14	ns	+6
Ross, Alberg, & McNelis, 1997—One year of intervention							0.13	ns	+5
Ross et al., 1998—One year of intervention							0.31	ns	+12
Smith et al., 1993—One year of intervention							0.56	ns	+21
Domain average for alphabetics across all studies							0.33	na	+13
*Averages by years of SFA^® implementation*
Average of results from studies with three years of intervention (two studies)							0.43	na	+17
Average of results from studies with two years of intervention (two studies)							0.22	na	+9
Average of results from studies with one year of intervention (three studies)							0.33	na	+13
na = not applicable nr = not reported ns = not statistically significant ¹This appendix reports findings considered for the effectiveness rating and the average improvement indices. Earlier findings from longitudinal studies are not included in these ratings but are reported in Appendix A4.1. Subgroup findings from the studies are not included in these ratings but are reported in Appendix A4.4. ² The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Borman et al. (2006), a correction for multiple comparisons was needed so the significance levels may differ from those reported in the original study. There was no need to adjust for clustering because the findings were based on HLM analyses. In the case of the six other studies, corrections for both clustering and multiple comparisons were needed so the significance levels may differ from those reported in the original studies. ⁹ Standard deviations and adjusted means have been received through communication with the author (G. Borman, personal communication, 2006). ¹⁰WWC combined means and standard deviations for the SFA^® schools and for the control schools. Adjusted posttest means (with pretests standard scores as covariates) were used for effect size calculations. Kindergarten and grade 1 cohorts from Abbottston elementary school received four years of intervention. ¹¹ Authors reported effect sizes that used the comparison group standard deviation in the denominator (Glass’s delta). Effect size was computed by subtracting the comparison group mean from the intervention group mean and dividing the result by the comparison group standard deviation. ¹² Authors reported effect sizes adjusted for PPVT pretest scores. ¹³ The WWC derived the pooled standard deviation from the reported means and effect size. ¹⁴ Authors reported the pooled standard deviation. ¹⁵ The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes.

Top

Appendix A3.3 Summary of findings for comprehension domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Borman et al., 2006 (randomized controlled trial)⁸ —Three years of intervention
WRMT: Passage Comprehension subtest⁹	Reading comprehension	Kindergarten	35/1,425	481.41 (14.20)	478.33 (15.33)	3.08	0.21	Statistically significant	+8
Dianda & Flaherty, 1995 (quasi-experimental design)⁸ —Two years of intervention
WLPB: Passage Comprehension subtest	Reading comprehension	English-speaking kindergarten (1992 cohort)	7/219	nr	nr	na	0.44	ns	+17
Ross & Casey, 1998 (quasi-experimental design)⁸ —Two years of intervention
WRMT: Passage Comprehension subtest	Reading comprehension	Kindergarten	7/288	16.09 (8.46)	15.40 (8.70)	0.69	0.08	ns	+3
Ross, Alberg, & McNelis, 1997 (quasi-experimental design)⁸ —One year of intervention
WRMT: Passage Comprehension subtest	Reading comprehension	Grade 1	6/252	nr	nr	na	0.01¹¹	ns	0
Ross et al., 1998 (quasi-experimental design)⁸ —One year of intervention
WRMT: Passage Comprehension subtest	Reading comprehension	Grade 1	4/97	19.19	17.73	1.46 (8.19)¹²	0.18	ns	+7
Smith et al., 1993 (quasi-experimental design)⁸ —One year of intervention
Peabody Picture Vocabulary Test	Vocabulary development	Kindergarten (Cohort 1)	4/148	nr	nr	na	0.17¹⁰	ns	+7
WRMT: Passage Comprehension subtest	Reading comprehension	Grade 1 (Cohort 2)	4/136	16.37 (8.07)	13.91 (9.31)	2.46	0.28	ns	+11
Averages for comprehension¹³
Borman et al., 2006—Three years of intervention							0.21	Statistically significant	+8
Dianda & Flaherty, 1995—Two years of intervention							0.44	ns	+17
Ross & Casey, 1998—Two years of intervention							0.08	ns	+3
Ross, Alberg, & McNelis, 1997—One year of intervention							0.01	ns	0
Ross et al., 1998—One year of intervention							0.18	ns	+7
Smith et al., 1993—One year of intervention							0.23	ns	+9
Domain average for comprehension across all studies							0.19	na	+8
*Averages by years of SFA^® implementation*
Results from study with three years of intervention (one study)							0.21	Statistically significant	+8
Average of results from studies with two years of intervention (two studies)							0.26	na	+10
Average of results from studies with one year of intervention (three studies)							0.14	na	+6
na = not applicable nr = not reported ns = not statistically significant ¹ This appendix reports findings considered for the effectiveness rating and the average improvement indices. Earlier findings from longitudinal studies are not included in these ratings but are reported in Appendix A4.2. Subgroup findings from the studies are not included in these ratings, but are reported in Appendix A4.5. ² The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, the kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Borman et al. (2006), there was no need to adjust for clustering because the findings were based on HLM analyses. In the cases of Dianda and Flaherty (1995), Ross and Casey (1998), Ross, Alberg, and McNelis (1997), and Ross et al. (1998), a correction for clustering was needed so the significance levels may differ from those reported in the original studies. In the case of Smith et al. (1993), corrections for both clustering and multiple comparisons were needed so the significance levels may differ from those reported in the original study. ⁹ Standard deviations and adjusted means have been received through communication with the author. ¹⁰ Authors reported effect sizes that used the comparison group standard deviation in the denominator (Glass’s delta). Effect size was computed by subtracting the comparison group mean from the intervention group mean and dividing the result by the comparison group standard deviation. ¹¹ Authors reported effect sizes adjusted for PPVT pretest scores. ¹² Authors reported the pooled standard deviation. ¹³ The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes.

Top

Appendix A3.4 Summary of findings for general reading achievement domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Dianda & Flaherty, 1995 (quasi-experimental design)⁸,⁹
*Four years of intervention*
3 WLPB subtests and Durrell Reading subtest combined	General reading	English-speaking kindergarten (1992 cohort)	6/136	nr	nr	na	0.23¹⁰	ns	+9
*Three years of intervention*
3 WLPB subtests and Durrell Reading subtest combined	General reading	English-speaking kindergarten (1993 cohort)	6/167	nr	nr	na	0.34¹⁰	ns	+13
*Two years of intervention*
3 WLPB subtests and Durrell Reading subtest combined	General reading	English-speaking kindergarten (1994 cohort)	6/153	nr	nr	na	0.27¹⁰	ns	+11
Madden et al., 1993 (quasi-experimental design)⁹,¹¹ —Three years of intervention
Durrell Oral Reading subtest	General reading	Prekindergarten (Cohort 1)	10/492	5.59 (4.78)	4.26 (5.16)	1.33	0.27	ns	+11
Durrell Oral Reading subtest	General reading	Kindergarten (Cohort 2)	10/440	11.99 (7.28)	8.84 (6.05)	3.15	0.47	ns	+18
Durrell Oral Reading subtest	General reading	Grade 1 (Cohort 3)	10/410	16.66 (7.00)	13.25 (7.13)	3.41	0.48	ns	+19
Ross & Casey, 1998 (quasi-experimental design)⁹ —Two years of intervention
Durrell Oral Reading subtest	General reading	Kindergarten	7/288	5.35 (4.63)	4.70 (4.30)	0.65	0.15	ns	+6
Ross, Alberg, & McNelis, 1997 (quasi-experimental design)⁹ —One year of intervention
Durrell Oral Reading subtest	General reading	Grade 1	6/252	nr	nr	na	0.04¹²	ns	+2
Ross et al., 1998 (quasi-experimental design)⁹ —One year of intervention
Durrell Oral Reading subtest	General reading	Grade 1	4/97	7.01	6.46	0.55 (3.52)¹³	0.16	ns	+6
Smith et al., 1993 (quasi-experimental design)⁹ —One year of intervention
Durrell Oral Reading subtest	General reading	Grade 1	4/138	6.74 (4.25)	4.68 (3.83)	2.06	0.51	ns	+19
Averages for general reading achievement¹⁴
Dianda & Flaherty, 1995¹⁰ —Two to four years of intervention							0.28	ns	+11
Madden et al., 1993—Three years of intervention							0.41	ns	+16
Ross & Casey, 1998—Two years of intervention							0.15	ns	+6
Ross, Alberg, & McNelis, 1997—One year of intervention							0.04	ns	+2
Ross et al., 1998—One year of intervention							0.16	ns	+6
Smith et al., 1993—One year of intervention							0.51	ns	+19
Domain average for general reading achievement across all studies							0.26	na	+10
*Averages by years of SFA^® implementation*
Results from study with four years of intervention (one study)							0.23	ns	+9
Average of results from studies with three years of intervention (two studies)							0.37	na	+14
Average of results from studies with two years of intervention (two studies)							0.21	ns	+8
Average of results from studies with one year of intervention (three studies)							0.24	na	+9
na = not applicable nr = not reported ns = not statistically significant ¹This appendix reports findings considered for the effectiveness rating and the average improvement indices. Earlier findings from longitudinal studies are not included in these ratings but are reported in Appendix A4.3. Subgroup findings from the studies are not included in these ratings but are reported in Appendix A4.6. ² The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ Data are taken from Livingston and Flaherty (1997). ⁹ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the cases of Dianda and Flaherty (1995), Madden et al. (1993), and Smith et al. (1993), a correction for clustering and multiple comparisons was needed so the significance levels may differ from those reported in the original studies. In the cases of Ross and Casey (1998), Ross, Alberg, and McNelis (1997), and Ross et al. (1998), a correction for clustering was needed so the significance levels may differ from those reported in the original studies. ¹⁰Authors reported effect sizes that used the comparison group standard deviation in the denominator (Glass’s delta). Effect size was computed by subtracting the comparison group mean from the intervention group mean and dividing the result by the comparison group standard deviation. ¹¹WWC combined means and standard deviations for the SFA^® schools and for the control schools. Adjusted posttest means (with pretests standard scores as covariates) were used for effect size calculations. Kindergarten and grade 1 cohorts from Abbottston elementary school received four years of intervention. ¹² Authors reported effect sizes adjusted for PPVT pretest scores. ¹³ Authors reported the pooled standard deviation. ¹⁴The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes.

Top

Appendix A4.1 Summary of earlier findings from longitudinal studies for alphabetics domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Borman et al., 2006 (randomized controlled trial)⁸ —Two years of intervention
WRMT: Letter ID subtest	Letter knowledge	Kindergarten and Grade 1	38/3,353	451.42 (14.08)	449.46 (11.19)	1.96	0.15	ns	+6
WRMT: Word ID subtest	Phonics	Kindergarten and Grade 1	38/3,353	449.52 (28.31)	444.82 (29.18)	4.70	0.16	ns	+6
WRMT: Word Attack subtest	Phonics	Kindergarten and Grade 1	38/3,353	487.92 (18.20)	483.29 (19.82)	4.63	0.24	Statistically significant	+10
ns = not statistically significant ¹This appendix presents earlier longitudinal findings for measures that fall in the alphabetics domain. Data that reflected students’ exposure to the intervention for the longest period of time were used for intervention rating purposes and are presented in Appendix A3.2. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not applied to findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Borman et al. (2006), there was no need to adjust for clustering because the data were based on HLM analyses.

Top

Appendix A4.2 Summary of earlier findings from longitudinal studies for comprehension domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Borman et al., 2006 (randomized controlled trial)⁸ —Two years of intervention
WRMT: Passage Comprehension subtest	Reading comprehension	Kindergarten and Grade 1	38/3,353	472.00 (18.29)	469.87 (19.53)	2.13	0.11	ns	+4
ns = not statistically significant ¹ This appendix presents earlier longitudinal findings for measures that fall in comprehension domain. Data that reflected students' exposure to the intervention for the longest period of time were used for intervention rating purposes and are presented in Appendix A3.3. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Borman et al. (2006), there was no need to adjust for clustering because the findings were based on HLM analyses.

Top

Appendix A4.3 Summary of earlier findings from longitudinal studies for general reading achievement domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Dianda & Flaherty, 1995 (quasi-experimental design)⁸,⁹
*Three years of intervention*
3 WLPB subtests and Durrell Reading subtest combined	General reading	English-speaking kindergarten (1992 cohort)	6/136	nr	nr	na	0.44¹⁰	ns	+17
*Two years of intervention*
3 WLPB subtests and Durrell Reading subtest combined	General reading	English-speaking kindergarten (1993 cohort)	6/167	nr	nr	na	0.87¹⁰	Statistically significant	+31
na = not applicable nr = not reported ns = not statistically significant ¹ This appendix presents earlier longitudinal findings for measures that fall in general reading domain. Data that reflected students' exposure to the intervention for the longest period of time were used for intervention rating purposes and are presented in Appendix A3.4. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ Data are taken from Livingston and Flaherty (1997). ⁹ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Dianda and Flaherty (1995), a correction for clustering was needed so the significance levels may differ from those reported in the original study. ¹⁰ Authors reported effect sizes that used comparison group standard deviation in the denominator (Glass's delta). Effect size was computed by subtracting the comparison group mean from the intervention group mean and dividing the result by the comparison group standard deviation.

Top

Appendix A4.4 Summary of subgroup findings for alphabetics domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Madden et al., 1993 (quasi-experimental design)⁸,⁹ —Three years of intervention
WLPB: Letter-Word ID subtest	Phonics	Pre-kindergarten/ lowest 25% (Cohort 1)	10/126	16.65 (5.34)	12.56 (6.66)	4.09	0.67	Statistically significant	+25
WLPB: Word Attack subtest	Phonics	Pre-kindergarten/ lowest 25% (Cohort 1)	10/126	4.92 (4.38)	1.52 (3.39)	3.40	0.86	Statistically significant	+31
WLPB: Letter-Word ID subtest	Phonics	Kindergarten/ lowest 25% (Cohort 2)	10/112	19.19 (4.80)	15.50 (5.54)	3.69	0.71	Statistically significant	+26
WLPB: Word Attack subtest	Phonics	Kindergarten/ lowest 25% (Cohort 2)	10/112	4.73 (3.68)	1.48 (2.17)	3.25	1.07	Statistically significant	+36
WLPB: Letter-Word ID subtest	Phonics	Grade 1/ lowest 25% (Cohort 3)	10/104	25.06 (6.85)	21.31 (4.75)	3.75	0.63	ns	+24
WLPB: Word Attack subtest	Phonics	Grade 1/ lowest 25% (Cohort 3)	10/104	7.85 (6.52)	4.02 (4.02)	3.83	0.70	Statistically significant	+26
Ross & Casey, 1998 (quasi-experimental design)⁹ —Two years of intervention
WRMT: Word ID subtest	Phonics	Kindergarten/ lowest 25%	7/79	27.10 (14.25)	25.10 (13.40)	2.00	0.15	ns	+6
WRMT: Word Attack subtest	Phonics	Kindergarten/ lowest 25%	7/79	10.11 (6.13)	7.80 (8.10)	2.31	0.30	ns	+12
Smith et al., 1993 (quasi-experimental design)⁹ —One year of intervention
WRMT: Letter ID subtest	Letter knowledge	Kindergarten/ lowest 25% (Cohort 1)	4/38	nr	nr	na	0.38¹⁰	ns	+15
WRMT: Word ID subtest	Phonics	Kindergarten/ lowest 25% (Cohort 1)	4/38	nr	nr	na	2.56¹⁰	Statistically significant	+49
WRMT: Letter ID subtest	Letter knowledge	Grade 1/ lowest 25% (Cohort 2)	4/38	nr	nr	na	–0.07¹⁰	ns	–3
WRMT: Word ID subtest	Phonics	Grade 1/ lowest 25% (Cohort 2)	4/38	28.16 (10.02)	18.53 (12.78)	9.63	0.82	ns	+29
WRMT: Word Attack subtest	Phonics	Grade 1/ lowest 25% (Cohort 2)	4/38	9.05 (5.37)	4.68 (5.76)	4.37	0.77	ns	+28
na = not applicable nr = not reported ns = not statistically significant ¹ This appendix presents subgroup findings (students in the lowest 25% of their grades) for measures that fall in the alphabetics domain. Total group scores were used for rating purposes and are presented in Appendix A3.2. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ WWC combined means and standard deviations for the SFA^® schools and for the control schools. Adjusted posttest means (with pretests standard scores as covariates) were used for effect size calculations. Kindergarten and grade 1 cohorts from Abbottston elementary school received four years of intervention. ⁹ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the cases of Ross and Casey (1998), Madden et al. (1993), and Smith et al. (1993), a correction for clustering was needed so the significance levels may differ from those reported in the original studies. ¹⁰ Authors reported effect sizes that used comparison group standard deviation in the denominator (Glass's delta).

Top

Appendix A4.5 Summary of subgroup findings for comprehension domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Ross & Casey, 1998 (quasi-experimental design)⁸ —Two years of intervention
WRMT: Passage Comprehension subtest	Reading comprehension	Kindergarten/lowest 25%	7/79	12.29 (7.79)	11.20 (8.20)	1.09	0.13	ns	+5
Smith et al., 1993 (quasi-experimental design)⁸ —One year of intervention
Peabody Picture Vocabulary Test	Vocabulary development	Kindergarten/lowest 25% (Cohort 1)	4/38	nr	nr	na	0.26⁹	ns	+10
WRMT: Passage Comprehension subtest	Reading comprehension	Grade 1/lowest 25% (Cohort 2)	4/38	9.84 (6.18)	8.11 (7.13)	1.73	0.25	ns	+10
na = not applicable nr = not reported ns = not statistically significant ¹ This appendix presents subgroup findings (students in the lowest 25% of their grades) for measures that fall in the comprehension domain. Total group scores were used for rating purposes and are presented in Appendix A3.3. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the cases of Ross and Casey (1998), Madden et al. (1993), and Smith et al. (1993), a correction for clustering was needed so the significance levels may differ from those reported in the original studies. ⁹ Authors reported effect sizes that used comparison group standard deviation in the denominator (Glass's delta). Effect size was computed by subtracting the comparison group mean from the intervention group mean and dividing the result by the comparison group standard deviation.

Top

Appendix A4.6 Summary of subgroup findings for general reading achievement domain¹

				Authors' findings from the study
				Mean outcome (standard deviation)²		WWC calculations
Outcome measure	Construct	Study sample³	Sample size (schools/students)	Success for All^® group	Comparison group	Mean difference⁴ (SFA^® – comparison)	Effect size⁵	Statistical significance⁶ (at α = 0.05)	Improvement index⁷
Madden et al., 1993 (quasi-experimental design)⁸,⁹ —Three years of intervention
Durrell Oral Reading subtest	General reading	Pre-kindergarten/lowest 25% (Cohort 1)	10/126	4.35 (4.30)	1.81 (3.66)	2.54	0.63	ns	+24
Durrell Oral Reading subtest	General reading	Kindergarten/lowest 25% (Cohort 2)	10/112	6.04 (4.62)	3.32 (3.37)	2.72	0.67	Statistically significant	+25
Durrell Oral Reading subtest	General reading	Grade 1/lowest 25% (Cohort 3)	10/104	12.92 (6.39)	8.08 (4.87)	4.85	0.85	Statistically significant	+30
Ross & Casey, 1998 (quasi-experimental design)⁹ —Two years of intervention
Durrell Oral Reading subtest	General reading	Kindergarten/lowest 25%	7/79	4.14 (3.84)	3.00 (3.60)	1.14	0.31	ns	+12
ns = not statistically significant ¹ This appendix presents subgroup findings (students in the lowest 25% of their grades) for measures that fall in the general reading domain. Total group scores were used for rating purposes and are presented in Appendix A3.4. ² The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. ³ The cohort is defined by the time the pretest is administered. For example, kindergarten cohort describes students who completed pretest measures in kindergarten. ⁴ Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. ⁵ For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. ⁶ Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. ⁷ The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. ⁸ WWC combined means and standard deviations for the SFA^® schools and for the control schools. Adjusted posttest means (with pretests standard scores as covariates) were used for effect size calculations. Kindergarten and grade 1 cohorts from Abbottston elementary school received four years of intervention. ⁹ The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the cases of Madden et al. (1993) and Ross and Casey (1998), a correction for clustering was needed so the significance levels may differ from those reported in the original studies.

Top

Appendix A5.1 Success for All ^® rating for the alphabetics domain

The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.¹

For the outcome domain of alphabetics, the WWC rated Success for All^® as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Success for All^® was assigned the highest applicable rating.

Rating received
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. Met. Two studies showed a statistically significant positive effect, and one of the studies had a strong design. AND Criterion 2: No studies showing statistically significant or substantively important negative effects. Met. No studies showed statistically significant or substantively important negative effects.
¹ For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.

Rating received

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

Met. Two studies showed a statistically significant positive effect, and one of the studies had a strong design.

AND
Criterion 2: No studies showing statistically significant or substantively important negative effects.

Met. No studies showed statistically significant or substantively important negative effects.

¹ For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A5.2 Success for All ^® rating for the comprehension domain

The WWC rates an intervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.¹

For the outcome domain of comprehension, the WWC rated Success for All^® as having mixed effects. It did not meet the criteria for positive effects because only one study showed statistically significant positive effects. In addition, it did not meet the criteria for potentially positive effects because more studies showed indeterminate effects than substantively important or statistically significant positive effects. The remaining ratings (no discernible effects, potentially negative effects, and negative effects) were not considered because Success for All^® was assigned the highest applicable rating.

Rating received
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. Not met. No studies showed a statistically significant or substantively important negative effect. OR Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a statistically significant or substantively important effect. Met. One study showed a statistically significant positive effect, one study showed a substantively important positive effect, and four studies showed indeterminate effects.
Other ratings considered
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. Not met. Only one study had a statistically significant positive effect in this domain. AND Criterion 2: No studies showing statistically significant or substantively important negative effects. Met. No studies showed statistically significant or substantively important negative effects in this domain.
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. Criterion 1: At least one study showing a statistically significant or substantively important positive effect. Met. One study had a statistically significant positive effect, and one study had a substantively important positive effect in this domain. AND Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate effects than showing statistically significant or substantively important positive effects. Not met. No studies showed statistically significant or substantively important negative effects in this domain, and more studies showed indeterminate effects (four) than statistically significant (one) or substantively important positive effects (one) in this domain.
¹ For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A5.3 Success for All ^® rating for the general reading achievement domain

The WWC rates anintervention's effects in a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.¹

For the outcome domain of general reading achievement, the WWC rated Success for All^® as having potentially positive effects. It did not meet the criteria for positive effects because only one study showed a statistically significant positive effect. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Success for All^® was assigned the highest applicable rating.

Rating received
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. Criterion 1: At least one study showing a statistically significant or substantively important positive effect. Met. Three studies showed substantively important positive effects. AND Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate effects than showing statistically significant or substantively important positive effects. Met. No studies showed statistically significant or substantively important negative effects. Three studies showed indeterminate effects and three studies showed substantively important positive effects.
Other ratings considered
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. Not met. No studies showed a statistically significant positive effect. AND Criterion 2: No studies showing statistically significant or substantively important negative effects. Met. No studies showed statistically significant or substantively important negative effects.
¹ For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A6 Extent of evidence by domain

	Sample size
Outcome domain	Number of studies	Schools	Students	Extent of evidence¹
Alphabetics	7	73	3,909	Medium to large
Fluency	0	0	0	na
Comprehension	6	65	2,565	Medium to large
General reading achievement	6	37	2,573	Medium to large
na = not applicable/not studied ¹ A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small."

Top

What Works Clearinghouse

Appendices

What Works Clearinghouse Headlines