Appendix A1 Study characteristics: Frasco, 2008
| Characteristic | Description |
|---|---|
| Study citation | Frasco, R. D. (2008). Effectiveness of Reading First for English language learners: Comparison of two programs (Doctorial dissertation, Walden University, 2008). Dissertation Abstracts International, 69(03A), 141–879. |
| Participants |
The study was based on 36 first-grade English language learner students. Seventeen of these students were randomly assigned to the Read Well® intervention group and 19 were assigned to the Macmillan/McGraw-Hill control group. Two students attrited from the control group, resulting in 17 intervention group students and 17 control group students for a total of 34 students in the analysis sample. Overall attrition was 5.6% and differential attrition was 10.5%. |
| Setting | The study took place in a rural elementary school in eastern Colorado. English language learners constituted 61% of the school population. Eighty-two percent of the study body qualified for free and reduced-price lunch during the 2007–08 academic school year. |
| Intervention | For approximately three months, students received a minimum of 90 minutes of daily instruction in Read Well® 1, which was utilized as their core reading program. The pacing depended on the level of mastery for each individual participant, allowing students to accelerate or slow down according to their grasp of the material. |
| Comparison | Students in the control group were taught reading using Macmillan/McGraw-Hill 2003 core reading program and also received 90 minutes of daily instruction. The pacing of the program is based on completing a story weekly, with intervention provided for students after giving the unit test during Week 6 of each unit. This study encompassed two skills-based units, or 12 weeks of instruction. Teachers used lesson maps and templates that included recommendations for where to replace or add activities for struggling readers such as English language learner students. |
| Primary outcomes and measurement | Study measures in the reading achievement domain included the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Nonsense Word Fluency Subtest and Gray’s Oral Reading Test–Fourth Edition (GORT-4). Study measures in the English language development domain included the Peabody Picture Vocabulary Test–Third Edition (PPVT-III). The study also reports results on the Comprehensive Test of Phonological Processing (CTOPP) that are excluded from this report, since the test measures phonemic awareness, which is not part of any English Language Learners domain. For a more detailed description of these outcome measures, see Appendices A2.1–A2.2. |
| Staff/teacher training | Three teachers and four instructional aides were involved in the study. Teachers and instructional aides received professional development to implement the program. |
Appendix A2.1 Outcome measures for the reading achievement domain
| Outcome measure | Description |
|---|---|
|
Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Nonsense Word Fluency Subtest |
The DIBELS assessment is a screening tool used by teachers to test students one-on-one to determine early literacy skills. The Nonsense Word Fluency subtest is a one-minute probe that assesses the alphabetic principle. Students are given a list with consonant-vowel-consonant (CVC) and vowel-consonant (VC) nonsense words. Students are asked to orally produce each individual sound or read the whole nonsense word (as cited in Frasco, 2008). |
|
Gray’s Oral Reading Test–Fourth Edition (GORT-4) |
GORT-4 consists of norm-referenced tests of oral reading rate, accuracy, fluency, and comprehension. In general, the entry point for grades 1 and 2 is Story 1. Each story has five comprehension questions to answer following the oral reading of the given passage (as cited in Frasco, 2008). |
Appendix A2.2 Outcome measures for the English language development domain
| Outcome measure | Description |
|---|---|
|
Peabody Picture Vocabulary Test–Third Edition (PPVT-III) |
The PPVT-III assesses receptive vocabulary. Students are shown pictures by the examiner. The examiner gives the student a vocabulary term, and the students identify the term by pointing to the picture or providing an oral response (as cited in Frasco, 2008). |
Appendix A3.1 Summary of study findings included in the rating for the reading achievement domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation)2 |
WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Read Well® group3 |
Comparison group | Mean difference4 (Read Well®– comparison) | Effect size5 | Statistical significance6
(at α = 0.05) |
Improvement index7 |
|
Frasco, 20088
|
||||||||
| DIBELS: Nonsense Word Fluency Subtest | Grade 1 | 34 | 72.00 (17.34) |
73.35 (27.27) |
–1.35 | –0.06 | ns | –2.30 |
| Gray’s Oral Reading Test–Fourth Edition | Grade 1 | 34 | 12.00 (3.39) |
12.06 (4.18) |
–0.06 | –0.02 | ns | –0.61 |
| Domain average for reading achievement9 | –0.04 | na | –1.46 | |||||
|
ns = not statistically significant 2 The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. The standard deviations shown here differ from those shown in Frasco (2008), because the WWC uses posttest standard deviations, while the study reported standard deviations of pretest to posttest gains. 3 The mean of the intervention group reported here was calculated as the comparison group posttest mean plus the difference in pretest to posttest gains reported by the author. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference reflects the difference between groups in pretest to posttest gain scores calculated by the author. 5 For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. Effect sizes calculated based on data from Frasco (2008) use a difference-in-differences approach; that is, the numerator of the effect size is equal to the difference between the pretest-posttest mean difference for the intervention group and the pretest-posttest mean difference for the comparison group. The denominator reflects the pooled posttest standard deviation for the intervention and comparison groups. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The statistical significance reported here is not consistent with the author’s reported significance, since the WWC uses posttest standard deviations to provide the same metric across studies (as opposed to the study, which used standard deviations of pretest to posttest gains). 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting favorable results for the intervention group. 8 The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Frasco (2008), no corrections for clustering or multiple comparisons were needed. 9 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the English language development domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation)2 |
WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (students) | Read Well® group3 |
Comparison group | Mean difference4 (Read Well®– comparison) | Effect size5 | Statistical significance6 (at α = 0.05) |
Improvement index7 |
|
Frasco, 20088 |
||||||||
| PPVT-III | Grade 1 | 34 | 89.12 (17.65) |
81.41 (7.52) |
7.71 | 0.56 | ns | 21.06 |
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices for the English language development domain. |
||||||||
Appendix A4.1 Read Well® rating for the reading achievement domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of reading achievement, the WWC rated Read Well® as having no discernible effects for English language learners.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
OR
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
OR
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND
|
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. |
Appendix A4.2 Read Well® rating for the English language development domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of English language development, the WWC rated Read Well® as having potentially positive effects for English language learners. It did not meet the criteria for positive effects as it had only one study with a statistically significant and substantively important positive effect. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, as Read Well® was assigned the highest applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
| 1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. |
Appendix A5 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Reading achievement | 1 | 1 | 34 | Small |
| English language development | 1 | 1 | 34 | Small |
| Mathematics achievement | na | na | na | na |
|
na = not applicable/not studied 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small." For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. |
||||