Skip Navigation

National Center for Education Evaluation and Regional Assistance


The Impact of a Reading Intervention for Low-Literate Adult ESL Learners

NCEE 2011-4003
December 2010

Summary of Study Findings

Two-thirds of Sam and Pat Classes Observed Demonstrated Evidence of Implementing Sam and Pat as Intended

About two-thirds (65 percent) of the Sam and Pat classes observed met the study's instructional fidelity criteria regarding the use of Sam and Pat materials and engagement in reading instruction. More specifically, these teachers met the following criteria that were established in collaboration with the developers before the study began:

  • Sam and Pat materials must be used for a minimum of 1 hour of instruction per class day;
  • Each class day must include at least 1 hour of instruction in reading development; and
  • Each class day, instruction should occur in at least three of the reading development instructional areas (e.g., phonics, fluency, reading comprehension).

Because we did not observe all hours of instruction throughout the term, we cannot determine how many hours of Sam and Pat instruction were received by each student. However, students in the Sam and Pat group met for an average of 79 hours total over the course of the term (not shown in tables). The Sam and Pat developers recommended that the text be implemented for a minimum of 60 hours per term.

More Reading Instruction Observed in Sam and Pat Classes, While More English Language Instruction Observed in Control Classes

The Sam and Pat classrooms spent more time on reading development instruction than control classrooms (66 percent vs. 19 percent of observed time intervals, respectively), and the difference was statistically significant (Figure ES.1). Conversely, the control classrooms spent more time on English language acquisition instruction than did Sam and Pat classrooms (68 percent vs. 27 percent of observed time intervals, respectively), and this difference was statistically significant. The control classrooms also spent more time on functional reading, writing and math instruction (content related to English language acquisition instruction) than Sam and Pat classrooms (18 percent vs. 5 percent of observed time intervals, respectively).4

Figure ES.1: Percent of Observed Instructional Intervals Spent in Key Instructional Areas, by Group

Students Made Gains, but There Were No Overall Impacts of Sam and Pat on Students' Reading and English Language Skills

On average, students participating in the study made statistically significant gains over the course of the term (effect sizes of 0.23 to 0.40). These gains are equivalent to 1 to 2 months of growth on the reading assessments, and 5 to 6 months of growth on the English language assessments.5 However, there were no statistically significant impacts of Sam and Pat on the reading and English language outcomes measured for the overall sample (Figure ES.2). Effect sizes ranged from -0.06 to 0.01.

Figure ES.2: Impact of Sam and Pat on Reading and English Language Skills: Differences Between Sam and Pat and Control Groups at the End of the Term

No Impacts of Sam and Pat on Reading and English Language Outcomes Found for Subgroups Based upon Student Native Language and Cohort

There were no statistically significant impacts found for students with a non-Roman-based alphabet background, native Spanish speakers, students from the first study cohort, or students from the second study cohort. Effect sizes ranged from -0.14 to 0.09.

Some Suggestive Evidence of a Positive Impact on Reading Outcomes for Lower Literacy Students

No statistically significant impacts were found for the students in the sample with relatively higher literacy levels (effect sizes ranged from -0.08 to 0.03). However, there was a suggestive finding for students who tested in the lower literacy score range at the beginning of the term. Within this subgroup, Sam and Pat group students scored higher on the Woodcock Johnson word attack (decoding) assessment than control group students (effect size = 0.16). Because this difference was not statistically significant after adjusting for multiple comparisons, however, it is possible that the effect is due to chance alone. No impacts were found for the lower literacy students on the other reading and English language outcomes measured.

Student Exposure to Reading or English Language Instruction Unrelated to Most Reading and English Language Outcomes Measured, Although Weak Relationships Found Between Exposure to Instruction and One English Language Outcome

Student exposure to instruction was measured by the combination of reading and English language instruction provided in study classes and the number of hours students attended study classes. No statistically significant relationships were found between exposure to instruction and any of the reading outcomes measured and two of the three English language outcomes measured. However, the amount of exposure to English language instruction was positively and statistically significantly correlated with ROWPVT scores. The opposite pattern was found for reading instruction; exposure to reading instruction had a negative and statistically significant relationship with scores on the ROWPVT. However, the standardized coefficients in both cases were small (0.034 and -0.032, respectively). As an example, the 0.034 coefficient on the ROWPVT assessment indicates that, after controlling for total student attendance hours, an increase of 10 percent in the number of English language instruction intervals a student attended is associated with a 0.34 point increase on the test (which had a sample mean of 29). In addition, similar to the student attendance results, we cannot rule out the possibility that the statistically significant relationships were driven by other factors. Therefore, these findings should be interpreted with caution.

Top

4 We can only characterize implementation by reporting that (1) 65 percent of Sam and Pat classes met the study's fidelity criteria, and (2) significantly more reading instruction was delivered in these classes, as compared to the control group classes.
5 It should be noted that publisher guidelines for the grade and age equivalent calculations used to determine months of gains are based on norming populations that differ from the study population. (The WJ assessments were normed on a nationally representative sample of U.S. residents aged 2 to 90+; the OWLS on a representative U.S. sample aged 3 to 21 years; and the ROWPVT on a representative U.S. sample aged 2 to 18 years.) No norming data exist for low-literate adult ESL learners. Additionally, the study used simplified or translated testing instructions when students did not appear to understand the tester's directions. For these reasons, the number of months of growth should be interpreted with caution.