Skip Navigation

National Center for Education Evaluation and Regional Assistance


The Impact of a Reading Intervention for Low-Literate Adult ESL Learners

NCEE 2011-4003
December 2010

Summary of Study Design and Methods

The study was designed to estimate the impact of Sam and Pat relative to standard ESL instruction (i.e., the kind of instruction ESL students in study sites would receive in the absence of the study) on reading and English language outcomes.

The evaluation employed a randomized research design that included the following:

  • 10 adult education program sites;
  • 33 teachers;
  • 66 classes; and
  • 1,344 low-literate adult ESL learners.

The program sites were a purposive sample. From among the states with the largest adult ESL enrollments, we selected sites that had enrollments of adult ESL literacy learners large enough to support the study design, 2 or more classes for ESL literacy students that met at the same time and in the same location, and an enrollment process that would accommodate random assignment.

Within each site, teachers and students were randomly assigned to one of two groups:

  • The Sam and Pat group, which was intended to include a minimum of 60 hours of Sam and Pat-based instruction per term, with any remaining class time being spent on the standard instruction provided by the program; and
  • The control group, which consisted of the standard instruction provided by the program.

Teachers (or cla sses) within each program site were randomly assigned in pairs, so that, within each pair, the Sam and Pat and control class met at the same time, in the same or an adjacent building, and for the same number of hours. Data collection for the study occurred between September 2008 and May 2009 with two cohorts of students, one that attended in fall 2008 and the second in spring 2009. Students were tested on the study's battery of assessments, which included tests of reading and English language skills at the beginning of the term and after about 12 weeks of instruction. A description and schedule for the study's data collections are provided in Table ES.1.

The following tests were selected to measure the range of skills that could potentially be impacted by Sam and Pat-based instruction:

Reading Skills

  • Woodcock-Johnson Letter-Word Identification (WJID; Woodcock, McGrew, & Mather, 2001)
  • Woodcock-Johnson Passage Comprehension (WJPC; Ibid.)
  • Woodcock-Johnson Word Attack (WJWA; Ibid.)
  • SARA Decoding (SARA Dec; Sabatini & Bruce, in press)

English Language Skills

  • Oral and Written Language Scales (OWLS; Carrow-Woolfolk, 1996)
  • Receptive One-Word Picture Vocabulary Test (ROWPVT; Brownell, 2000)
  • Woodcock-Johnson Picture Vocabulary Test (WJPV; Woodcock, McGrew, & Mather, 2001)

Table ES.1: Data Collection Schedule

The basic analytic strategy for assessing the impacts of Sam and Pat was to compare reading and English language outcomes for students who were randomly assigned to either the Sam and Pat or the control group, after controlling for student and teacher background characteristics (e.g., gender and ethnicity). The average outcome in the control group represents an estimate of the scores that would have been observed in the Sam and Pat group if they had not received the intervention; therefore, the difference in outcomes between the Sam and Pat and control groups provides an unbiased estimate of the impacts of Sam and Pat.

Top