The study was designed to estimate the impact of Sam and Pat relative to standard ESL instruction (i.e., the kind of instruction ESL students in study sites would receive in the absence of the study) on reading and English language outcomes.
The evaluation employed a randomized research design that included the following:
The program sites were a purposive sample. From among the states with the largest adult ESL enrollments, we selected sites that had enrollments of adult ESL literacy learners large enough to support the study design, 2 or more classes for ESL literacy students that met at the same time and in the same location, and an enrollment process that would accommodate random assignment.
Within each site, teachers and students were randomly assigned to one of two groups:
Teachers (or cla sses) within each program site were randomly assigned in pairs, so that, within each pair, the Sam and Pat and control class met at the same time, in the same or an adjacent building, and for the same number of hours. Data collection for the study occurred between September 2008 and May 2009 with two cohorts of students, one that attended in fall 2008 and the second in spring 2009. Students were tested on the study's battery of assessments, which included tests of reading and English language skills at the beginning of the term and after about 12 weeks of instruction. A description and schedule for the study's data collections are provided in Table ES.1.
The following tests were selected to measure the range of skills that could potentially be impacted by Sam and Pat-based instruction:
Reading Skills
English Language Skills
Table ES.1: Data Collection Schedule
The basic analytic strategy for assessing the impacts of Sam and Pat was to compare reading and English language outcomes for students who were randomly assigned to either the Sam and Pat or the control group, after controlling for student and teacher background characteristics (e.g., gender and ethnicity). The average outcome in the control group represents an estimate of the scores that would have been observed in the Sam and Pat group if they had not received the intervention; therefore, the difference in outcomes between the Sam and Pat and control groups provides an unbiased estimate of the impacts of Sam and Pat.