Two-thirds of Sam and Pat Classes Observed Demonstrated Evidence of Implementing Sam and Pat as Intended
About two-thirds (65 percent) of the Sam and Pat classes observed met the study's instructional fidelity criteria regarding the use of Sam and Pat materials and engagement in reading instruction. More specifically, these teachers met the following criteria that were established in collaboration with the developers before the study began:
Because we did not observe all hours of instruction throughout the term, we cannot determine how many hours of Sam and Pat instruction were received by each student. However, students in the Sam and Pat group met for an average of 79 hours total over the course of the term (not shown in tables). The Sam and Pat developers recommended that the text be implemented for a minimum of 60 hours per term.
More Reading Instruction Observed in Sam and Pat Classes, While More English Language Instruction Observed in Control Classes
The Sam and Pat classrooms spent more time on reading development instruction than control classrooms (66 percent vs. 19 percent of observed time intervals, respectively), and the difference was statistically significant (Figure ES.1). Conversely, the control classrooms spent more time on English language acquisition instruction than did Sam and Pat classrooms (68 percent vs. 27 percent of observed time intervals, respectively), and this difference was statistically significant. The control classrooms also spent more time on functional reading, writing and math instruction (content related to English language acquisition instruction) than Sam and Pat classrooms (18 percent vs. 5 percent of observed time intervals, respectively).4
Students Made Gains, but There Were No Overall Impacts of Sam and Pat on Students' Reading and English Language Skills
On average, students participating in the study made statistically significant gains over the course of the term (effect sizes of 0.23 to 0.40). These gains are equivalent to 1 to 2 months of growth on the reading assessments, and 5 to 6 months of growth on the English language assessments.5 However, there were no statistically significant impacts of Sam and Pat on the reading and English language outcomes measured for the overall sample (Figure ES.2). Effect sizes ranged from -0.06 to 0.01.
No Impacts of Sam and Pat on Reading and English Language Outcomes Found for Subgroups Based upon Student Native Language and Cohort
There were no statistically significant impacts found for students with a non-Roman-based alphabet background, native Spanish speakers, students from the first study cohort, or students from the second study cohort. Effect sizes ranged from -0.14 to 0.09.
Some Suggestive Evidence of a Positive Impact on Reading Outcomes for Lower Literacy Students
No statistically significant impacts were found for the students in the sample with relatively higher literacy levels (effect sizes ranged from -0.08 to 0.03). However, there was a suggestive finding for students who tested in the lower literacy score range at the beginning of the term. Within this subgroup, Sam and Pat group students scored higher on the Woodcock Johnson word attack (decoding) assessment than control group students (effect size = 0.16). Because this difference was not statistically significant after adjusting for multiple comparisons, however, it is possible that the effect is due to chance alone. No impacts were found for the lower literacy students on the other reading and English language outcomes measured.
Student Exposure to Reading or English Language Instruction Unrelated to Most Reading and English Language Outcomes Measured, Although Weak Relationships Found Between Exposure to Instruction and One English Language Outcome
Student exposure to instruction was measured by the combination of reading and English language instruction provided in study classes and the number of hours students attended study classes. No statistically significant relationships were found between exposure to instruction and any of the reading outcomes measured and two of the three English language outcomes measured. However, the amount of exposure to English language instruction was positively and statistically significantly correlated with ROWPVT scores. The opposite pattern was found for reading instruction; exposure to reading instruction had a negative and statistically significant relationship with scores on the ROWPVT. However, the standardized coefficients in both cases were small (0.034 and -0.032, respectively). As an example, the 0.034 coefficient on the ROWPVT assessment indicates that, after controlling for total student attendance hours, an increase of 10 percent in the number of English language instruction intervals a student attended is associated with a 0.34 point increase on the test (which had a sample mean of 29). In addition, similar to the student attendance results, we cannot rule out the possibility that the statistically significant relationships were driven by other factors. Therefore, these findings should be interpreted with caution.