Data Collection and Outcome Measures

Exhibit ES.2 summarizes the study's three-year, multi-source data collection plan. The present report reflects data for 2004-05 and 2005-06. Key outcome measures include student reading comprehension, teacher reading instructional practices, and student engagement with print.

Exhibit ES-2. Data Collection Schedule for the Reading First Impact Study

Student reading comprehension was assessed with the Stanford Achievement Test, 10th Edition (SAT 10, Harcourt Assessment, Inc., 2004). Its comprehension subtests are well documented, broadly accepted, and widely used.² Test scores are analyzed in two forms: scaled scores and the percentage of students who read at or above grade level, based upon national SAT 10 norms. The SAT 10 was administered to students in grades one, two, and three during spring 2005 and spring 2006, with completion rates of 80 percent or higher for both waves.

Classroom instruction was assessed in first grade and second grade reading classes through an observation system developed by the study team called the Instructional Practice in Reading Inventory (IPRI). Observations were conducted in each study school on two consecutive days in spring 2005, fall 2005, and spring 2006, with completion rates over 96 percent.

Measures of classroom instruction were created from IPRI data to represent the components of reading instruction emphasized by the Reading First legislation:³

• Total daily minutes of instruction in all five dimensions: This measure equals the total number of minutes of instruction in phonemic awareness, phonics, vocabulary, fluency, and comprehension during the daily reading block, which is the time period designated for reading instruction.



• Minutes of instruction per day in each of the five dimensions: These five measures correspond to the number of minutes of instruction in each of the five dimensions per daily reading block.



• Percentage of three-minute observational intervals with instruction in the five dimensions that involve highly explicit instruction: This measure records instances of "highly explicit instruction" that occur during instruction in any of the five dimensions. Highly explicit instruction means active teaching, modeling or explaining concepts, or helping children use reading strategies.



• Percentage of three-minute observational intervals with instruction in the five dimensions that involve high quality student practice: This measure records instances of "high quality student practice" that occur during instruction in any of the five dimensions. High quality student practice involves dimension-specific opportunities for students to practice their skills.

Student engagement with print was assessed beginning in fall 2005 through classroom observations using the Student Time-on-Task and Engagement with Print (STEP) instrument to measure the percentage of students engaged in academic work who are reading or writing print. The STEP, which was developed by the study team, was used to observe classrooms in both fall 2005 and spring 2006, with a completion rate of over 97 percent.

Top