Year One Evaluation Report/Impact Study: Illinois Striving Readers
Dimitrov, Dimiter; Jurich, Sonia; Frye, Michael; Lammert, Jill; Sayko, Sarah; Taylor, Laura (2012). RMC Research Corporation. Retrieved from: https://eric.ed.gov/?id=ED600855
-
examining514Students, grade9
Passport Reading Journeys Intervention Report - Adolescent Literacy
Review Details
Reviewed: November 2019
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Passport Reading Journeys.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Gates-MacGinitie Reading Test |
Passport Reading Journeys vs. Business as usual |
0 Days |
9th grade students;
|
517.10 |
517.16 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Grade 9 EXPLORE |
Passport Reading Journeys vs. Business as usual |
0 Days |
9th grade students;
|
11.99 |
12.34 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 43%
Male: 57% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Illinois
-
Race Black 58% Other or unknown 12% White 30% -
Ethnicity Hispanic 5% Not Hispanic or Latino 95%
Study Details
Setting
The study took place in six public high schools in four school districts in Illinois during the 2010–11 school year. All of the schools were Title I schools that had not made, or were at risk of not making, adequate yearly progress under the No Child Left Behind Act. The Striving Readers grant from the U.S. Department of Education funded the study.
Study sample
The study authors identified 855 incoming ninth-grade students in the six participating high schools who met the eligibility criteria for the study. Each eligible student was matched to another similar eligible student, forming a pair. The authors then randomly assigned one student in each pair to receive the intervention and the other student to the comparison group so that 427 students were assigned to the intervention group and 428 students were assigned to the comparison group. The study examined the GMRT comprehension outcome for 232 intervention and 216 comparison students and the EXPLORE general literacy achievement outcome for 264 intervention and 250 comparison students. For both outcome measures, the sample loss after random assignment (attrition) was within the acceptable threshold for the review: the overall attrition rate was between 40% and 48%, and the differential attrition rate was between 3 and 4 percentage points. To be eligible for the study, ninth-grade students had to have taken the Grade 8 EXPLORE test (ACT, 2011) and scored at the bottom two quartiles, and could not have an individualized education plan. Six teachers, one in each of the six schools, provided the intervention to the students assigned to the intervention group. Of the students subject to random assignment, 58% were African-American, 5% were Hispanic, 30% were White, 43% were female, 85% were eligible for the free or reduced-price lunch program, and 18% were receiving special education services.
Intervention Group
Students in the intervention group received the Passport Reading Journeys program over the course of the 2010–11 school year. This supplemental intervention was offered during an elective period and involved daily lessons in reading skills related to science or social studies. The lessons were 50 minutes in duration (or 45 minutes when schools have 90-minute block scheduling). The lessons were organized into 14 ten-lesson expedition sequences, each two weeks long. Students also used two online tools available as part of Passport Reading Journeys: VocabJourney, a computer application for learning vocabulary words and improving comprehension skills at students’ own pace, and ReadingScape, a library of books and texts students could use to practice their reading skills independently.
Comparison Group
Students assigned to the comparison condition received the instruction in English language arts (ELA) that was already available in their schools and elective classes that did not provide any supplemental literacy instruction. The ELA instruction was available to both the intervention and comparison groups.
Support for implementation
Cambium Learning Group, the developer of the intervention, provided professional development and support to the six teachers delivering the intervention. This professional development and support included a launch training, online product training, coursework on adolescent literacy, and ongoing consultation. The launch training consisted of two eight-hour sessions that instructed teachers on how to deliver the intervention with fidelity. The online training modules provided instruction in a self-paced, interactive environment and enabled teachers to search, annotate, and bookmark information they found valuable. Cambium also provided one-on-one support to teachers delivering the intervention through trained experts who visited each school to observe how the intervention was being implemented. The number of visits was determined by the needs of the teachers and the contract between each school district and Cambium. The launch training, the online product training, and the online support are included in the cost of the standard program package. The coursework, ongoing consultation, and coaching was conducted at additional cost to school districts or schools.
IES Performance Measure
Review Details
Reviewed: February 2018
- IES Performance Measure (findings for Passport Reading Journeys)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Gates-MacGinitie Reading Tests (GMRT): Total |
Passport Reading Journeys vs. Business as usual |
0 Days |
Full sample;
|
N/A |
N/A |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
EXPLORE: Reading subtest |
Passport Reading Journeys vs. Business as usual |
0 Days |
Full sample;
|
N/A |
N/A |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Male: 57% -
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Illinois
-
Race Black 58% -
Ethnicity Hispanic 5%
Study Details
Setting
The study involved six high schools in four public school districts across Illinois (Danville, Decatur, Springfield, and Kankakee) during the 2010-2011 school year. All of the schools are Title I eligible schools that had not made or were at risk of not making adequate yearly progress under No Child Left Behind.
Study sample
The baseline sample was 57% male and 70% non-White (58% African American, 5% Hispanic, 30% other), while 85% qualified for free or reduced-price lunch and 18% were classified as special education.
Intervention Group
The PRJIII supplemental reading intervention was delivered to eligible ninth grade students by Reading Intervention Teachers. The program was delivered during "elective" periods to ensure that students in the intervention condition could participate in core content area classes. The PRJIII program involves daily, 50 minute lessons in reading skills related to science or social studies. Lessons are organized into 14 "expeditions" (2-week, ten lesson sections) that are taught throughout the entire school year. In this study, it was recommended that teachers with 90-minute blocks adapt the program by reducing the PRJIII lessons from 50 to 45 minutes and teach two lessons per day.
Comparison Group
Students in the comparison condition could enroll in electives that were not related to reading instruction.
Support for implementation
PRJII is published by the Cambium Learning Group, which offers a variety of professional development activities for teachers. This includes such activities as a launch training, online product training, coursework on adolescent literacy, and ongoing consultative support. The launch training consists of two 8-hour sessions that instructs teachers how to deliver the PRJ program with fidelity. The on-line modules provide instruction in a self-paced, interactive environment, and allow teachers to search, annotate, and bookmark information that they find to be valuable. The coursework on adolescent literacy is not part of the regular PRJII package and involves two 15-hour courses taught by contracted university professors who are content experts. Finally, Cambium also provides individualized support to teachers delivering the PRJIII program through experts called Voyager Implementation Specialists. The individual conducts visits to each school to observe how the intervention is being implemented, with the number of visits being determined by the needs of the teachers and the contract between the school district and Cambium.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).