Evaluating the Efficacy of a Multidimensional Reading Comprehension Program for At-Risk Students and Reconsidering the Lowly Reputation of Tests of Near Transfer
Fuchs, D., Hendricks, E., Walsh, M. E., Fuchs, L. S., Gilbert, J. K., Tracy, W. Z., Patton, S. III, Davis, N., Kim, W., Elleman, A. M., & Peng, P. (2017). Unpublished Manuscript.
-
examining116Students, grades3-5
Department-funded evaluation
Review Details
Reviewed: December 2018
- Department-funded evaluation (findings for Reading comprehension (COMP))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Near-transfer reading comprehension (author developed) |
Reading comprehension (COMP) vs. Business as usual |
0 Days |
COMP vs. Control Grade: 5;
|
13.05 |
11.11 |
No |
-- | ||
|
Near-transfer reading comprehension (author developed) |
Reading comprehension (COMP) vs. Business as usual |
0 Days |
COMP vs. Control Grade: 3;
|
9.35 |
9.20 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Near-transfer reading comprehension (author developed) |
Reading comprehension (COMP) vs. Other intervention |
0 Days |
Comp vs. [WM]Comp Grade: 5;
|
13.05 |
12.47 |
No |
-- | ||
|
Near-transfer knowledge acquisition (author developed) |
Reading comprehension (COMP) vs. Other intervention |
0 Days |
COMP vs. [WM]COMPGrade: 5;
|
16.15 |
16.63 |
No |
-- | ||
|
Near-transfer knowledge acquisition (author developed) |
Reading comprehension (COMP) vs. Other intervention |
0 Days |
COMP vs. [WM]COMP Grade: 3;
|
13.75 |
14.95 |
No |
-- | ||
|
Near-transfer reading comprehension (author developed) |
Reading comprehension (COMP) vs. Other intervention |
0 Days |
COMP vs. [WM]COMP Grade: 3;
|
9.85 |
11.26 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 49%
Male: 51% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
South
-
Race Black 39% Other or unknown 32% White 29% -
Ethnicity Hispanic 24% Not Hispanic or Latino 76%
Study Details
Setting
The study took place in a large urban school district. There were a total of 120 3rd and 5th-grade students in 13 schools across the three study groups. The third-grade students were from 8 schools and 28 classrooms, and the fifth-grade students were from 5 schools and 22 classrooms.(pp. 5,7)
Study sample
Students in the analytic sample were about evenly split between males and females, with 49 percent of the sample being female. Black students made up 39 percent of the sample, and white students were 29 percent of the sample. Race is not observed for the 24 percent of the sample that is Hispanic, and is not reported for 8 percent of the sample whose race is "other." The racial and ethnic characteristics of the grade 3 subgroup was quite different from the grade 5 sample. For example, in the grade 3 sample, 46 percent of students were Hispanic as compared to 2 percent in the grade 5 sample. Nearly all students in the sample, 94 percent, were eligible for free or reduced price lunch, and 6 percent had individualized education plans. The study does not present sample characteristics separately by intervention condition. (Table 2, p. 31)
Intervention Group
The study examined the effects of a reading comprehension intervention alone (COMP) and in combination with a working memory training component ([WM]COMP). For each of 14 weeks, both the COMP and [WM]COMP groups received tutoring three times per week. The first two tutoring sessions of each week lasted 45 minutes and were delivered to pairs of students. The third session of each week lasted 20 minutes and was delivered individually. All sessions were scripted. Both intervention groups received instruction in "before-reading strategies," which included, for example, identifying the title, headings, pictures, and charts; identifying and looking up bolded vocabulary words; and considering background information relevant to the passage. Both groups also received instruction in "after-reading strategies" which included retelling important facts from the passage, identifying the most important person or thing and the most important information about that person or thing, and constructing a main idea statement. Each tutoring session also began with a "speeded cloze activity" to enhance fluency and comprehension. This activity involved reading a paragraph summarizing the previous lesson's text, but with words omitted, and students had to choose which word from a pair belonged in the blank. (pp. 8-11) In addition to these activities, students in the [WM]COMP condition participated in activities to strengthen their working memory. For example, after completing the cloze activities students in the [WM]COMP group were asked to recall, in order, all the words they had selected to fill in the blanks. Thirty percent of students in the COMP condition missed at least some general reading instruction, most frequently Response to Intervention (RTI) or other intervention services (unrelated to COMP or [WM]COMP). Forty-seven percent of students in the [WM]COMP condition missed some general reading instruction, including activities such as centers, independent work, RTI or other intervention services (unrelated to COMP or [WM]COMP), and skills practice. (pp. 8-11)
Comparison Group
In the comparison condition, students received the business as usual reading instruction from their regular classroom teachers. All students (intervention and comparison) may have also received additional supports generally available to all students, including after school tutoring, English language tutoring, other small group tutoring, occupational therapy, or other available services. (p. 33)
Support for implementation
Tutors received three half-day training sessions, at which they learned the underlying rationale for each intervention component, saw demonstrations of instructional activities, and participated in role-playing exercises. After the training sessions, tutors practiced the COMP and [WM]COMP protocols with another tutor for six hours. They also "tutored" a project coordinator, who acted as a student, and had to show 90 percent adherence to both intervention protocols. (pp. 13-14)
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).