Improving Content Knowledge and Comprehension for English Language Learners: Findings from a Randomized Control Trial
Vaughn, Sharon; Martinez, Leticia R.; Wanzek, Jeanne; Roberts, Greg; Swanson, Elizabeth; Fall, Anna-Mária (2017). Journal of Educational Psychology, v109 n1 p22-34. Retrieved from: https://eric.ed.gov/?id=EJ1125544
-
examining1,629Students, grade8
Single Study Review
Review Details
Reviewed: June 2021
- Single Study Review (findings for Promoting Adolescents' Comprehension of Text (PACT))
- Randomized Controlled Trial
- Meets WWC standards with reservations because it is a compromised cluster randomized controlled trial, but it satisfies the baseline equivalence requirement for the individuals in the analytic intervention and comparison groups.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Modified Assessment of Social Studies Knowledge (MASK) content reading comprehension |
Promoting Adolescents' Comprehension of Text (PACT) vs. Business as usual |
0 Days |
English Learners;
|
7.82 |
8.21 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Modified Assessment of Social Studies Knowledge (MASK) content reading comprehension |
Promoting Adolescents' Comprehension of Text (PACT) vs. Business as usual |
0 Days |
Non-English Learners;
|
11.06 |
10.61 |
No |
-- | ||
|
Gates-MacGinitie Reading Tests (GMRT-4) reading comprehension subtest |
Promoting Adolescents' Comprehension of Text (PACT) vs. Business as usual |
0 Days |
Non-English Learners;
|
96.85 |
96.65 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Assessment of Social Studies Knowledge (ASK) |
Promoting Adolescents' Comprehension of Text (PACT) vs. Business as usual |
0 Days |
English Learners;
|
19.34 |
16.61 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Assessment of Social Studies Knowledge (ASK) |
Promoting Adolescents' Comprehension of Text (PACT) vs. Business as usual |
0 Days |
Non-English Learners;
|
25.25 |
22.14 |
Yes |
|
||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
27% English language learners -
Female: 49%
Male: 47% -
Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
South, West
-
Race Asian 3% Black 14% Native American 14% Other or unknown 9% Pacific Islander 1% White 60% -
Ethnicity Hispanic 61% Not Hispanic or Latino 39%
Study Details
Setting
The study took place in grade 8 social studies classrooms in seven middle schools. The school were located in three school districts in the southwestern and southeastern United States.
Study sample
There were 1,629 students from 94 classrooms in the study. Of those, 435 students (27 percent of the sample) were English learners or had been reclassified to fluent English proficient status within the previous 2 years. Approximately 49 percent were female, 47 percent were male, and the remaining students did not report gender. Approximately 60 percent of students were White, 14 percent were Black or African American, 14 percent were Native American, 3 percent were Asian, 1 percent were Pacific Islander, and 9 percent were another race or did not specify their race. Sixty-one percent of students were Hispanic or Latino. Almost 10 percent of students were receiving special education services. The percentage of students who qualified for free and reduced price lunch ranged from 49 to 83 percent in the five schools for which that information was available.
Intervention Group
For this study, an existing literacy intervention called Promoting Adolescent Comprehension of Text (PACT) was modified to incorporate several research-based instructional features that have been shown to improve outcomes for English learners in the middle grades. The goal of the intervention was to improve English learners' content acquisition and reading comprehension skills. It was implemented for approximately 20 weeks during regular U.S. history classes and is comprised of three units that incorporate five main components: 1) comprehension canopy, a 15 minute exercise to engage students in reading comprehension activities while integrating new content; 2) essential words, an activity to teach the meaning of concepts connected to the content; 3) knowledge acquisition through text reading using a critical reading routine; 4) comprehension check activities that provide opportunities for text-based discussions and justifications; and 5) knowledge application to apply newly learned content through a problem solving activity. For the first 6-8 weeks, three consecutive units were taught for either 45 minutes daily, or for 90 minutes every other day. For the next 12 weeks, teachers implemented one component (knowledge acquisition through text reading) three times per week for 15 minutes per session.
Comparison Group
The comparison condition was business-as-usual class instruction in grade 8 social studies classes.
Support for implementation
Teachers received a one-day workshop training on the intervention, where trainers modeled each component of the intervention and led hands-on practice with the materials. After the first unit was completed, an additional 3-hour session was held to review the components and areas for improvement in implementation. During the first 8 weeks of the study, teachers received in-class support as needed by an assigned support person. In the following 12 weeks, the assigned support person provided monitoring and out-of-class support to teachers three or four times.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Wanzek, Jeanne; Swanson, Elizabeth; Vaughn, Sharon; Roberts, Greg; Fall, Anna-Mária. (2016). English Learner and Non-English Learner Students with Disabilities: Content Acquisition and Comprehension. Exceptional Children, v82 n4 p428-442.
IES Performance Measure
Review Details
Reviewed: February 2018
- IES Performance Measure (findings for IES Funded Studies (NCER))
- Randomized Controlled Trial
- Meets WWC standards with reservations because it is a randomized controlled trial with low attrition, but the randomization was compromised.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Assessment of Social Studies Knowledge (ASK) |
IES Funded Studies (NCER) vs. Business as usual |
0 Days |
Full sample;
|
24.34 |
20.76 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Assessment of Social Studies Knowledge (ASK) |
IES Funded Studies (NCER) vs. Business as usual |
0 Days |
Subsample of English language learners.;
|
20.07 |
16.61 |
No |
-- | ||
|
Modified Assessment of Social Studies Knowledge and Reading Comprehension (MASK) |
IES Funded Studies (NCER) vs. Business as usual |
0 Days |
Subsample of non English language learners;
|
10.77 |
10.61 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
27% English language learners -
Female: 49%
Male: 47% -
Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
South, West
-
Race Asian 3% Black 14% Native American 14% Pacific Islander 0% White 60% -
Ethnicity Hispanic 61%
Study Details
Setting
The study was conducted in seven schools across three school districts. Three school schools were located in the southwest, two of which are located in an urban district and one suburban predominantly Hispanic district. Four schools were located in one southeastern school district.
Study sample
The initial intervention sample included 51% female, 41% Caucasian, 10% African American, 39% Hispanic, and 8% American Indian, 8% receiving special education, 46% who spoke Spanish at home, and 24% English language learners. The initial comparison group sample included 46% female, 38% Caucasian, 8% African American, 41% Hispanic, and 10% American Indian, 11% receiving special education, 48% who spoke Spanish at home, and 29% English language learners. In the five schools, the percent who qualified for free or reduced-price lunch varied from 49 to 83 percent.
Intervention Group
Promoting Adolescents' Comprehension of Text (PACT), as implemented in this study, is a modified version of the PACT reading comprehension and content acquisition intervention. The intervention was modified to be implemented within social studies classes with EL students. PACT is a curricula featuring three units with five components: 1) comprehension canopy (a 10-15 minute introduction) to engage students in the purpose for reading - this includes a video with follow-up questions; 2) five essential words per unit are introduced after the introduction to teach the meaning of concepts connected with the content; 3) knowledge acquisition through text reading included a critical reading routine lasting 15 minutes, three times a week that covers information text on the topic; 4) team-based learning comprehension check, which includes heterogeneous teams of students and opportunities for both individual and group work twice per each unit, and 5) team-based learning knowledge application where teams engage in problem-solving activities around the topic.
Comparison Group
In the comparison classroom, teachers covered the same social studies content but did not interweave PACT components.
Support for implementation
The research team provided two training sessions and ongoing in-class support. They provided a 1-day professional development workshop that provided an overview, discussed the curriculum and provided opportunities for hands on practice. Each teacher was assigned someone to provide as-needed in-class support.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).