Evaluation of LANGUAGE! in Miami-Dade County Public Schools: Final report.
Zmach, C. C., Chan, T., Salinger, T., Chinen, M. H., Tanenbaum, C. T., & Taylor, T. S. (2009). Washington, DC: American Institutes for Research.
-
examining640Students, grades9-10
LANGUAGE!® Intervention Report - Adolescent Literacy
Review Details
Reviewed: February 2013
- Quasi-Experimental Design
- Meets WWC standards with reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for LANGUAGE!®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Florida Comprehensive Assessment Test (FCAT): Reading |
LANGUAGE!® vs. Business as usual |
Spring |
Grades 9-10;
|
N/A |
N/A |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Test of Silent Contextual Reading Fluency (TOSCRF) |
LANGUAGE!® vs. Business as usual |
Fall, Spring |
Grades 9-10;
|
N/A |
N/A |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
10% English language learners -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Florida
-
Race Black 26% Other or unknown 2% White 7% -
Ethnicity Hispanic 65% Not Hispanic or Latino 35%
Study Details
Setting
The student analysis sample included in this report is drawn from 18 high schools eligible for Title I funding in the Miami–Dade County Public School district.
Study sample
Analysis sample. The ninth- and tenth-grade samples are the only samples which meet WWC evidence standards with reservations. Therefore, they are presented in this report. After dropping the unmatched students, students with missing information, and students who were no longer enrolled in an IR+ class in the same school in spring 2008, the TOSCRF and FCAT analytic high school samples included 640 students and 632 students, respectively. As no information on the extent of overlap between the two analytic samples was provided in the study, the WWC review process treats them as two distinct samples. The TOSCRF student sample included 320 students (190 ninth graders and 130 tenth graders) who used the LANGUAGE!® curriculum and 320 students (190 ninth graders and 130 tenth graders) who used the comparison curriculum in the IR+ classes. The FCAT sample included 316 students (194 ninth graders and 122 tenth graders) who used the LANGUAGE!® curriculum and 316 students (194 ninth graders and 122 tenth graders) who used the comparison curriculum in the IR+ classes. Characteristics of district and study schools. In 2006, 68% of ninth graders and 73% of tenth graders in the district scored below proficient (Level 2 or below) on the FCAT. During the 2005–06 school year: • between 1,547 and 4,509 students attended the study high schools. • the percentage of students in study high schools that were eligible for free or reducedprice lunch ranged from 40% to 68%. • Black and Hispanic students represented between 77% and 99% of the student population in study high schools. Characteristics of the TOSCRF student sample. Among the TOSCRF analytic sample: • Forty-seven percent of ninth graders and 40% of tenth graders were female. • Hispanic students represented 52% of ninth graders and 42% of tenth graders, while Black students represented 43% of ninth graders and 55% of tenth graders. • About 68% of ninth graders and 59% of tenth graders were eligible for free or reduced-price lunch. • Thirty-one percent of ninth graders and 33% of tenth graders were classified as receiving special education services. Characteristics of the FCAT student sample. The demographic characteristics of the FCAT sample were similar to the TOSCRF sample. Among the FCAT student sample: • Thirty-six percent of ninth graders and 48% of tenth graders were female. • Hispanic students represented 52% of ninth graders and 39% of tenth graders, while Black students represented 45% of ninth graders and 56% of tenth graders. • About 66% of ninth graders and 58% of tenth graders were eligible for free or reduced-price lunch. • Thirty-six percent of ninth graders and 32% of tenth graders were classified as receiving special education services.
Intervention Group
The intervention was delivered during the daily IR+ class, which typically lasted 90 minutes. The lessons were administered by the teacher to the whole classroom with some days set aside for differentiated instruction throughout the school year. Schools received a pacing guide designed to facilitate the completion of two book levels, out of six levels (book levels A-F described under program details), during the year. The intervention lasted a full academic year. The study team rated teachers on the fidelity of implementation of the curriculum. Fiftyfour percent of teachers received a medium fidelity rating, and 46% of teachers received a low fidelity rating.
Comparison Group
All comparison classrooms used the same commercially published curriculum in their daily IR+ class, which typically lasted 90 minutes (the study authors did not provide the name of the curriculum). The curriculum focused on strengthening reading and writing skills and developing vocabulary. Typically, 20 minutes of class time was spent on whole-group, direct instruction; 60 minutes was spent on small-group rotations; and 10 minutes was spent on whole-group, wrap-up instruction. The small-group rotations took the form of small-group direct instruction, technology-based individualized instruction, and modeled and independent reading.
Outcome descriptions
The outcomes are gain scores for the TOSCRF and FCAT reading scores, i.e., gains in the TOSCRF and FCAT reading scores from the prior administration of the test to the current test. The TOSCRF was administered to the participating classes by study staff in October and then again seven months later. The FCAT is a state-wide assessment given each spring. For a more detailed description of these outcome measures, see Appendix B.
Support for implementation
Sopris West, the program publisher at the time of the study, was contracted to provide professional development to support teachers’ implementation of the LANGUAGE!® curriculum. The intervention teachers attended a two-day training session before the school year. During the school year, teachers, coaches, and mentors received school visits from LANGUAGE!® trainers and National Trainers, who conducted classroom observations, provided individual coaching and professional development for teachers, modeled lessons, and held question-and-answer sessions. Coaches and school administrators received a half-day of initial training and a day of training in the fall and spring. The comparison teachers received the usual professional development services provided by their schools.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).