The Impact of Achieve3000 on Elementary Literacy Outcomes: Randomized Control Trial Evidence, 2013-14 to 2014-15. Eye on Evaluation. DRA Report No. 16.02
Hill, Darryl V.; Lenard, Matthew A. (2016). Wake County Public School System. Retrieved from: https://eric.ed.gov/?id=ED581879
-
examining6,307Students, grades2-5
Publication
Review Details
Reviewed: July 2024
- Publication (findings for Achieve3000®)
- The WWC did not review the research because the WWC does not currently have standards to assess the quality of the analysis.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
| Show Supplemental Findings | |||||||||
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
0 Days |
Year 2, Grade: 2, 3, 4, 5 sample;
|
577.94 |
520.72 |
Yes |
|
||
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
0 Days |
Year 1, Grade: 2, 3, 4, 5 sample;
|
550.13 |
562.00 |
Yes |
|
||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
North Carolina End-of-Grade Reading Comprehension |
Achieve3000® vs. Business as usual |
0 Days |
Year 1, Grade 4, 5 sample;
|
989.43 |
991.06 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
North Carolina End-of-Grade Reading Comprehension |
Achieve3000® vs. Business as usual |
0 Days |
Year 2, Grade 4, 5 sample;
|
991.77 |
985.11 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
0 Days |
Year 1, Grade 2, 3 sample;
|
674.88 |
656.43 |
Yes |
|
||
| Show Supplemental Findings | |||||||||
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
0 Days |
Year 2, Grade 2, 3 sample;
|
663.89 |
650.85 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
9% English language learners -
Female: 49%
Male: 51% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
North Carolina
-
Race Black 26% Other or unknown 74% -
Ethnicity Hispanic 19% Not Hispanic or Latino 81% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 36% No FRPL 64%
Study Details
Setting
The study was conducted in 32 elementary schools in the Wake County Public School System (WCPSS) in Raleigh, North Carolina. As a countywide district, WCPSS has schools representing suburban, urban, and rural areas.
Study sample
No demographic data were available on the study sample; however, in the 32 participating schools, the student population was 26 percent Black and race was not reported for 74 percent of students. Moreover, 19 percent of students were Hispanic, 49 percent were female, 9 percent were English learners, 12 percent were in special education, and 36 percent were economically disadvantaged.
Intervention Group
Achieve3000 is a technology-based early literacy program that uses individual students’ Lexile scores to deliver differentiated non-fiction reading passages. Achieve3000® was implemented twice weekly for 30-minute sessions in order to achieve an intended annual goal of 80 completed activities On initial use, students took a 30-minute test to measure their baseline reading achievement. For each lesson, students followed Achieve3000’s five-step literacy routine. Across both years of the study, about 8 percent of intervention students completed at least 80 lessons (i.e., the developer’s recommended dosage), about 20 percent used 40-79 lessons, about 50 percent used 1-39 lessons, and 22 percent of students completed no activities. The Achieve3000 intervention was used to supplement a standard core reading curriculum; however, the authors did not identify which core curriculum was used.
Comparison Group
The comparison condition was business-as-usual reading instruction. Classrooms in comparison schools did not receive a supplemental curriculum.
Support for implementation
The study included professional development to train teachers, consisting of two 2.5-hour large-group training sessions and one 1-hoursmall-group session. Teachers were able to obtain follow-up help if needed.
Achieve3000® Intervention Report - Adolescent Literacy
Review Details
Reviewed: February 2018
- Randomized Controlled Trial
- Meets WWC standards with reservations because it is a compromised randomized controlled trial, but the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Achieve3000®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
North Carolina End-of-Grade Test |
Achieve3000® vs. Business as usual |
1 Year |
2015 sample, grades 4-5;
|
991.77 |
985.11 |
No |
-- | ||
|
North Carolina End-of-Grade Test |
Achieve3000® vs. Business as usual |
1 Year |
2014 sample, grades 4-5;
|
989.43 |
991.06 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
North Carolina End-of-Grade Test |
Achieve3000® vs. Business as usual |
1 Year |
2015 AIG sample, grades 4-5;
|
1221.25 |
1244.40 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
1 Year |
2015 sample, grades 2-5;
|
577.94 |
520.72 |
Yes |
|
||
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
1 Year |
2014 sample, grades 2-5;
|
550.13 |
562.00 |
Yes |
|
||
| Show Supplemental Findings | |||||||||
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
1 Year |
2014 AIG sample, grades 2-5;
|
985.28 |
999.20 |
No |
-- | ||
|
LevelSet Lexile score |
Achieve3000® vs. Business as usual |
1 Year |
2015 AIG sample, grades 4-5;
|
983.61 |
952.40 |
Yes |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
9% English language learners -
Female: 49%
Male: 51% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
North Carolina
-
Race Black 26% White 51% -
Ethnicity Hispanic 19% Not Hispanic or Latino 82%
Study Details
Setting
The study was conducted in the Wake County Public School System (WCPSS) in Raleigh, North Carolina. As a countrywide district, WCPSS has schools representing suburban, urban, and rural areas.
Study sample
The authors used a cluster randomized controlled trial design to study the effects of KidBiz3000® on reading achievement for students in grades 2–5. The study took place over two school years (from 2013–14 to 2014–15) in 32 elementary schools. In summer of 2013, the authors matched pairs of schools on the basis of their average 2013 end-of-grade (EOG) reading composite scores, and then from within each matched pair, randomly assigned one school to the intervention group and one school to the comparison group. In both study years, the same 16 KidBiz3000® schools and 16 comparison schools participated in the study. The WWC considers random assignment jeopardized because the analytic sample included students who enrolled in study schools after random assignment. For the general literacy domain, the 2-year combined analysis sample included 22,583 students in grades 2–5 in 24 schools: 11,802 students were in the Achieve3000® group, and 10,781 students were in the comparison group. For the comprehension domain, the 2-year combined analysis sample included 12,542 students in grades 4–5 in 32 schools: 6,585 students were in the Achieve3000® group, and 5,957 students were in the comparison group. Because some of the same students were analyzed in both years of the study, these reported sample sizes count some individual students more than once. No demographic data were available on the analytic study sample; however, in the 32 participating schools, the student population was 51% White, 26% African-American, and 19% Hispanic. Moreover, 12% of students had disabilities, 9% of students in study schools were English learners, and 7% of students were academically and intellectually gifted (AIG). Approximately, one-third of the district's students were certified for free or reduced-price lunch.
Intervention Group
KidBiz3000® was implemented in 30-minute sessions, two times per week over a school year. On initial use, students took a 30-minute test to measure their baseline reading achievement. For each lesson, students followed Achieve3000®’s five-step literacy routine. Across both years of the study, about 8% of intervention students completed at least 80 lessons (i.e., the developer’s recommended dosage), about 21% used 40-79 lessons, about 50% used 1-39 lessons, and 22% of students completed no activities. The Achieve3000® intervention was used to supplement a standard core reading curriculum; however, the authors did not identify which core curriculum was used.
Comparison Group
The comparison condition was business-as-usual reading instruction. Classrooms in comparison schools did not receive a supplemental curriculum.
Support for implementation
The study included professional development to train teachers, consisting of two 2.5-hour large group training sessions, and one 1-hour small group session. Teachers were able to obtain follow-up help if needed.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Hill, Darryl V.; Lenard, Matthew A.; Page, Lindsay Coleman. (2016). The Impact of Achieve3000 on Elementary Literacy Outcomes: Evidence from a Two-Year Randomized Control Trial. Society for Research on Educational Effectiveness.
Achieve3000® Intervention Report - Beginning Reading
Review Details
Reviewed: February 2018
- Randomized Controlled Trial
- Meets WWC standards with reservations because it is a compromised randomized controlled trial, but the analytic intervention and comparison groups satisfy the baseline equivalence requirement.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Achieve3000®.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
||
|---|---|---|---|---|---|---|---|---|---|
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
1 Year |
2014 sample, grades 2-3;
|
674.88 |
656.43 |
No |
-- | ||
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
1 Year |
2015 sample, grades 2-3;
|
663.89 |
650.85 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
1 Year |
2014 AIG sample, grades 2-3;
|
1049.40 |
1066.30 |
No |
-- | ||
|
DIBELS Oral Reading Fluency |
Achieve3000® vs. Business as usual |
1 Year |
2015 AIG sample, grades 2-3;
|
1063.60 |
1107.20 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
9% English language learners -
Female: 49%
Male: 51% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
North Carolina
-
Race Black 26% White 51% -
Ethnicity Hispanic 19% Not Hispanic or Latino 82%
Study Details
Setting
The study was conducted in the Wake County Public School System (WCPSS) in Raleigh, North Carolina. As a countywide district, WCPSS has schools representing suburban, urban, and rural areas.
Study sample
The authors used a cluster randomized controlled trial design to study the effects of KidBiz3000® on the reading achievement of students in grades 2-5 (only analyses for grades 2-3 were eligible for review under the Beginning Reading Protocol). The study took place over two school years (from 2013–14 to 2014–15) in 32 elementary schools. In summer of 2013, the authors matched pairs of schools based on their average 2013 End-of-Grade (EOG) reading composite scores, and then from within each matched pair, randomly assigned one school to the intervention group and one school to the comparison group. In both study years, the same 16 KidBiz3000® schools and 16 comparison schools participated in the study. The WWC considers random assignment to be jeopardized because the analytic sample included students who enrolled in study schools after random assignment. The 2-year combined analysis sample included 14,493 students: 7,540 students were in the Achieve3000® group, and 6,953 students were in the comparison group. The reported sample sizes count some students more than once because some second grade students in 2013–14 may also appear as third-grade students in 2014–15. No demographic data were available on the study sample in grades 2–3; however, in the 32 participating schools, the student population was 51% White, 26% African-American and 19% Hispanic. Moreover, 12% of students had disabilities, 9% of students in study schools were English learners, and 7% of students were academically and intellectually gifted (AIG). Approximately, one-third of the district's students were certified for free or reduced-price lunch.
Intervention Group
KidBiz3000® was implemented in 30-minute sessions, two times per week over a school year. On initial use, students took a 30-minute test to measure their baseline reading achievement. For each lesson, students followed Achieve3000®’s five-step literacy routine. Across both years of the study, about 8% of intervention students completed at least 80 lessons (i.e., the developer’s recommended dosage), about 21% used 40-79 lessons, about 50% used 1-39 lessons, and 22% of students completed no activities. The Achieve3000® intervention was used to supplement a standard core reading curriculum; however, the authors did not identify which core curriculum was used.
Comparison Group
The comparison condition was business-as-usual reading instruction. Classrooms in comparison schools did not receive a supplemental curriculum.
Support for implementation
The study included professional development to train teachers, consisting of two 2.5-hour large-group training sessions and one 1-hour small-group session. Teachers were able to obtain follow-up help if needed.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Hill, Darryl V.; Lenard, Matthew A.; Page, Lindsay Coleman. (2016). The Impact of Achieve3000 on Elementary Literacy Outcomes: Evidence from a Two-Year Randomized Control Trial. Society for Research on Educational Effectiveness.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, statistical significance, and sample size of the findings within a domain, the WWC assigns effectiveness ratings as one of the following: Tier 1 (strong evidence), Tier 2 (moderate evidence), Tier 3 (promising evidence), uncertain effects, and negative effects. For more detail, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).