The Struggle to Pass Algebra: Online vs. Face-to-Face Credit Recovery for At-Risk Urban Students
Heppen, Jessica B.; Sorensen, Nicholas; Allensworth, Elaine; Walters, Kirk; Rickles, Jordan; Taylor, Suzanne Stachel; Michelman, Valerie (2017). Journal of Research on Educational Effectiveness, v10 n2 p272-296. Retrieved from: https://eric.ed.gov/?id=EJ1135796
-
examining12,224Students, grades9-10
IES Performance Measure
Review Details
Reviewed: September 2017
- IES Performance Measure (findings for Online Algebra I credit recovery courses)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
End-of-Semester Algebra Exam |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
0 Days |
Full sample;
|
272.97 |
279.62 |
Yes |
|
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Score on the PLAN Algebra subtest |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
0 Days |
Full sample;
|
5.42 |
5.27 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
PLAN Math |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
0 Days |
Full sample;
|
14.16 |
13.94 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Algebra Credit Recovery |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
0 Days |
Full sample;
|
66.00 |
78.00 |
Yes |
|
||
| Show Supplemental Findings | |||||||||
|
On-Track Indicator |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
2 Semesters |
Full sample;
|
28.00 |
25.00 |
No |
-- | ||
|
Credit in Geometry or higher |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
1 Semester |
Full sample;
|
53.00 |
54.00 |
No |
-- | ||
|
Credit in Geometry or higher |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
2 Semesters |
Full sample;
|
47.00 |
48.00 |
No |
-- | ||
|
Cumulative math credits earned |
Online Algebra I credit recovery courses vs. Face-to-Face Algebra credit recovery |
2 Semesters |
Full sample;
|
2.39 |
2.51 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
47% English language learners -
Female: 38%
Male: 62% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Illinois
-
Race Black 33% Other or unknown 2% White 8% -
Ethnicity Hispanic 57% Not Hispanic or Latino 43%
Study Details
Setting
The study was conducted in high schools in the Chicago Public Schools district. Schools that had large numbers of students failing Algebra IB (second-semester Algebra) and were open for summer school were recruited to participate in the study. Seventeen high schools participated in the study. The study took place during 2011 and 2012, with 4 schools participating only in 2011, 2 schools only in 2012, and the remainder of the schools participating in both years.
Study sample
Students participating in the study had failed Algebra IB and generally exhibited poor academic performance in other areas. On average, participating students had failed 4.5 semester courses and had prior math scores that were 0.29 standard deviations below the district average. In the previous year 40% of participating students had been suspended, 5% had moved schools, and the average student had missed 30 school days. Of the total 1,224 student who participated in the study across the two school years, 38 percent were female, 57 percent were Hispanic, 33 percent were African American, 8 percent were White, 2 percent were other races or ethnicities, 86 percent were eligible for free or reduced-price lunch, 12 percent were eligible for special education services, and 47 percent spoke Spanish as their home or native language. In the Algebra I condition 38% of students were female, 12% were special education students, 56% were Latino, and 46% were native Spanish speakers. In the face-to-face condition 37% of students were female, 12% were special education students, 58% were Latino, and 48% were native Spanish speakers.
Intervention Group
The intervention condition is The Aventa Learning (now called FuelEducation) online Algebra I summer course. The online credit recovery course was offered during a three- to four-week summer session and was scheduled for 60 hours. The course took place on site via computers at participating schools and was facilitated by an in-class mentor and taught by an online teacher. The in-class mentors were certified teachers, but were not necessarily licensed to teach mathematics. Aventa hired online teachers who were certified to teach mathematics. The course was organized into five units related to systems of equations, polynomials, quadratics and radicals, rational expressions, and exponentials. Students were encouraged to progress through the units in order, but had flexibility in how they chose to proceed. Students on average completed 2.5 out of the 5 units. Students communicated with the online teachers through back and forth online chats, whiteboard demonstrations, and the learning management system, which often did not require the teacher and student to be online at the same time. Students also had periodic opportunities for real-time communication with teachers and other students using an online platform called Elluminate Live! Students received immediate feedback on practice problems and assessment through the online system.
Comparison Group
The second credit recovery intervention is a face-to-face Algebra I summer course. The face-to-face summer course was a traditional Algebra I credit recovery course. The course was offered during a three- to four-week summer session and was scheduled for 60 hours. Certified math instructors taught each course at the participating schools. The teachers determined the content, sequence, and pacing of the material covered in the face-to-face classes. On average across the face-to-face classes, the study team observed that approximately half of the content focused on second semester Algebra, and half focused on first semester Algebra and pre-Algebra topics. The study team also observed that topics in 28 percent of the face-to-face classes were presented in an order that did not follow a sequence typically seen in a textbook or district-pacing guide. In-person communication occurred between face-to-face instructors and students during class time. Students did not receive immediate feedback on their performance.
Support for implementation
In-class mentors in the online class attended a training session before the summer course began. They received training on the use of the online course, how to monitor student progress, and how to communicate with the teachers of the online classes. The online teachers were given concurrent professional development and were supervised by Aventa.
The Struggle to Pass Algebra: Online vs. Face-to-Face Credit Recovery for At-Risk Urban Students
Review Details
Reviewed: February 2017
- Single Study Review (673 KB) (findings for Online Algebra I credit recovery courses)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
On track for high school graduation (%) |
Online Algebra I credit recovery courses vs. Business as usual |
2 Years |
Full sample;
|
28.00 |
25.00 |
No |
-- | |
|
Recovered Algebra I credit (%) |
Online Algebra I credit recovery courses vs. Business as usual |
1 Semester |
Full sample;
|
66.00 |
78.00 |
Yes |
|
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Score on the PLAN math test |
Online Algebra I credit recovery courses vs. Business as usual |
1 Year |
Full sample;
|
14.16 |
13.94 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Score on the PLAN Algebra subtest |
Online Algebra I credit recovery courses vs. Business as usual |
2 Years |
Full sample;
|
5.42 |
5.27 |
No |
-- | ||
|
Score on the end-of-course posttest |
Online Algebra I credit recovery courses vs. Business as usual |
1 Semester |
Full sample;
|
272.97 |
279.62 |
No |
-- | ||
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Cumulative math credits earned |
Online Algebra I credit recovery courses vs. Business as usual |
2 Years |
Full sample;
|
2.39 |
2.51 |
No |
-- | ||
| Show Supplemental Findings | |||||||||
|
Earned course credit in Geometry or higher (%) |
Online Algebra I credit recovery courses vs. Business as usual |
1 Year |
Full sample;
|
53.00 |
54.00 |
No |
-- | ||
|
Earned course credit in Geometry or higher (%) |
Online Algebra I credit recovery courses vs. Business as usual |
2 Years |
Full sample;
|
47.00 |
48.00 |
No |
-- | ||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 38%
Male: 62% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Midwest
-
Race Black 33% Other or unknown 59% White 8% -
Ethnicity Hispanic 57% Not Hispanic or Latino 43%
Study Details
Setting
The study took place in 17 Chicago Public Schools that participated in the study during the summers of 2011 and 2012. Schools were recruited to participate in the research because they had summer school programs and large numbers of students who failed Algebra I in the second semester of their freshman year.
Study sample
Over the two summers, 1,224 students participated. There were 613 students assigned to the intervention (online) condition, while 611 students were assigned to the comparison (face-to-face) condition. Eleven schools participated in both 2011 and 2012, four participated in 2011 only, and two schools participated in 2012 only. Each school had at least two credit recovery courses: one online, and the other face-to-face. There were 63 school staff who participated as either teachers and/or mentors. Over the two cohorts, there were 34 face-to-face algebra teachers and 30 in-class mentors. Aventa Learning, the online course provider, selected six online teachers for the study, all of whom were certified to teach mathematics. Students in the intervention condition were 38% female and 56% Latino, while the comparison condition was 37% female and 58% Latino. The full sample (i.e., not disaggregated by condition) was 38% female, 57% Hispanic, 33% African American, 8% White, and 2% other races/ethnicities. In the full sample, 86% of the students were eligible for free or reduced-price lunch, 12% were eligible for special education services, and 47% were native Spanish speakers. The proportion of students who passed Algebra IA (first semester of Algebra) was similar across groups (40% in the intervention group and 41% in the comparison group). Only 5% of students in each condition came from census blocks with concentrated poverty.
Intervention Group
Summer Algebra credit recovery courses designed by Aventa Learning were offered to high school students who had failed Algebra I in the second semester of their freshman year. The intervention condition was an online course that focused on typical second-semester Algebra I (designed as a 60-hour course). This included systems of equations, polynomials, quadratics and radicals, rational expressions, and exponentials. The instructional content of the online version was standardized with clear ordering of topics, but flexible in terms of student pacing (in face-to-face classes, teachers have flexibility of content and sequencing, but pacing is generally uniform for the whole class). The online courses were delivered via computer in computer labs at the high schools. Each course was taught by two instructors: one online teacher and one in-class mentor. The course took place over one or both of two 3- to 4-week summer sessions at each participating school.
Comparison Group
The comparison condition received traditional, face-to-face Algebra I instruction by a certified mathematics instructor. The content tended to include both second semester Algebra I topics, as well as pre-Algebra and first semester Algebra I topics; about 50% of the content was from the second semester Algebra I course, while the remaining 50% was derived from first semester Algebra and pre-Algebra courses. The course took place over one or both of two 3- to 4-week summer sessions at each participating school. Face-to-face courses had one instructor and were delivered in traditional classrooms. The face-to-face courses used teacher-created and published materials, including textbooks. Teachers had flexibility of content and sequencing, but pacing was generally uniform for the whole class. In-class mentors provided feedback and communication on students’ progress.
Support for implementation
The in-class mentors and the face-to-face teachers were paid their regular teaching rates, and the online teachers were paid by the number of students enrolled in the course, which ended up being slightly higher than face-to-face teachers. In-class mentors received training on how to use the online course system, how to monitor student progress, and how to communicate with online teachers. The online teachers received ongoing professional development and support from Aventa. The face-to-face teachers received traditional supports from their schools and district for teaching summer Algebra classes.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Heppen, J., Allensworth, E., Sorensen, N., Rickles, J., Walters, K., Taylor, S., ... Clements, P. (2016). Getting back on track: Comparing the effects of online and face-to-face credit recovery in Algebra I (Research Brief 1). Retrieved from http://www.air.org/
-
Rickles, J., Heppen, H., Taylor, S., Allensworth, E., Michelman, V., Sorensen, N.,...Clements, P. (2016). Getting back on track: Who need to recover algebra credit after ninth grade? (Research Brief 3). Retrieved from http://www.air.org/
-
Taylor, S., Clements, P., Heppen, J., Rickles, J., Sorensen, N., Walters, K., ... Michelman, V. (2016). Getting back on track: The role of in-person instructional support for students taking online credit recovery (Research Brief 2). Retrieved from http://www.air.org/
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).