A Multisite Randomized Study of an Online Learning Approach to High School Credit Recovery: Effects on Student Experiences and Proximal Outcomes
Jordan Rickles; Margaret Clements; Iliana Brodziak de los Reyes; Mark Lachowicz; Shuqiong Lin; Jessica Heppen (2023). Grantee Submission. Retrieved from: https://eric.ed.gov/?id=ED647309
-
examining1,683Students, grade9
Single Study Review
Review Details
Reviewed: November 2024
- Single Study Review (findings for Online credit recovery - Rickles et al. (2023))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
|---|---|---|---|---|---|---|---|
|
PSAT: Reading |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
1 Semester |
English 9 students;
|
-0.57 |
-0.61 |
No |
-- |
|
PSAT: Math |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
1 Semester |
Algebra I students;
|
-0.54 |
-0.46 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
|---|---|---|---|---|---|---|---|
|
Course recovery rate |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
Algebra I students;
|
62.50 |
71.40 |
No |
-- |
|
Course recovery rate |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
English 9 students;
|
51.00 |
67.40 |
Yes |
|
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
|---|---|---|---|---|---|---|---|
|
Student experiences - Emotional engagement (Rickles et al. 2023) |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
English 9 students;
|
0.06 |
0.01 |
No |
-- |
|
Student experiences - Emotional engagement (Rickles et al. 2023) |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
Algebra I students;
|
-0.15 |
0.06 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
|---|---|---|---|---|---|---|---|
|
Student experiences - Behavioral engagement (Rickles et al. 2023) |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
English 9 students;
|
0.03 |
0.01 |
No |
-- |
|
Student experiences - Behavioral engagement (Rickles et al. 2023) |
Online credit recovery - Rickles et al. (2023) vs. Business as usual |
0 Days |
Algebra I students;
|
-0.08 |
0.03 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
17% English language learners -
Female: 38%
Male: 62% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Black 9% Other or unknown 91% -
Ethnicity Hispanic 83% Other or unknown 17% -
Eligible for Free and Reduced Price Lunch Free or reduced price lunch (FRPL) 86% No FRPL 14%
Study Details
Setting
The study took place in 24 high schools in Los Angeles, California during the summer term of two consecutive school years.
Study sample
The study of an online credit recovery initiative was a non-compromised individual-level RCT that included multiple sites. The researchers randomly assigned students who had failed either Algebra 1 or ninth grade English either to a summer credit recovery class that used an online curriculum or to a traditional in-person summer credit recovery course. A total of 1,683 students in grade 9 were included in the study. This included 613 students in 28 Algebra 1 classes across 13 high schools and 1,124 students in 70 English 9 classes across 19 high schools. Fifty-four students were in both the Algebra 1 and English 9 classes, for a total of 1,737 observations. For these 54 students, the authors randomly selected which class to include in the sample such that the number of students reflect the unique student counts within a subject. Participants had to meet five eligibility criteria. First, students had to have entered the ninth grade in either the 2017/18 or 2018/19 school years. Second, students had to have enrolled in a high school within the district for the spring 2018 or spring 2019 semesters. Third, students had to have failed Algebra I or at least one semester of 9th grade English. Fourth, students had to be enrolled in one of the credit recovery courses that were part of this study (ninth grade English for students who entered ninth grade in 2017/18; Algebra I or ninth grade English for students who entered ninth grade in 2018/19). Fifth, students could not be classified below a specified threshold of English language development. Approximately 38% of the students were female, 9% were Black, and 91% had an unreported or other race. Approximately 83% were Hispanic and 17% had an unreported or unknown ethnicity. Approximately 86% of students were eligible for free or reduced-price lunch, and 17% were English learners.
Intervention Group
The Online Credit Recovery Model is an intervention program that delivers course recovery via online instruction to meet the needs of academically underserved students. The primary course content for this program was provided online, with a classroom teacher available to supplement the online material with additional instruction. The intervention was offered to individual students who had failed either Algebra 1 or at least one semester of their ninth-grade English class. Students in the intervention group received course content online which was supplemented by their school providing a subject-appropriate, credentialed in-class teacher. Students in the intervention condition met for 2.5 hours each day in a standard classroom during the district’s five-week summer session in addition to receiving the online course content.
Comparison Group
Students in the comparison group received a business-as-usual in-person course recovery summer program. Students in the comparison group met for 2.5 hours each day in a standard classroom during the district’s five-week summer session, receiving no additional online course content.
Support for implementation
The study did not describe whether any support or training were offered to the providers of the intervention.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, statistical significance, and sample size of the findings within a domain, the WWC assigns effectiveness ratings as one of the following: Tier 1 (strong evidence), Tier 2 (moderate evidence), Tier 3 (promising evidence), uncertain effects, and negative effects. For more detail, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).