The Open Learning Initiative: Measuring the Effectiveness of the OLI Statistics Course in Accelerating Student Learning
Lovett, Marsha; Meyer, Oded; Thille, Candace (2008). Journal of Interactive Media in Education, 2008. Retrieved from: https://eric.ed.gov/?id=EJ840810
-
examining61Students, gradePS
Practice Guide
Review Details
Reviewed: July 2020
- Practice Guide (findings for Open Learning Initiative (OLI))
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
2007 study - CAOS (RCT) |
Open Learning Initiative (OLI) vs. Business as usual |
0 Days |
Full sample;
|
68.00 |
53.00 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Pennsylvania
Study Details
Setting
The 2007 study which meets standards without reservations takes place in the context of an introduction to statistics course at Carnegie Mellon University.
Study sample
The study does not provide demographic or other characteristics on the students in the study. Students are undergraduates at a 4-year university.
Intervention Group
The intervention condition for the 2007 study involves (a) working with an online learning environment to acquire most of the course content, (b) meeting with an instructor approximately two times a week for 50-minute sessions to ask questions and review more challenging material, and (c) doing all of this at a pace designed to complete the semester’s material in approximately half the time (8 weeks instead of 15). The statistics course is a typical college-level, non-calculus-based introduction to statistics.
Comparison Group
The comparison condition was the traditional teaching of the Introductory Statistics course. The content was the same as that delivered in the intervention condition.
Support for implementation
The technology in the study is the OLI course itself, an interactive online statistics course.
Open Learning Initiative (OLI) Intervention Report - Supporting Postsecondary Success
Review Details
Reviewed: August 2019
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Open Learning Initiative (OLI).
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Comprehensive Assessment of Outcomes in a First Statistics course (CAOS) |
Open Learning Initiative (OLI) vs. Business as usual |
0 Days |
Full sample (2007 RCT);
|
68.00 |
53.00 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Pennsylvania
Study Details
Setting
The 2007 RCT study which meets standards without reservations takes place in an introduction to statistics course at Carnegie Mellon University.
Study sample
The study does not provide demographic or other characteristics on the students in the study other than that they are undergraduates at Carnegie Mellon University.
Intervention Group
The intervention condition for the 2007 study involves (a) working with an online learning environment to acquire most of the course content, (b) meeting with an instructor approximately two times a week for 50-minute sessions to ask questions and review more challenging material, and (c) doing all of this at a pace designed to complete the semester’s material in approximately half the time (8 weeks instead of 15). The statistics course is a typical college-level, non-calculus-based introduction to statistics.
Comparison Group
The comparison condition was the traditional teaching of the Introductory Statistics course. The content was the same as that delivered in the intervention condition.
Support for implementation
The technology in the study is the OLI course itself, an interactive online statistics course.
The Open Learning Initiative: Measuring the Effectiveness of the OLI Statistics Course in Accelerating Student Learning
Review Details
Reviewed: March 2016
- Single Study Review (668 KB) (findings for Open Learning Initiative (OLI))
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Percentage of correct answers on the Comprehensive Assessment of Outcomes in a first Statistics course (CAOS) |
Open Learning Initiative (OLI) vs. Business as usual |
8 Weeks |
2007;
|
68.00 |
53.00 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Pennsylvania
Study Details
Setting
The study was conducted in 2007 at Carnegie Mellon University, Pittsburgh, PA.
Study sample
The students were undergraduates at Carnegie Mellon University; no other information about demographics or grade level was provided.
Intervention Group
The intervention group was comprised of students who volunteered to participate in the OLI course and were randomly selected to receive the intervention. The OLI-Statistics course provided instruction through an online learning environment, two weekly group sessions with an instructor to address the most challenging material, and was paced to complete the introductory statistics curriculum in 8 weeks instead of 15. The group sessions were tailored by the instructor using reports of student performance generated by the online system.
Comparison Group
The comparison group volunteered to participate in the intervention but were not selected. Instead, they received treatment as usual in the form of the regular instructor-led Introductory Statistics course at Carnegie Mellon. This course involved three 50-minute lectures per week plus one 50-minute lab session.
Support for implementation
No information was provided about training and support for implementation of the intervention, which is delivered primarily in an online format. The authors did not mention special training or support for the faculty who conducted the in-person meetings.
Grant Competition
Review Details
Reviewed: February 2016
- Grant Competition (findings for Open Learning Initiative (OLI))
- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Comprehensive Assessment of Outcomes in a First Statistics course (CAOS) |
Open Learning Initiative (OLI) vs. Business as usual |
2007 |
2007 study;
|
0.73 |
0.53 |
No |
-- |
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Pennsylvania
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).