Co-Principal Investigator(s): Best, Stephen; Hanita, Makoto

Partners: Education Development Center; Michigan Department of Education, and Michigan Virtual Schools

Purpose: The Education Development Center (EDC), Michigan Department of Education (MDE), and Michigan Virtual Schools (MVS) formed a partnership to evaluate the impact of a short online orientation course (called Strategies for Online Success or SOS) on students' engagement in and completion of full online courses offered by MVS. Using an experimental design, the project team examined the impact on student engagement and course completion of being assigned the SOS course for first-time online students. The study also examined student take up of the SOS course and the relationship of take up to their outcomes in their online courses (both engagement and completion).

Project Activities: This partnership completed a randomized controlled trial to evaluate the impacts of the SOS online orientation course on student engagement and online course completion. MVS implemented the SOS module. High school students registering for a MVS online course were randomly assigned to either the voluntary SOS orientation course or to a comparison group that does not have access to the orientation course. MVS and MDE provided course data and student administrative data, respectively; and EDC performed the analysis. The partners worked together to disseminate the results.

Key Outcomes: The key findings are reported in Zweig et al. (2022) and summarized below:

• There were no significant differences in student engagement or course completion between students in the treatment and control groups. For course completion status, there was no significant difference in the relative risk of completing the course as opposed to dropping the course between students assigned the treatment and those that received BAU supports (relative risk ratio =1.08). Further, there was no evidence of an impact of SOS on the relative risk of dropping during the grace period as opposed to not completing the course.
• Of the exploratory analyses, the only variable that had a moderating effect was the indicator for whether a student enrolled on time. For students who enrolled on time, the relative risk ratio (RRR) for dropping during the grace period as opposed to not completing the course was positive, at 1.76; for late enrollers, this RRR was negative, at 0.72. This difference was significant, which suggests that students who enrolled on time, and potentially had the opportunity to take SOS prior to starting their course, were more likely to drop during the grace period than to drop out after the grace period or fail the course. The reverse was true for those that enrolled late.

Structured Abstract

Setting: The study was implemented in Michigan for students enrolled in online courses with Michigan Virtual.

Sample: The sample included high school students who did not have previous online learning experience, who enrolled in an online course with Michigan Virtual in fall 2018, and who could be matched to administrative and assessment data from the Michigan Department of Education.

Intervention: Strategies for Online Success (SOS) is an asynchronous orientation course. SOS prepares students for the transition from taking courses in-person to taking them online. It includes five components: a pre-assessment, three interactive content modules, and a post-assessment. The three modules were Online learning basics, Skills for online learning, and Online learning technology. The modules include components such as card sorts, self-checks, online teacher videos, and downloadable resources. SOS was estimated to take approximately 90—120 minutes to complete.

Research Design and Methods: This study used a student-level randomized controlled trial design to address the research questions. In fall 2018, students who enrolled in online courses through Michigan Virtual were randomized into treatment and business-as-usual (BAU) groups daily during the enrollment period. The final sample included 1,781 students who were enrolled in a Michigan Virtual course and could be matched to administrative and assessment data from the Michigan Department of Education: 910 students were in the treatment group (51 percent) and 871 students were in the BAU group (49 percent).

Control Condition: Students in the BAU group were not enrolled in SOS and had access to the typical supports for students. Students in both the BAU and treatment groups had an introductory unit in each of their courses. Unlike SOS, the introductory unit focused on the operational aspects of using the learning management system. The BAU group could also access the previous orientation materials on the Michigan Virtual website developed in 2000.

Key Measures: The following outcomes measures were used in this study:

• Course completion status: A three category variable (1) the student dropped the course during the grace period, (2) the student did not complete the course, and (3) the student completed the course.
• Student engagement: Total number of logins and a dichotomous variable for whether or not the student logged into the course every week.

Data Analytic Strategy: A multinomial regression model was estimated because the main outcome, course completion status, consisted of three mutually exclusive and exhaustive categories. The standard errors were clustered at the section level. Exploratory analyses examined the following potential moderator variables: (1) enrollment in core courses; (2) on-time enrollment; (3) prior achievement and (3) Grade 12. Covariates included 8th grade scaled scores in mathematics and English Language art, prior year attendance rate, and enrollment into core courses as covariates. A multilevel logistic regression was also used with a binary indicator of completion status (1=yes, 0=no) as a sensitivity analysis. A multi-level logistic regression was also used for the indicator of whether the student logged into the course every week and a fixed effects model was used when the outcome was total number of logins.

PRODUCTS AND PUBLICATIONS

ERIC Citations: Find available citations in ERIC for this award here.

Select Publications:

Zweig, J., Hanita, M., Stafford, E., & Khanani, N. (2022). Impact of an Orientation on Online Students' Course Outcomes. 54 (5), 655–678, DOI: 10.1080/15391523.2021.1911007.

Zweig, J., Stafford, E., & Hanita, M. (2022). Enrollment Timing in Supplementary Online Courses: Do Students Who Enroll On-Time Have Better Course Outcomes?. Journal of Online Learning Research, 8 (2), 163–180.