Mobilizing Developmental Education: The Causal Effect of Mobile App Courseware on the College Outcomes of Developmental Education Students
Giani, Matt S.; Martin, Allison (2021). Educational Evaluation and Policy Analysis, v43 n4 p668-687 Dec 2021. Retrieved from: https://eric.ed.gov/?id=EJ1317418
-
examining2,122Students, gradePS
Single Study Review
Review Details
Reviewed: September 2024
- Single Study Review (findings for Mobile app courseware)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Received an Associate's degree |
Mobile app courseware vs. Business as usual |
0 Days |
Full Sample;
|
8.60 |
7.80 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Received a certificate |
Mobile app courseware vs. Business as usual |
0 Days |
Full Sample;
|
13.90 |
13.60 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Next-Semester GPA |
Mobile app courseware vs. Business as usual |
1 Semester |
Full sample;
|
1.97 |
1.87 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Credit earned next semester |
Mobile app courseware vs. Business as usual |
1 Semester |
Full Sample;
|
5.62 |
5.15 |
No |
-- | |
|
College persistence - enrolled in higher education for another term |
Mobile app courseware vs. Business as usual |
1 Semester |
Full Sample;
|
69.50 |
67.00 |
No |
-- | |
|
Next Semester Course Pass Rate |
Mobile app courseware vs. Business as usual |
1 Semester |
Full sample;
|
68.60 |
66.00 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Developmental education passing |
Mobile app courseware vs. Business as usual |
0 Days |
Full sample;
|
56.80 |
51.30 |
Yes |
|
|
|
Developmental education grade |
Mobile app courseware vs. Business as usual |
0 Days |
Full sample;
|
1.81 |
1.67 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 64%
Male: 36% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Louisiana
-
Race Black 40% Other or unknown 14% White 46% -
Ethnicity Hispanic 6% Not Hispanic or Latino 94%
Study Details
Setting
The study was conducted at the Bossier Parish Community College (BPCC) and Northwestern State University (NSU) in the mid-sized community of Shreveport, a metropolitan area in northwest Louisiana.
Study sample
Almost two-thirds (64%) percent of the study sample were female and 65% received Pell grants. The following percentages describe the population students at BPCC: 46% White non-Hispanic, 40% Black, and 8% for all other racial/ethnic groups; 6% of the students were Hispanic/Latinx. However, the authors noted that more than 50% of students reported “other” for their race/ethnicity.
Intervention Group
The Open Campus™ apps consist of free and fully online modules of developmental education content to assist undergraduates in brushing up on core academic topics before taking developmental education placement exams. Undergraduates enrolled in developmental education courses used the Open Campus™ app from spring 2017 through spring 2019 as a supplemental instructional resource in many classrooms. After downloading the app and creating an account, students were presented the option to enroll in English 99 (Developmental Writing), Math 98 (Beginning Algebra I), or Math 99 (Beginning Algebra II). After enrolling, students completed a 50-item quiz within the app that diagnoses topic areas for which they needed the greatest assistance and were shown the percentage of correct answers followed by instructions on which app modules to focus on. Each module consisted of three to five-minute video lectures where instructors explained the concepts and a 10-item quiz with additional questions related to the specific topic area. For the module quizzes, students were shown each question, whether their answer was correct or incorrect, and for each incorrect answer, they were pointed to the specific module to review to learn more about that topic. Students could attempt the quizzes as many times as needed, but only their highest grade was kept in their profile. The mobile apps contained supplemental features to increase engagement, such as digital badges based on student progress within the modules, student head-to-head quiz challenges with a leader board, and email nudges to inactive students in the app for a prolonged period. The email notifications are an opt-in feature, and students can unsubscribe from the notification.
Comparison Group
Students in the comparison condition were enrolled in similar developmental education courses as students in the intervention condition but did not have access to the Open Campus™ mobile application. However, these students did have access to all other campus resources available to students in the developmental education track.
Support for implementation
Developmental education faculty members were offered a professional development workshop immediately prior to the spring 2017 semester for cohort 1 and fall 2017 semester for cohort 2. The workshop provided instructors with hands-on time using the apps, with additional examples of how they could incorporate the apps into their instruction, with information on which course sections would be assigned to the intervention and comparison groups to allow for additional preparation time if their section was selected for implementation.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).