Co-Principal Investigator: Hauk, Shandy

Purpose: This project tested if and how using Khan Academy web-based activities in regular classroom instruction improved community college student learning of algebra. Many students arrive at community college needing opportunities to learn or strengthen mathematics skills in order to complete a college-level mathematics class. To increase student success, some community colleges have turned to Khan Academy's free, online, mathematics resources for students and instructors. This project investigated the impact on student success of class use of Khan Academy's Algebra Basics Mission. The Mission included instructional supports like mind-set lessons, demonstration videos, as well as mathematics problems for students to complete with immediate feedback on answers; the Mission adaptively selected problems to offer to students based on student strengths and errors.

Project Activities: The researchers conducted a small pilot study in the spring of 2015 and then a multi-site cluster-randomized controlled trial study across the 2015-16 and 2016–17 academic years to test the impact of Khan Academy as used in blended classrooms in California community college algebra courses. In addition to documenting use of Khan Academy and student learning through a common test, the researchers also collected data to help identify the factors that contribute to higher quality and more effective implementations of Khan Academy, including teacher preparation, student characteristics, and course structure.

Key Findings: The main findings of this study are as follows:

• Inclusion of Khan Academy as part of the instructional materials correlated to increases in students' post-test scores (Hauk, Matlen, & Thomas, 2017)
• The greatest student gains were in the classes of cautiously optimistic instructors, namely those who voiced concerns about using the approach but still engaged with it (Hauk & Matlen, 2017).

Products: This project collected data on the effect of using Khan Academy resources in postsecondary algebra education and it produced peer-reviewed publications and presentations.

Structured Abstract

Setting: The research took place in 20 of the 106 community colleges throughout California that offered semester-long algebra courses in the years of the study. Of these, 11 were designated as minority serving institutions. Also, 4 of the schools served rural areas and 6 served urban areas.

Sample: The analytic sample consisted of 588 students of 34 instructors across 20 colleges. Enrollment in the 20 colleges, according to federal racial/ethnic categories, was representative of the statewide enrollment and included approximately 44% Hispanic, 29% White, 13% Asian, 5% African American, 3% Filipino, 1% Native American/Pacific Islander, and 4% multi-ethnicity students.

Intervention: Khan Academy is a free, internet-based learning environment that includes instructional videos, adaptive problem sets, and tools for instructors to provide individualized recommendations and assignments to students. When used in the study's community colleges, Khan Academy was part of a blended classroom: a classroom that contained traditional elements (e.g., in-person class meetings) and online resources. While each instructor selected the exact balance, there were three types of blended classrooms in the study, each used by about one-third of the classes: (a) Khan Academy was a main component of instruction (e.g., Khan Academy assignments were a primary form of practice and assessment for as much as 20% of the course grade), (b) Khan Academy use was a supplement to instruction (i.e., most classwork and homework was completed on paper or through a text-book digital resource and teachers additionally assigned the Khan Academy Algebra Basics Mission to the entire class or to individuals), (b) computer lab rotations using Khan Academy (i.e., where students rotated from teacher-led classroom instruction to computer-assisted laboratory learning either on a schedule or as needed).

Research Design and Methods: The study team contacted faculty and administrators in each of the California community college mathematics departments at the start of the study. Instructors participated for two semesters. In the first semester, treatment instructors familiarized themselves with implementing the Algebra Basics Mission with their local algebra curriculum. In the second semester, these teachers implemented Khan academy. The data from the second semester was Each Khan Academy Algebra Basics Mission session took 30 to 70 minutes of instructor-participant time (every session had embedded tasks and quizzes related to preparing to teach with the Khan Academy tools). These web-based sessions included presentation of the five main developer intentions regarding use of a Mission: give assignments to be completed in Khan Academy at least weekly, make mastery of the Algebra Basics Mission 20% of the course grade, use three Khan Academy-provided mindset lessons in-class early in the semester, review and respond to student progress reports (a monitoring tool in the Khan Academy system), and encourage students through recommendations beyond the mastery challenges (an assigning tool in the Khan Academy system). Treatment instructors repeated implementations in the spring semester (the efficacy phase), where the data for evaluation of impact were collected.

Control Condition: Control group students received "business-as-usual" instruction in community college algebra courses. Control group classes did not use Khan Academy but might use other technology tools (e.g., online homework connected to the course textbook).

Key Measures: The primary outcomes of interest were students' course completion rates and mathematics achievement as measured by performance on the Mathematics Diagnostic Testing Project (MDTP) elementary algebra assessment Students were also offered the opportunity to complete surveys about their background and experiences with mathematics. Data collected from instructors included assessments of their professional knowledge and beliefs about effective teaching in mathematics, of technology use in mathematics instruction, and weekly short reports on their teaching. Researchers also observed classes and conducted interviews with instructors from both experiment and comparison groups using a standard observation protocol adjusted for use in postsecondary settings.

Data Analytic Strategy: Researchers used multi-level mixed-effects methods (hierarchical linear models) to analyze Khan Academy's impact on the outcomes of interest in order to account for the nested structure of the data: students within classes/instructors within community college sites. Additional analyses focused on how different methods of using Khan Academy interacted with instructor and student characteristics and behaviors to impact student outcomes.

Products and Publications

Journal article, monograph, or newsletter

Hauk, S., & Kaser, J. (2020). A search to capture and report on feasibility of implementation. American Journal of Evaluation, 40(1), 145–155. https://doi.org/10.1177/1098214019878784

Kaser, J., & Hauk, S. (2016). To be or not to be an online math instructor?. MathAMATYC Educator, 7(3), 41.

Tipton, E., & Matlen, B. J. (2019). Improved generalizability through improved recruitment: Lessons learned from a large-scale randomized trial. American Journal of Evaluation 40(3), 414–430. https://doi.org/10.1177/1098214018810519.

Proceedings

Hauk, S. & Matlen, B. J. (2016). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra. In T. Fukawa-Connelly, N. Infante, K. Keene & M. Zandieh (Eds.), Proceedings of the 19th Conference on Research in Undergraduate Mathematics Education (pp. 204–209). Pittsburgh, PA. Full Text

Hauk, S. & Matlen, B. J. (2017). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra [Conference Long Paper]  In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 360–372). San Diego, CA. Full Text

Hauk, S., & Matlen, B. (2018). Implementation and Impact of a Web-Based Activity and Testing System in Community College Algebra. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 21st Annual Conference on Research in Undergraduate Mathematics Education (pp. 908–916), San Diego, CA. Full Text

Hauk, S., Matlen, B. J., & Thomas, L. (2017). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra [Contributed Report]. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 638–645). Full text

Hauk, S., Salguero, & K., Kaser, J. (2016). How "good" is "good enough"? Exploring fidelity of implementations for a web-based activity and testing system in developmental algebra Instruction. In T. Fukawa-Connelly, N. Infante, K. Keene & M. Zandieh (Eds.), Proceedings of the 19th Conference on Research in Undergraduate Mathematics Education (pp. 210–217). Pittsburgh, PA. Full Text

Hubbard, A. (2017). Data cleaning in mathematics education research: The overlooked methodological step [Conference Long Paper]. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 129–140). San Diego, CA. Full text

Publicly available data: To request access to research data sets, contact the principal investigator.