Project Activities
The researchers conducted a small pilot study in the spring of 2015 and then a multi-site cluster-randomized controlled trial study across the 2015-16 and 2016–17 academic years to test the impact of Khan Academy as used in blended classrooms in California community college algebra courses. In addition to documenting use of Khan Academy and student learning through a common test, the researchers also collected data to help identify the factors that contribute to higher quality and more effective implementations of Khan Academy, including teacher preparation, student characteristics, and course structure.
Structured Abstract
Setting
The research took place in 20 of the 106 community colleges throughout California that offered semester-long algebra courses in the years of the study. Of these, 11 were designated as minority serving institutions. Also, 4 of the schools served rural areas and 6 served urban areas.
Sample
The analytic sample consisted of 588 students of 34 instructors across 20 colleges. Enrollment in the 20 colleges, according to federal racial/ethnic categories, was representative of the statewide enrollment and included approximately 44% Hispanic, 29% White, 13% Asian, 5% African American, 3% Filipino, 1% Native American/Pacific Islander, and 4% multi-ethnicity students.
Intervention
Khan Academy is a free, internet-based learning environment that includes instructional videos, adaptive problem sets, and tools for instructors to provide individualized recommendations and assignments to students. When used in the study's community colleges, Khan Academy was part of a blended classroom: a classroom that contained traditional elements (e.g., in-person class meetings) and online resources. While each instructor selected the exact balance, there were three types of blended classrooms in the study, each used by about one-third of the classes: (a) Khan Academy was a main component of instruction (e.g., Khan Academy assignments were a primary form of practice and assessment for as much as 20% of the course grade), (b) Khan Academy use was a supplement to instruction (i.e., most classwork and homework was completed on paper or through a text-book digital resource and teachers additionally assigned the Khan Academy Algebra Basics Mission to the entire class or to individuals), (b) computer lab rotations using Khan Academy (i.e., where students rotated from teacher-led classroom instruction to computer-assisted laboratory learning either on a schedule or as needed).
Research design and methods
The study team contacted faculty and administrators in each of the California community college mathematics departments at the start of the study. Instructors participated for two semesters. In the first semester, treatment instructors familiarized themselves with implementing the Algebra Basics Mission with their local algebra curriculum. In the second semester, these teachers implemented Khan academy. The data from the second semester was Each Khan Academy Algebra Basics Mission session took 30 to 70 minutes of instructor-participant time (every session had embedded tasks and quizzes related to preparing to teach with the Khan Academy tools). These web-based sessions included presentation of the five main developer intentions regarding use of a Mission: give assignments to be completed in Khan Academy at least weekly, make mastery of the Algebra Basics Mission 20% of the course grade, use three Khan Academy-provided mindset lessons in-class early in the semester, review and respond to student progress reports (a monitoring tool in the Khan Academy system), and encourage students through recommendations beyond the mastery challenges (an assigning tool in the Khan Academy system). Treatment instructors repeated implementations in the spring semester (the efficacy phase), where the data for evaluation of impact were collected.
Control condition
Control group students received "business-as-usual" instruction in community college algebra courses. Control group classes did not use Khan Academy but might use other technology tools (e.g., online homework connected to the course textbook).
Key measures
The primary outcomes of interest were students' course completion rates and mathematics achievement as measured by performance on the Mathematics Diagnostic Testing Project (MDTP) elementary algebra assessment Students were also offered the opportunity to complete surveys about their background and experiences with mathematics. Data collected from instructors included assessments of their professional knowledge and beliefs about effective teaching in mathematics, of technology use in mathematics instruction, and weekly short reports on their teaching. Researchers also observed classes and conducted interviews with instructors from both experiment and comparison groups using a standard observation protocol adjusted for use in postsecondary settings.
Data analytic strategy
Researchers used multi-level mixed-effects methods (hierarchical linear models) to analyze Khan Academy's impact on the outcomes of interest in order to account for the nested structure of the data: students within classes/instructors within community college sites. Additional analyses focused on how different methods of using Khan Academy interacted with instructor and student characteristics and behaviors to impact student outcomes.
Key outcomes
The main findings of this study are as follows:
People and institutions involved
IES program contact(s)
Products and publications
Products: This project collected data on the effect of using Khan Academy resources in postsecondary algebra education and it produced peer-reviewed publications and presentations.
Journal article, monograph, or newsletter
Hauk, S., & Kaser, J. (2020). A search to capture and report on feasibility of implementation. American Journal of Evaluation, 40(1), 145-155. https://doi.org/10.1177/1098214019878784
Kaser, J., & Hauk, S. (2016). To be or not to be an online math instructor?. MathAMATYC Educator, 7(3), 41.
Tipton, E., & Matlen, B. J. (2019). Improved generalizability through improved recruitment: Lessons learned from a large-scale randomized trial. American Journal of Evaluation 40(3), 414-430. https://doi.org/10.1177/1098214018810519.
Proceedings
Hauk, S. & Matlen, B. J. (2016). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra. In T. Fukawa-Connelly, N. Infante, K. Keene & M. Zandieh (Eds.), Proceedings of the 19th Conference on Research in Undergraduate Mathematics Education (pp. 204-209). Pittsburgh, PA. Full Text
Hauk, S. & Matlen, B. J. (2017). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra [Conference Long Paper] In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 360-372). San Diego, CA. Full Text
Hauk, S., & Matlen, B. (2018). Implementation and Impact of a Web-Based Activity and Testing System in Community College Algebra. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 21st Annual Conference on Research in Undergraduate Mathematics Education (pp. 908-916), San Diego, CA. Full Text
Hauk, S., Matlen, B. J., & Thomas, L. (2017). Exploration of the factors that support learning: Web-based activity and testing systems in community college algebra [Contributed Report]. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 638-645). Full text
Hauk, S., Salguero, & K., Kaser, J. (2016). How "good" is "good enough"? Exploring fidelity of implementations for a web-based activity and testing system in developmental algebra Instruction. In T. Fukawa-Connelly, N. Infante, K. Keene & M. Zandieh (Eds.), Proceedings of the 19th Conference on Research in Undergraduate Mathematics Education (pp. 210-217). Pittsburgh, PA. Full Text
Hubbard, A. (2017). Data cleaning in mathematics education research: The overlooked methodological step [Conference Long Paper]. In A. Weinberg, C. Rasmussen, J. Rabin, M. Wawro, and S. Brown (Eds.), Proceedings of the 20th Conference on Research in Undergraduate Mathematics Education (pp. 129-140). San Diego, CA. Full text
Publicly available data: To request access to research data sets, contact the principal investigator.
Supplemental information
Co-Principal Investigator: Hauk, Shandy
- Inclusion of Khan Academy as part of the instructional materials correlated to increases in students' post-test scores (Hauk, Matlen, & Thomas, 2017)
- The greatest student gains were in the classes of cautiously optimistic instructors, namely those who voiced concerns about using the approach but still engaged with it (Hauk & Matlen, 2017).
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
