|Title:||Khan Academy Resources for Maximizing Mathematics Achievement: A Postsecondary Mathematics Efficacy Study|
|Principal Investigator:||Schneider, Steve||Awardee:||WestEd|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||3 years (7/1/2014-6/30/2017)||Award Amount:||$2,197,416|
|Goal:||Efficacy and Replication||Award Number:||R305A140340|
Co-Principal Investigator: Shandy Hauk
Purpose: Many students arrive to college underprepared for mathematics courses. In order to prepare students for college-level math—typically algebra—colleges offer remediation sequences, also called developmental math courses. Yet many students who take developmental mathematics courses are not successful. To improve developmental math courses and increase students’ success, some community colleges have turned to Khan Academy mathematics resources, an online, free resource that includes videos and practice sets. For this project, researchers are planning a large-scale efficacy study of Khan Academy in community college settings. This project will test the impact of the Khan Academy resources on student outcomes in developmental math when the resources are integrated into normal classroom activities.
Project Activities: The researchers will begin by conducting a small pilot study in the spring of 2015 to inform the development of study instruments, including implementation measures. This project will then conduct a multi-site cluster-randomized control study to test the impact of Khan Academy as used in blended classrooms in California community college Algebra I courses. For the main study, the researchers plan to recruit 36 California community colleges, with 4 participating instructors at each site. Half of participating instructors at each site will be randomly assigned to the treatment condition and half assigned to the control condition. One course section per participating instructor will be selected for the study. The researchers will also collect data to help identify the factors that contribute to higher-quality and more effective implementations of Khan Academy, including teacher preparation, student characteristics, and course structure.
Products: The products of this project will be initial evidence of the effect of using Khan Academy resources in developmental education. Peer-reviewed publications will also be produced.
Setting: The research will take place in multiple community colleges throughout California.
Sample: Researchers will recruit students enrolled in developmental algebra courses and instructors of developmental algebra courses.
Intervention: Khan Academy is a free, internet-based learning environment that includes instructional videos, adaptive problem sets, and tools for teachers to provide individualized coaching and assignments to students. When used in community colleges, Khan Academy is often part of a blended classroom, which, by definition, is a classroom that contains traditional elements (e.g., in-person instruction) and online resources. The blended classrooms involved in the current study are likely to use one of the following methods, depending on the instructor’s discretion: (a) Khan academy as a supplemental resource (i.e., where teachers assign particular Khan Academy videos and practice sets to the entire class or individual students), (b) computer lab rotations using Khan Academy (i.e., where students rotate from teacher-led instruction to computer-assisted instruction either on a schedule or as needed), and (c) flipped classrooms (i.e., where students use Khan Academy resources at home to study new material and receive guided practice subsequently with the instructor in-class).
Research Design and Methods: Prior to the main efficacy study, the researchers will conduct a small pilot study and refine the materials as necessary. During this time, the researchers plan to recruit 36 California community colleges for the study, with 4 participating instructors at each site for the main efficacy study. Half of the participating instructors at each site will be randomly assigned to the treatment condition, and only one course section per participating instructor will be selected for the study. These sections will be selected randomly, with the constraint that daytime and evening sections be proportionally represented in the sample. This means that some sites will have two daytime sections (one treatment, one control), and two evening/weekend sections (one treatment, one control) in the study.
Treatment teachers will work with Khan Academy teacher professional development and support staff to design their blended learning implementation models in Summer 2015. The teachers will be allowed to design their own blended learning implementation models so that the study remains faithful to what constitutes “typical practice” with Khan Academy. They will implement Khan Academy in their Algebra I courses for the first time in Fall 2015 and for the second time in Spring 2016.
Control teachers will continue to teach their Algebra I courses as before, representing a “business-as-usual” condition. Control teachers will be permitted to use any resources they have used in the course previously but must agree not to begin using Khan Academy resources during the study.
Control Condition: Control group students will receive “business-as-usual” instruction in developmental math courses at the same community colleges from instructors not in the experimental condition.
Key Measures: The primary outcomes of interest are students’ course completion rates and mathematics achievement, as measured by pass/fail status and performance on the Mathematics Diagnostic Testing Project (MDTP), an elementary algebra diagnostic. To collect data on instructors’ professional knowledge and beliefs, researchers will use the Assessment of Mathematics Pedagogical Content Knowledge (PCK), the Assessment of Technological Pedagogical Content Knowledge (TPACK), and a teacher belief survey and algebra problem difficulty ranking task developed by Nathan and Koedinger (2000). Researchers will also collect students’ course evaluations.
Data Analytic Strategy: Researchers will use a three-level mixed-effects hierarchical linear model to analyze Khan Academy’s impact on the outcomes of interest in order to account for the nested structure of the data: students within classes/instructors within community college sites. Additional analyses will focus on how different methods of using Khan Academy interact with teacher and student characteristics and behaviors to impact student outcomes.
Journal article, monograph, or newsletter
Kaser, J., and Hauk, S. (2016). To be or not to be an Online Math Instructor?. MathAMATYC Educator, 7(3).
Hauk, S. and Matlen, B. J. (2016, March). Exploaration of the Factors That Support Learning With Digitally-Delivered Activities and Testing in Community College Algebra. In S. Brown (Ed.), Proceedings of the 18th Conference on Research in Undergraduate Mathematics Education.
Hauk, S., Salguero, K., and Kaser, J. (2015). Classroom Observation, Instructor Interview, and Instructor Self-Report as Tools in Determining Fidelity of Implementation for an Intervention in Community College Algebra. In Research in Undergraduate Mathematics Education. Pittsburgh, PA: Research in Undergraduate Mathematics Education.
Hauk, S., Salguero, K., Kaser, J. (2016). Classroom Observation, Instructor Interview, and Instructor Self-Report as Tools in Determining Fidelity of Implementation for an Intervention in Community College Algebra. Pittsburgh, PA: Research in Undergraduate Mathematics Education.
Hauk, S., Salguero, K., Kaser, J. (2016, March). How ";good"; is ";good enough";? Exploring fidelity of implementations for a web-based activity and testing system in developmental algebra instruction. In S. Brown (Ed.), Proceedings of the 19th Conference on Research in Undergraduate Mathematics Education.
Matlen, B. J. and Hauk, S. (2016). Exploration of the Factors that Support Learning: Web-based Activity and Testing Systems in Community College Algebra. In 19th Conference on Research in Undergraduate Mathematics Education (pp. 204–209). Pittsburgh, PA: Research in Undergraduate Mathematics Education (RUME).
Matlen, B.J., and Hauk, S. (2016). Exploration of the Factors That Support Learning: Web-Based Activity and Testing Systems in Community College Algebra. In Annual Conference on Research in Undergraduate Mathematics Education. Pittsburgh, PA: Research in Undergraduate Mathematics Education.