|Investigating the Use of Virtual Labs to Promote Chemical Reasoning and Practice Skills
|Science, Technology, Engineering, and Mathematics (STEM) Education [Program Details]
|3 years (9/1/2017–8/31/2020)
|Development and Innovation
Co-Principal Investigator: Yaron, David
Purpose: The purpose of this project was to develop and test the usability, feasibility, and promise of a fully developed series of innovative, online chemistry activities, ChemVLab+, that allow students to develop reasoning and science practice skills aligned with the Next Generation Science Standards (NGSS). High school chemistry is an important target for intervention with nearly twice as many high school students enrolling in chemistry compared to physics, and improved chemistry instruction can potentially have a large impact on students' scientific literacy. The current project builds off of a previous IES Development and Innovation grant by creating eight online chemistry activities that deepen and expand content from the prior prototype materials to systematically address a broader array of science practices outlined in the NGSS.
Project Activities: The researchers developed and pilot tested ChemVLab+, a set of eight online activities targeting high school chemistry. The activities use real-world contexts to help students integrate disciplinary core ideas, with crosscutting concepts and science practices. The activities allow for open-ended investigations in a virtual chemistry lab and provide just-in-time feedback to students. Reports inform teachers of student proficiency across learning objectives.
Key Outcomes: Key outcomes are forthcoming.
Setting: The study took place in diverse high school chemistry classrooms in urban, suburban, and rural schools in California, Pennsylvania, and New Jersey.
Sample: Ten chemistry teachers and approximately 700 students in grades 9 through 12 participated in the study. The students participating in the study represented a range of socio-economic levels and levels of prior science achievement.
Intervention: In a previous IES Development and Innovation grant, the researchers created ChemVLab+, a series of eight prototype activities that promote the development of conceptual understanding and science inquiry skills. In this project, the team created online activities that deepen and expand content from the prior ChemVLab+ prototype materials to systematically address a broader array of science practices outlined in the Next Generation Science Standards. The online chemistry activities allowed students to perform experiments and analyze data in a flexible, multimedia virtual chemistry lab environment.
The activities focus on the practice of designing investigations, which is essential to understanding the process of science but is difficult to teach and learn. Each activity in the series was designed to fit into one to two 45-minute class periods and allow students to pause and continue where they left off. The activities provide individualized instruction with feedback tailored to each student's interactions with the system. As students move through the activities, they respond to interactive prompts to plan and carry out investigations in a virtual chemistry lab. For all activities, students can request hints and will receive feedback (either explanations or additional practice) if they are on the wrong track.
A key component of the activities is the ChemCollective virtual laboratory simulation that allows students to plan and carry out investigations in an open-ended, but scaffolded, environment. Such open-ended simulation environments offer an alternative to textbooks and have shown promise for allowing students to visualize invisible processes, develop conceptual understanding, and demonstrate science practice skills. The activities generated log files that maintained a complete record of students' interactions.
Research Design and Methods: The research included an iterative cycle of development and testing. During phase 1 of the study, the researchers conducted data mining of existing log files from approximately 1,300 students that used the previous ChemVLab+ prototype activities. In addition, the researchers developed mock-ups of the activities that were designed to work with the new HTML5 version of the ChemCollective virtual lab, developed an online post-test assessment, assessed its reliability and validity, and sought expert review to ensure alignment of the activities and assessments with the learning objectives.
After the expert review of mock-ups of the activities, in phase 2, the research included two major cycles of usability and classroom feasibility testing. For the usability studies, think aloud studies with eight students and four high school chemistry teachers were conducted to determine whether the activities were operating as intended, elicited targeted core ideas and practices, and were motivating and engaging for the students. For the feasibility study, the researchers tested a subset of the activities during the spring of the 2017–2018 school year and 2018–2019 school year. During the 2019–2020 school year, the researchers captured data on all eight activities through classroom observations and student think-alouds. Throughout usability and feasibility testing, the researchers conducted classroom observations and teacher interviews as well as collected student interaction and post-test assessment data from the Open Learning Initiative system, teacher instructional logs, and a teacher questionnaire.
In phase 3 of the study, the researchers assessed the promise of the fully developed intervention using a cluster-randomized control trial design, randomly assigning half of for each teacher's classes to use the ChemVLab+ activities, and assigning the other half to use active control materials based on Khan Academy offerings. The researchers recruited 11 teachers to participate in year 3 of the project (during the 2020–21 school year), though 1 teacher was unable to complete the study because of issues related to decreased instructional time due to COVID-19. Classroom observations, teacher logs, student activity logs, and post-test data were collected.
Control Condition: Active control activities were derived from freely available Khan Academy offerings and included videos and problem-solving exercises with feedback. The researchers selected Khan Academy as both ChemVLab+ and Khan Academy are freely available online resources that teachers might use to supplement chemistry instruction. By including an active control, the researchers controlled for time on task and classroom instruction variations. Both sets of activities were placed on the same platform and were aligned to ensure the same topics were addressed in roughly the same amount of time.
Key Measures: Measures to assess usability and feasibility include verbal protocols from cognitive lab studies, classroom observation protocols, student responses on assessment items, log data on student actions in the Virtual Lab environment, instructional logs, teacher questionnaires, and measures of motivation and engagement. Measures to assess evidence of promise include researcher developed post-test measures of chemistry learning.
Data Analytic Strategy: Researchers analytic techniques included descriptive statistics, qualitative analyses, and item response theory. To determine the impact of the activities on learning outcomes, the research team conducted a two-level hierarchical linear model analysis, with students nested within classrooms.
Related IES Project: Embedded Assessments Using the ChemCollective Virtual Lab (R305A100069)
Publications and Products
ERIC Citations: Find available citations in ERIC for this award here.
Project Website: http://www.chemvlab.org/home/index.php
Davenport, J. L., Rafferty, A. N., and Yaron, D. J. (2018). Whether and how authentic contexts using a virtual chemistry lab support learning. Journal of Chemical Education, 95(8), 1250-1259.
Liu, R., Stamper, J. C., & Davenport, J. (2018). A novel method for the in-depth multimodal analysis of student learning trajectories in intelligent tutoring systems. Journal of Learning Analytics, 5(1), 41-54
McCormick, S., Davenport, J. L., Rafferty, A. N., Raysor, S., Yani, J., & Yaron, D. (2023). ChemVLab+: Integrating Next Generation Science Standards practices with chemistry. Journal of Chemical Education. 100(6), 2116-2131.
McCormick, S., Powers, J., Davenport, J., and Yaron, D. (2021). ChemVLab+: Helping students think like chemists. California Classroom Science, 34(1).