Co-Principal Investigator:  David Yaron (Carnegie Mellon University)

Purpose: The purpose of this project is to develop and test the usability, feasibility, and promise of a fully developed series of innovative, online chemistry activities that allow students to develop reasoning and science practice skills aligned with the Next Generation Science Standards (NGSS).  High school chemistry is an important target for intervention with nearly twice as many high school students enrolling in chemistry compared to physics, and improved chemistry instruction can potentially have a large impact on students' scientific literacy. The current project builds off of a previous IES Development and Innovation grant by creating ten online chemistry activities that deepen and expand content from the prior prototype materials to systematically address a broader array of science practices outlined in the NGSS.

Project Activities: The researchers will develop a set of ten online activities targeting high school chemistry. The activities will use real-world contexts to help students integrate disciplinary core ideas, with crosscutting concepts and science practices. The activities will allow for open-ended investigations in a virtual chemistry lab and will provide just-in-time feedback to students. Reports will inform teachers of student proficiency across learning objectives.

Products: The project will produce innovative chemistry activities and data on students' chemistry learning. The researchers will make the chemistry activities freely available online to ensure the activities reach large populations of student. In addition, the findings from the project will be published in peer-reviewed journals.

Structured Abstract

Setting: The study will take place in diverse high school chemistry classrooms in urban and suburban California schools.

Sample: Twelve chemistry teachers and approximately 1,200 students in grades 9–12 will participate in the study. The students participating in the study will represent a range of socio-economic levels and levels of prior science achievement.

Intervention: In a previous IES Development and Innovation grant, the researchers created ChemVLab+, a series of eight prototype activities that promote the development of conceptual understanding and science inquiry skills. The online chemistry activities allowed students to perform experiments and analyze data in a flexible, multimedia virtual chemistry lab environment. The researchers will build off of the previous work by creating ten online activities that deepen and expand content from the prior ChemVLab+ prototype materials to systematically address a broader array of science practices outlined in the Next Generation Science Standards.

The activities will focus on the practice of designing investigations, which is essential to understanding the process of science but is difficult to teach and learn. Each activity in the series will be designed to fit into one to two 45-minute class periods and will allow students to pause and continue where they left off. The activities will provide individualized instruction with feedback tailored to each student's interactions with the system. As students move through the activities, they will respond to interactive prompts to plan and carry out investigations in a virtual chemistry lab. For all activities, students can request hints and will receive feedback (either explanations or additional practice) if they are on the wrong track.

A key component of the proposed activities is the ChemCollective virtual laboratory simulation that allows students to plan and carry out investigations in an open-ended, but scaffolded, environment. Such open-ended simulation environments offer an alternative to textbooks and have shown promise for allowing students to visualize invisible processes, develop conceptual understanding, and demonstrate science practice skill. The activities will generate log files that maintain a complete record of students' interactions.

Research Design and Methods: The research includes an iterative cycle of development and testing. During Phase I of the study, the researchers will conduct data mining of existing log files from approximately 1,300 students that used the previous ChemVLab+ prototype activities. In addition, the researchers will develop mock-ups of the activities that will be designed to work with the new HTML5 version of the ChemCollective virtual lab, develop an online pre- and post-test assessment and assess its reliability and validity, and seek expert review to ensure alignment of the activities and assessments with the learning objectives.

After the expert review of mock-ups of the activities, in Phase 2, the research will move to two major cycles of usability and classroom feasibility testing. For the usability studies, think aloud studies with 15 students and 6 high school chemistry teachers to will be conducted to determine whether the activities are operating as intended, eliciting targeted core ideas and practices, and are motivating and engaging for the students. For the feasibility study, a subset of the activities will be tested during the spring of the 2017–2018 school year, and all ten activities will be tested during the 2018–2019 school year.  The same 6 high school chemistry teachers will participate in both usability and classroom feasibility testing. The researchers will conduct classroom observations, teacher interviews, collect log data and teacher instructional logs, administer pre- and post-test assessments, and a teacher questionnaire.

In Phase 3 of the study, the researchers will assess the promise of fully developed intervention using a cluster-randomized control trial design, randomly assigning two classes for each teacher to use the ChemVLab+ activities, and assigning the other two classes to the control condition. Twelve teachers will be recruited to participate in Year 3 of the project (during the 2019–20 school year). Classroom observations, teacher logs, student activity logs, and pre- and post-test data will be collected.

 Control Condition: The control condition is the business-as-usual chemistry instruction.

Key Measures: Measures to assess usability and feasibility will include verbal protocols from cognitive lab studies, classroom observation protocols, student responses on assessment items, log data on student actions in the Virtual Lab environment, instructional logs, teacher questionnaires, and measures of motivation and engagement. Measures to assess evidence of promise include state science tests and researcher developed pre- and post-test measures.

Data Analytic Strategy: Researchers will use analytic techniques include descriptive statistics, qualitative analyses, and item response theory. To determine the impact of the activities on learning outcomes, the research team will conduct a two-level hierarchical linear model analysis, with students nested within classrooms.

Project Website: http://www.chemvlab.org/home/index.php

Related IES Project: Embedded Assessments Using the ChemCollective Virtual Lab (R305A100069)