Purpose: Recent national science education reports emphasize that student science learning should go beyond students solely knowing basic facts to deeper understanding and the ability to engage in meaningfully scientific practice. This project is developing SimScientists, a web-based simulation intervention designed to supplement and extend existing middle school science instructional materials. By offering problem-driven curriculum inquiry activities, embedded formative assessments, reflection activities, and teacher professional development, SimScientists is intended to address the urgent need to promote complex science learning, particularly for underperforming students.

Project: The research team will first develop prototypes of the SimScientists modules. The technical infrastructure will serve as a foundation for further development of simulation-based curriculum supplements. Formative evaluations of the intervention will proceed from small-scale feasibility and usability testing to two rounds of pilot testing in schools. Student learning will be measured by project benchmark and formative assessments, available district science test results, and teacher ratings of student proficiencies on standards. Analyses will include pre/post test comparisons, learning trajectories, IRT analyses, correlational studies, and cross-case study summaries.

Products: The expected outcomes of this research will include simulation-based curriculum and assessment SimScientists modules for three topics in middle school science (e.g., force and motion, ecosystems, and climate and weather) that will supplement and extend the science knowledge and skills typically addressed in static print materials. In addition, this project will provide professional development materials that are aligned with the science content and skills.

Structured Abstract

Purpose: In this project, WestEd will develop and test a technology-based intervention for middle school students designed to promote deep understanding of scientific ideas and the practices of science. SimScientists modules will incorporate design principles for effective instruction and assessment derived from learning research.

Setting: The setting for this study will be a range of diverse middle schools in the San Francisco area.

Population: Participants across the research studies in this project will include approximately 10 middle school science teachers and classrooms, with 25 students per classroom.

Intervention: In SimScientists, students will be presented with engaging problems to be solved by employing science inquiry processes. Using simulations, students will investigate the nature of components in a system and their interactions. Simulations will provide multiple representations that model dynamic, causal, and temporal phenomena that are typically difficult to depict in static print. Students will respond to tasks embedded within the simulation-based curriculum lessons. A scoring system will then analyze these responses and provide information for formative assessment of learning progress and difficulties. SimScientists will provide immediate feedback to students in the form of correct answers and examples of appropriate explanations, as well as individualized links to a graduated coaching system that will offer alternative ways to scaffold the targeted standards-based science knowledge and inquiry skills. In-class reflection activities will engage students in scientific argumentation and discourse and in self-assessment and monitoring. Simulation-based benchmark assessments with documented technical quality will permit evaluation of proficiency on hard-to-measure science standards related to understanding connected knowledge and using inquiry abilities.

Research Design and Methods: Alpha versions of the simulation-based curriculum modules and professional development materials will be developed first. Three teachers and students will be employed for think-aloud feasibility and usability testing of early versions of the materials. An iterative cycle of pilot testing and revision will be conducted during the second and third years of the project in the classrooms of the three teacher-developers. Based on feedback, pilot tests and revisions to the modules, a final version will be pilot tested in six classrooms. A combination of qualitative and quantitative methods will be used to address the evaluation questions about the feasibility of SimScientists and its promise for enhancing students' science achievement.

Key Measures: Student learning will be measured by project benchmark and formative assessments, available district science test results, and teacher ratings of student proficiencies on standards.

Data Analytic Strategy: To determine the feasibility and usability of the modules, the team will examine student learning trajectories, and cross-case study summaries. To evaluate the assessment components, the team will use IRT analyses to inform decisions about items to retain in the final version of the modules. Analyses of data gathered during the final pilot test will include pre-/post-test comparisons of student outcomes.

Related IES Projects: SimScientists Assessment System (R305A120390) and SimScientists Model Progressions (R305A130160)

Products and Publications

Book chapter

Quellmalz, E.S., and Silberglitt, M.D. (2017). Simscientists: Affordances of Science Simulations for Formative and Summative Assessment. In H. Jiao and R.W. Lissitz (Eds.), Technology Enhanced Innovative Assessment: Development, Modeling, and Scoring From an Interdisciplinary Perspective (pp. 71–94). Information Age Publishing.