Co-Principal Investigators: Barbara Buckley, Mark Loveland, Daniel Brenner

Purpose: The use of models and simulations has helped change the nature of inquiry for scientists and students. Simulations can represent dynamic science systems "in action," making invisible phenomena visible, and making these models available for active investigations of authentic problems. When coupled with a technical infrastructure that captures and analyzes students' ongoing actions and answers, interactive simulations can provide personalized feedback and coaching, as well as reports that teachers can use to adjust instruction. Building off of a prior IES Goal 2 Development and Innovation grant (SimScientists: Interactive Simulation-Based Science Learning Environments), researchers will develop and test additional simulation-based supplements for middle school life science systems. In addition, the researchers will develop and validate the learning progressions, trajectories, and connections among multiple life science systems advocated in the Next Generation Science Standards.

Project Activities: Researchers will develop and test a systems model framework to (1) hypothesize three learning trajectories for core middle school life science systems (Genetics, Evolution, and Ecosystems); (2) develop three simulation-based instructional suites to promote and assess students' understanding, integration, and investigation of components (C), interactions (I), and emergent (E) behaviors of each system; (3) test and refine the hypothesized learning trajectories in middle school classrooms; and (4) explore students' ability to explain causal connections among the systems.

Products: The outcomes of the project include fully learning trajectories and simulation-based instructional suites for the life science systems of genetics, evolution, and ecosystems. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: This study will be conducted with middle school teachers (grades six to eight) and students in California and Washington.

Sample: During Year 1 of the study, one middle school teacher will participate as co-developer of the new curriculum modules and assessments for genetics and evolution. In Years 2 and 3 of the project, six middle school teachers and their students will participate. It is estimated that each middle school teacher will have four classes of approximately 25 students, for a total of 600 students in the middle school sample. The students in the sample will represent a range of socioeconomic levels, and will include disadvantaged students and English language learner students.

Intervention: The fully developed intervention will include model progressions, learning trajectories, and simulation-based instructional suites for the middle school life science systems of genetics, evolution, and ecosystems. Researchers will begin by developing new benchmark assessments for genetics and evolution, and modify the ecosystems benchmark assessment. Each benchmark assessment will provide summative information about student placement on the model progression and learning trajectory. Curriculum modules will be developed to include: (1) a driving problem of a type important to the discipline (e.g., invasive species in an ecosystem) and set in an authentic context meaningful to students, (2) a presentation/demonstration of the dynamic system in action, and (3) a sequence of tasks to foster integration of and elicit use of science knowledge and inquiry practices appropriate for the model level. The SimScientists Learning Management System (LMS) that delivers the web-based curriculum modules and assessments also collects data during student use and generates a progress report for each student by content and inquiry target and by model level for each simulation-based activity. Reflection activities will also be developed to scaffold collaboration and scientific discourse among students on tasks designed to promote transfer of understandings of the components (C), interactions (I), and emergent (E) behaviors of systems by providing additional instruction and differentiated tasks in a new system.

Professional development includes a 2-day summer workshop session followed by just-in-time webinars on the new modules (including an overview of the teacher's role, learning trajectories, and guidance for aligning curriculum modules with state standards and instructional texts).

Research Design and Methods: Year 1 of the project begins with a 6-month design phase during which researchers will create model progressions and learning trajectories, develop alpha versions of the middle school simulation-based curriculum modules for genetics and evolution, and modify the extant ecosystems modules and benchmark. The researchers will conduct cognitive labs for each suite of simulation-based instructional modules and benchmark assessments with three to five students, and listen as they think aloud while responding to the instructional components, feedback and coaching, and the embedded and benchmark assessments. Teachers will be asked to review each suite in a similar fashion. These cognitive labs will provide initial evidence about alignment of the instructional and assessment tasks and items to the hypothesized model progressions and learning trajectories. The modules will be revised as necessary. The modules will then be tested in the classroom of the teacher co-developer and revised as necessary.

Year 2 includes a 6-month period of revision. Revisions will be made to the modules based on findings from the feasibility and usability testing during Year 1. In addition, during Year 2, eight middle school life science teachers in Washington State will participate in Pilot Test 1 of the beta versions of the new genetics, evolution, and modified ecosystem suites. Based on feedback and data from Pilot Test 1, further revisions to the modules will be made for Pilot Test 2 during Year 3. In Pilot Test 2, researchers will conduct a small randomized control study to examine the effects of the instructional modules on student learning by randomly assigning half of each teacher's classes to use the simulation-based instructional modules and the other half of the classes to the control condition. A conventional pre/post-test and the simulation-based benchmark assessment will be administered to all classes.

Control Condition: For Pilot Test 2, classrooms assigned to the control condition will implement their business-as-usual practices for teaching genetics, evolution, and ecosystems.

Key Measures: The key measures include pre- and post-tests for the three life science systems units (genetics, evolution, ecosystems), the embedded and benchmark assessments for each module, school district-level science assessment data, computer logs, classrooms observations, and teacher interviews and surveys.

Data Analytic Strategy: Promise of the modules to improve student learning will be evaluated in three ways. First, differences between the treatment and control classrooms will be examined by comparing raw gain scores from the pre- and post-tests and scores on the simulation-based benchmark assessments. Second, progress along each learning trajectory will be evaluated by estimating model level mastery using a Bayes Net classifier for each of the embedded and benchmark assessments. This analysis will provide evidence of whether student gains were consistent with mastery of more of the system model levels. Finally, a hierarchical linear regression model will be fit to the data to fully evaluate sources of variance in the simulation-based benchmark ability estimates produced by the Bayes Net.

Related IES Projects: SimScientists: Interactive Simulation-Based Science Learning Environments (R305A080614)

Products and Publications

Book chapter

Davenport, J.L., and Quellmalz, E.S. (2017). Assessing Science Inquiry and Reasoning Using Dynamic Visualizations and Interactive Simulations. In R. Lowe, and R. Ploetzner (Eds.), Learning From Dynamic Visualizations: Innovations in Research and Practice (pp. 203–232). New York: Springer.

Quellmalz, E. S., and Silberglitt, M. D. (2017). Affordances of Science Simulations for Formative and Summative Assessment. In H. Jiao and R.W. Lissitz (Eds.), Technology Enhanced Innovative Assessment: Development, Modeling, and Scoring From an Interdisciplinary Perspective (pp. 71–94).

Quellmalz, E.S., and Silberglitt, M.D. (2017). Simscientists: Affordances of Science Simulations for Formative and Summative Assessment. In H. Jiao and R.W. Lissitz (Eds.), Technology Enhanced Innovative Assessment: Development, Modeling, and Scoring From an Interdisciplinary Perspective (pp. 71–94). Information Age Publishing.

Quellmalz, E.S., Silberglitt, M.D, Buckley, B.C., Loveland, M.T., and Brenner, D.G (2016). Simulations for Supporting and Assessing Science Literacy. In Y. Rosen, S. Ferrara, and M. Mosharraff (Eds.), Handbook of Research on Technology Tools for Real-World Skill Development (pp. 191–229).

Quellmalz, E.S., Silberglitt, M.D., Buckley, B.C., Loveland, M.T., and Brenner, D. (2016). Simulations for Assessing and Supporting Science Literacy. In Y. Rosen, S. Ferrara, and M. Mosharraf (Eds.), Handbook of Research on Computational Tools for Real-World Skill Development (pp. 191–229). Hershey, PA: IGI Global.