Purpose: The research team will develop web-based activities that use simulations and representational tools to automatically score two Next Generation Science Standards (NGSS) high school physical science practices, mathematics use and modeling, and provide students scaffolding as they conduct inquiry to help them learn these practices. These practices are critical to high school STEM courses since mathematics is a critical barrier for many students.

Project Activities: The team will design and develop web-based activities that use simulations and representational tools to automatically score the mathematics use and modeling of these activities in high school physical science. They will design and develop knowledge-engineered and data mining-based algorithms to auto-score students' performance and demonstrate the potential efficacy of the scaffolding approach at helping students learn and transfer these practices both within and across topics. These materials will be empirically tested with students, including a sample of low-performing math students.

Products: The project results will contribute to many key areas of STEM learning and assessment, as well as to the sustainability of education innovations. Products include multiple assessment activities that teachers can use over the course of the school year, as well as tracking and reporting on students' competencies over time.

Structured Abstract

Setting: This project will take place in a variety of school settings (urban & rural) in New Jersey, Massachusetts, California, and Kentucky in public high schools.

Sample: The research sample includes 2,000 grade 9 to 10 students, who are racially, ethnically, and economically diverse. This will also include a sub-set of students who are low performing on mathematics. The sample is meant to be representative of the American high school student population.

Intervention: The research team will implement the intervention in a scalable web-based platform called Inq-ITS. Formative assessment activities in Inq-ITS will automatically assess high school students' competencies on NGSS practices focusing on mathematics use and modeling in real time as they conduct inquiry to investigate topics in the physical sciences using simulations, a type of model, and representational tools. These activities are stealth, performance-based assessments. As students "show what they know" by conducting their investigations in Inq-ITS, a pedagogical agent will provide fine-grained support when the system detects that students need help on these two key practices.

Research Design and Methods: The researchers will conduct individual interviews with teachers and collect think aloud protocols with students (including students who perform poorly on math) to inform the design of the assessment activities and scaffolds. The team will iteratively design the materials and then do classroom pilot studies using controlled experiments in which scaffolding by the pedagogical agent will be randomly assigned to students within-classroom, per simulation topic (6 in total). Paper-and-pencil out-of-system measures for the two practices of interest will also be administered to students for each topic for the purposes of gathering evidence of construct validity of Inq-ITS measures.

Control Condition: Students in the control condition will have access to the same investigation activities, simulations, and representational tools as in the intervention condition. They will not, however, have access to the scaffolding provided by the pedagogical agent.

Key Measures: The research team will assess improvement in student performance on mathematics use and modeling practices within Inq-ITS for both within Physical Sciences topics (near transfer) and across Physical Sciences topics (far transfer).  They will also collect additional measures of students' competencies on the two practices of interest using pencil and paper tests (1 per topic assessed in Inq-ITS; 6 in total). The researchers will examine student performance on the two practices of interest on the out-of-system measures for students who were scaffolded (within Inq-ITS) on these two practices vs. those who were not scaffolded. If borne out, this would provide further evidence of construct validity of the Inq-ITS measures, in addition to correlation and factor  analyses of the two sets of items.

Data Analytic Strategy: The researchers will assess student proficiency in mathematics use and modeling practices using a combination of knowledge engineering and educational data mining methods. They will also use hierarchical logistic regression models, along with correlation, factor analysis, and regression analyses to provide evidence of construct validity for the Inq-ITS measures.

Cost Analysis: The researchers will use the ingredients method to monitor the costs associated with the implementation of the intervention, such as training (i.e., professional development), personnel (i.e., teacher time), and equipment and materials (i.e., printing materials, access to technology, Inq-ITS technologies). The final analyses will attend to total costs associated with these items, and include any new costs realized during implementation. In order to collect data for the cost analysis, the researchers will survey and/or interview participating teachers and school administrators (and any other relevant stake holders such as science coordinators or technology directors) following the completion of the pilot study to identify all resources (and corresponding cost estimates) needed for schools to implement the system and its intervention for the pilot study. The survey and follow-up interview questions will include information related to the price/value, quantity, and percentage of use of for training, personnel, and equipment and materials resources (including for resources that may not be an expenditure).

Related IES Projects: Inq-ITS Online Labs for High School Physics and Physical Science (91990019C0037); The Development of an Intelligent Pedagogical Agent for Physical Science Inquiry Driven by Educational Data Mining (R305A120778); ASSISTment Meets Science Learning (AMSL) (R305A090170)