Project Activities
In a prior IES grant (
R305A100692), the researchers developed and tested a learning progression for scientific argumentation, using a set of pencil and paper items to assess the constructs, and analyzed the evidence to refine and confirm the model. In this project, the researchers will further develop and validate materials to assess scientific argumentation, a central feature of the Next Generation Science Standards, and further refine the learning progressions for Structure of Matter and Ecology. The researchers will convert the existing paper-and-pencil based assessments to online assessments for easier and wider adoption in science classrooms which, wherever possible, can be graded by a computer.
Structured Abstract
Setting
The study will take place in an urban school district in California.
Sample
The sample will include teachers and students from Grades 8 to 10. The students are from a diverse, urban school district with 27% of students speaking English as a second language and 58% receiving free or reduced price lunch.
Assessment
The researchers will use the BEAR Assessment System and its online companion (BASS) to develop and refine assessment materials. The BEAR Assessment System (BASS) is an integrated approach to developing assessments that provides meaningful interpretations of student work relative to the cognitive and developmental goals of the domain. It is grounded in four building blocks (e.g., progress map, item design, outcome space, and measurement model) which guide assessment development. In the current project, the researchers will iterate through the four building blocks to gather high quality empirical evidence to improve the assessments.
Research design and methods
In Year 1, the researchers will work with teachers to modify and develop additional items to address any possible gaps in coverage of the learning progressions. An initial pilot of approximately 30 students using a think-aloud interview format will be conducted. By the end of Year 1, the researchers will pilot-test a small set of assessment tasks in electronic format.
Control condition
There is no control condition for this project.
Key measures
Key measures include students' performance on the computer-based assessment items, interviews with students and teachers, and samples of classroom instructional materials.
Data analytic strategy
The researchers will use a unidimensional item response theory model to investigate the quality of the individual constructs. In addition, the team will utilize the multidimensional random coefficients multinomial model (and other models depending on the data presentation) to investigate the complex nature of the relationships between the constructs within each learning progression.
People and institutions involved
IES program contact(s)
Products and publications
Products: The products include an online assessment to measure students' scientific argumentations skills within the domains of Structure of Matter and Ecology, and peer reviewed publications.
Supplemental information
Co-Principal Investigator: Jonathan Osborne (Stanford University)
The researchers will take advantage of the computerized delivery to create new item types and new item templates that combine the measurement of both student competence with scientific argumentation and the features of the disciplinary core ideas in Structure of Matter and Ecology. Researchers will use the computer scored items as formative assessments to provide instant feedback to students and teachers, as well as using the items for summative assessments.
In Year 2, revisions will be made to the Year 1 assessment items and materials, and further testing will occur with approximately 300 students. An additional round of data will be collected in Year 2 using a larger group of 1,000-1,500 students who will take all the available items under test-like conditions online. By the end of Year 2, the tasks that began as paper–and-pencil will be operational in their computer-based form along with other tasks developed from templates. The researchers will refine methods for conducting computer-based scoring of the items that have been transferred from paper-and-pencil format.
In Year 3, based on the feedback from the project's advisory board, researchers will conduct additional analyses if needed. If necessary, the researchers would administer a final trial of the same scope as the one in Year 2.
Examples of the online items for scientific argumentation developed with funding from IES grant Learning Progressions in Middle School Science Instruction and Assessment (R305A100692) are available at http://scientificargumentation.stanford.edu/.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
