Co-Principal Investigators: Billman, Alison; Brodsky, Lauren; Weiser, Gary

Purpose: The purpose of this proposed project is to develop and pilot, Formative Assessment Bundling Literacy and Elementary Science in the NGSS (or FABLES), a set of classroom-based assessment resources with accompanying professional learning to support teachers in monitoring and enhancing their students' integrated science and literacy learning. Having usable and instructionally beneficial assessment tools can help teachers better grasp the Next Generation Science Standards (NGSS) vision for science learning and more adeptly plan instruction to move students forward in their learning of science content, practices, and literacy. When combined with professional learning, the benefits expand to encompass the strengthening of teachers' instructional and assessment practices, and the deepening of teachers' pedagogical content knowledge. Providing the FABLES intervention as an open-education resource on an online platform that is freely available to schools and districts will also promote rapid adoption and subsequently robust science instruction and learning.

Project Activities: The researchers will use the Next Generation Science Assessment for Young Scientists design approach that is undergirded by evidence-centered design methodology to iteratively develop and test high-quality formative assessment resources. A pilot study will be conducted to provide evidence of impact on teachers' pedagogical content knowledge as they develop more thorough insight into their students' science and literacy learning, as well as evidence of impact on student science and literacy achievement as a result of regular formative assessment opportunities to use and apply what they know to make sense of phenomena.

Products: The research team will develop: (1) 21 technology-delivered formative assessment tasks and rubrics covering key proficiencies needed to achieve the NGSS performance expectations in 3rd grade physical science and Earth and space science integrated with literacy and language skills; (2) teacher resources including teacher practice guides aimed at supporting implementation, scoring, and future instruction based on the formative assessments; (3) 32 hours of professional learning workshops/webinars designed to introduce key ideas at the intersection between student's three-dimensional science understandings and their literacy development; (4) an online portal that makes products freely available to districts, schools, and teachers across the U.S. and supports dissemination of project work.

Structured Abstract

Setting: Research and development will take place in California 3rd grade science classrooms implementing the Amplify Science K-5 curriculum.

Sample: For the pilot study, approximately 30 teachers and their classrooms, corresponding to about 750 3rd grade students, will be randomly assigned to either the treatment (FABLES intervention) or control conditions.

Intervention: The FABLES intervention will consist of two sets of formative assessment resources for third grade classrooms — one for Physical Science and one for Earth and Space Science. Each set of resources will include four key features: (1) NGSS-aligned assessment tasks with rubrics for interpreting student performance; (2) teacher practice guides for implementing formative assessment routines; (3) professional learning to help teachers learn about science assessment and instruction consistent with the Framework and the NGSS; and (4) an online platform for assessment delivery and providing easy access to teacher resource materials.

Research Design and Methods: A cross-institutional team will develop FABLES including WestEd, the University of California Berkeley's Lawrence Hall of Science, and Concord Consortium. These institutions will leverage their collective expertise in NGSS teaching and learning, young learners' development of science proficiency and literacy, science assessment, technology-enhanced formative assessment, and teacher professional learning. To develop high-quality formative assessment resources, the team will use the Next Generation Science Assessment for Young Scientists (NGSA-YS) design approach that is undergirded by evidence-centered design methodology. The researchers will iteratively refine the tasks via cognitive interviews, expert review, and field testing, and iteratively refine the teacher resources via expert review, teacher feedback, and field testing. The tasks and resources will be organized into an intervention supported by teacher professional learning that can be used weekly during physical science and Earth and space science instruction. Finally, a pilot test will examine the usability, feasibility, and instructional value of FABLES in order to inform the NGSA-YS design approach and determine the promise of the intervention.

In the pilot study, teachers (and their classrooms) will be randomly assigned to treatment or control conditions via blocks of school or school-pairing (depending on the number of teachers recruited from each school). For multi-school blocks, a Mahalanobis distance metric will be used to pair schools that are most similar on key demographic factors while minimizing the total distance between all within-block schools. Treatment teachers will implement the FABLES intervention on a weekly basis over the course of teaching their instructional sequence in Physical Science. Treatment teachers will receive professional learning experiences during the summer prior to the implementation school year as well as attend midyear workshops/webinars to help them make sense of their students' responses. In a follow-up round of implementation, treatment teachers will implement a weekly second set of FABLES in their classroom over the course of teaching their instructional sequence in Earth and Space Science.

Control Condition: Control condition classrooms will implement their business-as-usual curriculum and instructional strategies.

Key Measures: At the student level, Northwest Evaluation Association Measures of Academic Progress (NWEA MAP) Growth for Science and Language Use tests will be used to measure student learning. At the teacher level, the Assessing the Impact (AIM) Forces and Motion assessment and Content Knowledge for Teaching English Language Arts (CKT ELA) will be used to measure teacher pedagogical content knowledge development. Classroom observations in-person will use the Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) observation protocol while observations taken via video recording will be qualitatively coded and analyzed.

Data Analytic Strategy: The researchers will investigate the construct and instructional validity of the intervention products (i.e., assessment tasks, rubrics, teacher practice guides) and the promise of the intervention for improving teacher knowledge and student learning. In addition to qualitative measures of feasibility and usability of the FABLES intervention during the development phase, the pilot study will enable researchers to synthesize pretest performance, measures of fidelity of implementation, and other relevant covariates in a two-level HLM when evaluating the association between implementing FABLES and student outcomes and a one-level regression analysis when evaluating the association between implementing FABLES and teacher outcomes.

Cost Analysis: While FABLES will be available free-of-charge to teachers and students, costs associated with implementing FABLES above and beyond business-as-usual science teaching (including the opportunity costs of implementation) will be calculated using the ingredients method. Those costs will then be used to determine a cost-effectiveness ratio based on findings from the pilot study.