Topic: Assessment, standards, and accountability research

Co-Principal Investigators: Bailey, Jessica; Choi, Kilchan; DeLisi, Jacqueline; Greenfield, Daryl; Kook, Janna; Moran, Meredith

Partner Institutions: University of Miami; National Center for Research on Evaluation, Standards, and Student Testing at University of California, Los Angeles; Lawrence Hall of Science at University of California, Berkeley

Purpose: CAESART will directly respond to the vision of the Next Generation Science Standards (NGSS) by addressing the nationwide gap in the availability of high-quality science instruction and assessment for young students (grades K–2). CAESART will connect and build on existing networks of science researchers, leaders, and practitioners at the state, district, and school levels to engage in research and to share information generated through Center activities. CAESART's approach addresses several questions that together will provide insight into how to measure student science learning and provide evidence about whether interventions related to high-quality integrated science and literacy curriculum sustained over multiple school years can improve students' science outcomes.

Project Activities: Researchers will conduct a landscape review of existing pre-K–5 science assessments and summarize the results in a public database; research, develop, test and validate a set of technology-based assessments to capture young students' science learning through adaptive and game-based approaches; use these assessments to measure the impact of an integrated science curriculum on science learning; and, engage science and education stakeholders to align the research to their priorities and contexts, identify implications from the research, and build capacity for science assessment.

Focused Program of Research: CAESART will conduct a landscape reviewof pre-K–grade 5 science assessments. They will carry out two measurement studies to create and validate new innovative, adaptive, and game-based assessments that fill a significant gap in assessing science learning in the early grades. Lens on Science (Lens), an IRT-based preschool science assessment, will be expanded to assess children's physical and life science learning in grades K, 1, and 2. Lens will leverage the affordances available in technology-delivered adaptive assessments, requiring less time to assess young children with limited attention spans compared to typical fixed format assessments, and providing engaging assessment items that are tailored to each child's ability level. A variety of item-level (for example, item properties and correct and actual response) and summary (for example, IRT ability score, number and order of items presented, and standard error) data are stored automatically in a secure online relational database that allow for analyses that address both measurement properties (for example, validity and item bias) and outcome data. The second measurement study will develop and validate two digital game-based performance tasks (GBPTs) to assess grade K physical science (K–PS) performance expectations that embody the NGSS performance expectations and allow students to reason about science phenomena and problems. The GBPTs will engage students in open-ended tasks with multiple solutions so they can demonstrate what they know. Manipulations and telemetry data that tracks students' interactions with the technology and behaviors in the tasks support efficient and reliable measurement of students' processes and reasoning used to solve tasks. In addition, researchers will conduct impact research to test the use of these assessments at scale to examine 1) the effect of an integrated science-literacy curriculum on grade K students' understanding of science concepts and science practices and 2) the effects of sustaining an integrated science and literacy curriculum over two academic years on grade 1 students' science learning outcomes.

National Leadership and Outreach Activities: National leadership, capacity building, and outreach activities include providing training opportunities (for example, workshops, webinars, and mentoring) and resources to build research and research use capacity; sharing CAESART's assessment resources and research findings; convening stakeholders through national events; and collaborating with national science organizations and stakeholders to inform research and disseminate evidence-based resources, assessments, and findings.

Structured Abstract

Setting: Researchers will recruit participants nationally, with concentrated efforts in Miami, Los Angeles, and the Northeast, ensuring the sample allows for generalizability to the desired population, particularly those typically underserved by science education.

Sample: Measurement study 1(Lens) will include 2,400 students in K–2 classrooms. Measurement study 2 (GBPTs) will include 300 K students from four schools. The impact studies will include 44 elementary schools (22 treatment and 22 control), 132 grade K and 132 grade 1 teachers and an estimated 1,584 students. Year 1 will include three grade K teachers in each school and 12 of their grade K students. Year 2 will include three grade 1 teachers and all the grade 1 children from the original year 1 sample in grade K in each of the 44 study schools. Each sample will be representative of characteristics important for learning and science achievement, attending to characteristics associated with underserved communities (that is, language, geography, and socioeconomic status).

Research Design and Methods: The landscape review (Year 1) includes a systematic review of existing science assessments through a comprehensive search and analysis. The Lens measurement study includes three phases: item development (Years 1–2), pilot and analysis (Years 1–3), and field testing (Years 2–3). The GBPTs measurement study includes seven phases (Years 1–3), spanning from defining performance levels to developing prototypes to field testing, and uses an evidence-centered design process. The design for the impact research (Years 3–4) uses a three-level block randomized design, randomizing at the school level, using a blocked cluster randomized design in which schools are the unit of randomization. The design is blocked by district and randomized within districts to ensure schools in the two groups have similar district contexts. To understand the effect of an integrated science-literacy curriculum (Amplify Science), the research team will compare outcomes for students whose teachers are given access and required to use the curriculum to outcomes of students whose teachers will receive the materials in the year following the study. To understand the effects of sustaining an integrated science and literacy curriculum over two academic years, the research team will compare outcomes for grade 1 students who experienced the grade K curriculum and whose teachers are given access and required to use the curriculum to outcomes of students whose teachers will receive the materials in the year following the study. Teachers in the comparison condition will implement science instruction as usual. Cost effectiveness studies will be included.

Data Analytic Strategy: The landscape review includes using a coding scheme for categorizing assessment data, identifying assessments for inclusion, and compiling into a database. The Lens measurement study uses a Rasch model (a one-parameter item-response theory model) for the psychometric analyses, which include examining difficulty, fit, item discrimination, principal components analysis, differential item functioning (DIF), and differential test functioning. The GBPTs measurement study data analysis includes calculating game-based indicators to use in validity (for example, DIF), statistical analyses (for example, latent variable two time point model), and qualitative analyses to examine potential mode effects. Correlational analyses will be conducted to examine performance on the Lens and on the GBPTs. All analyses estimating student outcomes from the impact research will follow an intent-to-treat (ITT) estimate of the average impact of the treatment for each unit offered in the intervention, regardless of whether schools and teachers implemented the intervention. The analytic process includes attrition analyses, missing data analyses, and descriptive analyses of the sample overall, then disaggregated by the treatment and control condition; and impact analyses of the overall impact, followed by moderator analysis and then mediator analysis.

Outcomes: CAESART's long-term outcomes focus on students and include equitable access to sustained high quality science instruction and learning experiences, improved science learning for K–2 students measured using the newly developed and validated Lens and GBPTs assessments, and increased student preparation for future science learning. Lens will provide valid inferences of three-dimensional science learning for life science and physical science disciplinary content for children in grades K to 2, incorporating disciplinary core ideas, science and engineering practices, and crosscutting concepts aligned with the K–2 grade band in the Framework for K–12 Science Education (NRC, 2012). Leveraging the affordances available in technology-delivered adaptive assessments, the GBPTs will provide assessments of the K-PS performance expectations. Complementing the broader Lens assessment, the GBPTs collect more specific fine-grained data about students' knowledge through telemetry data of their behaviors with game mechanics aligned to science phenomena. Additionally, medium- and short-term outcomes are examined, which relate to an increased availability and use of high-quality science assessments, effective curricula, and pedagogical practices, and increased capacity to conduct science research and use research.

Cost Analysis: Researchers will use a nested cost analysis spreadsheet to enumerate and value each resource, itemizing each ingredient under each component (for example, personnel and training), while noting the data source as well as any assumptions. Using available data sets, each ingredient will be assigned a dollar value using both national and local prices to provide a range of anticipated costs. The research team will perform a sensitivity analysis to account for assumptions introduced during the cost-effective modeling, including the average number of students in the study, the average amount of technical assistance the teacher sought out, and years of teacher experience.

Related IES Projects: Lens on Science: Development and Validation of a Computer-Administered, Adaptive, IRT-Based Science Assessment for Preschool Children (R305A090502); Enfoque en Ciencia: Extending the Cultural and Linguistic Validity of a Computer Adaptive Assessment of Science Readiness for Use with Young Latino Children (R305A130612)

Products and Publications

Products: The landscape review will provide the nation's first database of pre-K–grade 5 measures of science learning. The measurement studies will result in (1) an adapted validated Lens assessment to provide a computer-adaptive assessment of science learning related to life and physical science in K–2 and (2) validated digital GBPTs to assess kindergarten physical science performance expectations that embody NGSS expectations. Products from the impact research include information about the impact of Amplify Science on young learners' science outcomes, fidelity of implementation measures, cost information regarding the intervention, policy briefs, study reports, publications, and presentations. National leadership, capacity building, and outreach products include training materials, resources, policy briefs, reports, webinars, blogs, presentations, and publications.

ERIC Citations: Find available citations in ERIC for this award here.

*NSF is partnering with IES to provide joint funding for the Center.