Co-Principal Investigator: Jennifer Yu (SRI International)

Purpose: The purpose of this study is to test the efficacy of the intervention, Universally Designed for Learning Science Notebook (UDSN), aimed at increasing fourth-grade students' science content knowledge. This study will be a replication study that builds on the results of a prior efficacy trial, but will cover a broader range of science content and have a larger sample. Eighty-two percent of fourth-grade students with disabilities performed at or below the basic level on the most recent science assessment of the National Assessment of Education Progress (2009). There is evidence from this research team's prior efficacy study that student use of UDSN had a significant impact on their science learning; however, this prior study was conducted with limited content in science (Magnetism and Electricity). More research is needed on the efficacy of such an approach at a large scale with a broader range of science content. This project seeks to fill this gap by studying the efficacy of implementing UDSN with two widely-used science curricula with fourth-grade students.

Project Activities: The researchers will test the impact of UDSN in students across three states using a randomized controlled trial. They will also test for moderation of student characteristics with subgroup analyses and test for the mediation of mechanisms by which UDSN supports students in improving their science knowledge.

Products: The products of this project include evidence of the efficacy of UDSN for improving student science knowledge across a broad range of science content, peer-reviewed publications and presentations.

Structured Abstract

Setting: The study will be conducted in elementary schools in Texas and Indiana. A school in Massachusetts will test quality assurance (to ensure the technology is working properly) prior to the efficacy study.

Sample: There will be 36 fourth-grade teachers and their students participating in this efficacy study. Each class is expected to include approximately 20 students, including 3 students with learning disabilities. Approximately 33 teachers and their students will participate in the initial quality assurance testing.

Intervention: UDSN is as an intervention that provides students with the space to collect, organize, and display their scientific observations and data they collect, as well as reflections and thoughts about their inquiry experiences. Teachers use the information as ongoing formative assessment and provide feedback to the students. The intervention also includes accessibility features such as text-to-speech technology, real time highlighting, keyboard accessible actions, multimedia glossary, and word-by-word English-to-Spanish translation. In addition, the design incorporates pedagogy to support active science learning through conceptual anchors (words and pictures) and navigation structures (plan, get data, explain).

Research Design and Methods: Before the start of the efficacy trial, initial quality assurance testing will determine if any technology bug fixes are needed or if infrastructure support is needed to scale up of the implementation of UDSN. The researchers will then use a randomized controlled trial to compare the outcomes of students in classes using the UDSN versus the traditional paper-based science notebooks. Teachers will be randomly assigned to either the UDSN or the control condition in Year 1, and in Year 2 their classes will be assigned the opposite condition. Only students in classes assigned to the UDSN condition will have logins to use the program. All teachers will be trained in both UDSN and the traditional paper notebook methods. Data will be collected to monitor contamination (use of intervention-related approaches with traditional paper notebook classes) and intervene as necessary. The researchers have included a moderation and subgroup analysis in their design to determine if effect sizes are influenced by UDSN differentially for students with learning disabilities or other subgroups (e.g., gender, ethnicity, free and reduced lunch status).

Control Condition: Students in the control condition will use traditional paper-based science notebooks.

Key Measures: Student outcome measures will include two measures of student characteristics—student record abstract (e.g., student aptitude and reading scores used for special education eligibility and demographics) and the Learning Disability Diagnostic Inventory (teacher rating scale for listening, speaking, reading, writing, mathematics and reasoning skills of students). Four measures of science content learning will be used: Pearson Interactive Science and STEMscopes Unit Tests (aligned to curriculum and completed at the end of each unit), ASK Science Content Tests (measure of knowledge of science content), Measures of Academic Progress (broader measure of science knowledge), and an assessment of student notebooks using the Assessment of Student Notebook Work rubric. Fidelity will be measured through a researcher-developed assessment, as well as data on dosage, adherence and quality data gathered through the UDSN usage log information and a log that teachers complete. Three measures of social validity will be used to understand the general support, perceived beneficial effects, and therapeutic alliance—classroom survey of teachers, teacher log of the use of class time, and student surveys.

Data Analytic Strategy: Hierarchical linear model regression analyses will be used to understand the impact of the intervention on student outcomes, the presence of any subgroup moderating variables, and any mediation effects (e.g., notebook usage and science engagement). In addition, a treatment on the treated analysis will be conducted to link fidelity of implementation with student outcomes using an instrumental variable approach (two-stage least-square model).

Related IES Project: The Universally Designed Science Notebook: An Intervention To Support Science Learning For Students With Disabilities (R324A070130)