Co-Principal Investigators: Cho, Sun-Joo; Biswas, Gautam

Purpose: In the era of COVID-19 and digital learning, IES has identified areas of needed research including exploring how reading with digital devices relates to learning outcomes like reading comprehension. This project team address this need by building understandings of in-the-moment behaviors readers experience (explored via eye-gaze, emotional response, digital behaviors, and highlighting) when reading on a screen or reading on paper to answer questions, which is similar to many classroom and testing tasks. Researchers then consider differences related to different reading contexts (reading static pdfs vs. reading in an open learning environment). They will work with a substantially larger sample than existing studies and focus on an understudied population (grades 5–8). This study will make a significant contribution to understanding how students read digitally as well as suggest process differences between digital and paper reading.

Project Activities: The research team will use extent data that has been largely coded and collect new data. Project activities include revision of software, data collection, coding, and analysis across four years. Year 1 efforts focus on extent data analysis at the process level as well as observing digital reading experiences and piloting data collection materials. In Year 2, the research team will continue extent data  analysis and also collect new data. In Year 3, they will code and analyze the new data, and in Year 4,  they will analyze the new data more fully. They will use coding, statistical, and machine learning expertise to deal with the multi-sourced big process data challenge that reading process data presents. The extent data were coded across multiple years via software developed by the research team. They will revise software to improve accuracy of coding and support efficient data collection.

Products: The research team will generate several types of data, including software, methods, tools, techniques, training materials, experimental data, and publications.  They will produce software to code reading process data and research methodology (statistical and machine learning approaches) to analyze reading process data. They will present findings at conferences and disseminate via social media and  create a website.

Structured Abstract

Setting: This study will take place within an urban school district in the Southeastern US.

Sample: Participants include 381 5th -8th graders for the extent data and 250 fifth graders  for the new data from diverse demographic and language backgrounds.

Factors: The research team will examine, in relation to reading comprehension, patterns in reading behaviors (eye-gaze, highlighting, emotional response, and digital reading behaviors) as well as reading medium (digital and paper reading), text characteristics (short vs long), and task differences (static pdf reading vs. reading in an open learning environment). Patterns are likely to be instructionally malleable, and the  texts and tasks readers or teachers choose are also malleable depending on findings.

Research Design and Methods: The research team will combine extent data analysis (within- and between-subjects design) with collection, coding, and analysis of new data (within-subjects design). Then, they will collect new data to address gaps in the literature using a within-subjects design to increase power. For example, the team's extent data includes detailed process (reading behavior) data for digital reading but less for paper reading and involves reading of a single passage for a single task (answering comprehension questions). The proposed data collection in Year 2 fills these gaps by collecting similar reading process data for digital and paper reading of multiple static, bound texts of different lengths as well as digital reading in an open learning environment. The research design is iterative such that findings each year guide later work. In Year 1, the research team will perform extent data analysis and spend time in classrooms, observing digital reading experiences and piloting data collection materials for the full data collection effort proposed to take place in Year 2. In Year 2, the research team will finish extent data analysis and collect the new data. In Year 3, they will code and begin analysis of the new data, guided by findings with the extent data from Years 1 and 2. In Year 4, they will complete analysis of the new data. This study will be pre-registered with the Registry of Efficacy and Effectiveness Studies.

Control Condition: Due to the nature of this study, there is no control condition.

Key Measures: For the extent data, key measures that include a researcher developed post-test reading comprehension developed (outcome) and content-knowledge assessment (pre-test) as well as in-the- moment reading behavior data (indicating reading process, both outcome and predictor) for eye- gaze, emotional response, highlighting, and digital reading behaviors. The research team will analyze reader characteristics as moderators (Race, ELL, preference, ability). For the new data, the team will create parallel reading passages and comprehension assessments (outcome; post-test reading comprehension) using item response theory (IRT) to allow comparable performance. They will also collect in-the-moment behaviors (see above) as outcome and predictors, including behaviors during paper reading conditions. The research team will also consider reader characteristics (see above) plus text (length) and task (static pdf vs. open learning  environment reading) characteristics as moderators.

Data Analytic Strategy: The main data-analytic strategy will involve latent variable models and machine learning to detect patterns and link the patterns to reading comprehension while considering moderators such as medium (digital or paper) and reader, text, and task characteristics. A within-and between-subjects design was used for the extent data whereas the research team will use a within-subjects design for the proposed data collection.