Description:

Co-Principal Investigator: Katherine S. Binder (Mount Holyoke College)

Purpose: The purpose of this study is to better understand the test-taking behavior of 3rd, 5th, and 8th grade students. Tests of reading comprehension (RC) are meant to measure the processes engaged in by readers when reading for understanding, yet research suggests that two individuals might engage in very different processes and earn the same score on a single test. Furthermore, the same student might earn very different scores on two different tests, with one test score suggesting the student has comprehension deficits and the other suggesting he does not. These differences occur because students' test-taking strategies vary both within a testing session, and tests vary in their characteristics (e.g., text length, text type, question type) and the reading processes that they measure.

Project Activities: The team will complete four studies using eye tracking procedures to record students' on-line reading and question reading and responding behavior. In the first study, the researchers will collect both on-line and off-line measures of RC within the same study. This will allow for the assessment of the relationship between the processes and products of RC. Across the three remaining studies the team will manipulate variables that might alter student's reading behavior, response accuracy, and the extent to which their response accuracy reflects their ability to develop a coherent representation of the text.

Products: Researchers will produce preliminary evidence of malleable factors such as test-taking strategies and test formats that are associated with assessing elementary students' comprehension skills, and information about how those factors differ as a function of students' reading achievement, working memory, and motivation. The team will also produce peer-reviewed publications.

Structured Abstract

Setting: Three school districts in Georgia will participate in the study.

Sample: Participants include approximately 700 third, fifth, and eighth grader students. Students without special education identification and whose first language is English will be recruited. Currently, 15% of students enrolled in the participating school districts are Hispanic or Latino. The racial composition of the sample is anticipated to reflect that of the school districts: 31% Black or African American, 48% White, 4% two or more races, 2 % Asian, less than 1 % American Indian/Alaska Native, and less than 1% Native Hawaiian or Other Pacific Islander.

Intervention: The research team will explore malleable factors such as test-taking strategies and test formats that are associated with assessing elementary students' comprehension skills, and information about how those factors differ as a function of students' reading achievement, working memory, and motivation. Results from these studies will have implications both for reading comprehension assessment and instruction.

Research Design and Methods: Researchers will carry out four studies to meet the project goals. In the first study the team will collect both on-line and off-line measures of RC. They will administer texts of different lengths and types as students' eye movements are recorded. A similar text administration technique will be followed in each of the three following studies, where students will read and respond to test questions online while their eye movements are simultaneously being collected. In the second study, the researchers will manipulate the test-taking strategy that students must employ. In the third study, the team will vary the question response format (i.e., multiple choice or constructive responses) and whether students have access to text when responding. In the fourth study, the researchers will manipulate the number of questions presented per screen as well as whether students are asked to provide evidence supporting their answers.

Control Condition: There is no control condition in this exploratory study.

Key Measures: Researchers will measure reading achievement (Woodcock Reading Mastery Tests, Third Edition), working memory, and motivation (Motivation Assessment Scale). Students' eye movements will be recorded while they are reading texts and reading and responding to associated reading comprehension questions. The team will measure first fixation duration, gaze duration, fixation count, and total fixation duration.

Data Analytic Strategy: Researchers will carry out mixed analyses of variance and cross-classified hierarchical linear models to answer multiple research questions across the four studies.