Co-Principal Investigators: Kingston, Neal; Brasseur, Irma; Davis, Marcia; Tonks, Stephen

Purpose: Because reading motivation has the potential to impact literacy achievement, learning about and measuring reading motivation is crucial to designing interventions and measuring student response to those interventions. Unfortunately, there are few valid, reliable measures of reading motivation that can inform instructional decisions. Currently, measures of reading motivation are targeted to young children or lack good psychometric properties. Researchers propose to develop and validate a new measure of reading motivation called the Adaptive Reading Motivation Measures (ARMM) for use with adolescents in grades 5–12. The ARMM will consist of multiple measures that will build from an established theoretical framework of reading motivation. The ARMM will measure the multiple constructs that are hypothesized to make up reading motivation such as self-efficacy, intrinsic motivation, extrinsic motivation, social goals, reading avoidance, and reading interests. By using an online computer adaptive format, the ARMM will simultaneously increase reliability while reducing the time needed to take the assessment. Once administered, assessment score reports will immediately be available to teachers, enabling them to understand each student's motivation, plan activities, and structure their classrooms accordingly. Researchers implementing reading motivation interventions could also gain information by using the proposed measures.

Project Activities: The ARMM will be developed using a seven-stage development framework. The assessment development process will begin with construct definition and assessment specification. An expert panel will be assembled to help define reading motivation constructs for adolescents and to write assessment specifications. Following this, the expert panel and reading educators will write the items. Items will then be piloted and validated with students in grades 5–12 throughout Kansas. A small-scale pilot study will also be conducted in Alameda, California. The findings of this pilot study will be used to ascertain item selection. Once this process is complete, the team will carry out a large-scale field test, prepare validation and technical documentation, and complete the documentation and dissemination of the final measures.

Products: The product of this project will be a fully developed and validated assessment of reading motivation for adolescents in grades 5–12 entitled the Adaptive Reading Motivation Measures (ARMM). Products will also include published reports of research findings.

Structured Abstract

Setting: School districts in both Kansas and California will participate in the project.

Population: Participants include 40 teachers and 16,800 students from diverse racial and ethnic backgrounds across grades 5–12.

Intervention: Researchers will develop and validate the ARMM, a new measure of reading motivation for use with adolescents in grades 5–12. The Adaptive Reading Motivation Measures (ARMM) will use an online computer adaptive format to measure a variety of constructs related to adolescents' reading motivation. These constructs will include self-efficacy, interest, social motivation, and extrinsic and intrinsic motivation. Assessment score reports will be automatically generated for teachers to guide instruction. For this adaptive testing format, approximately 50 items will be developed to measure each of the reading motivation constructs to be targeted. Each student will receive only 4 to 6 items in order to identify their score for that construct. Teachers could use the ARMM in multiple ways, including assessing the overall motivation of their class to guide lesson plans or measuring individual levels of reading motivation to guide the types of books assigned.

Research Design and Methods: The ARMM will be developed using a seven-stage development framework. The seven stages are: (1) construct definition; (2) assessment specification; (3) item development; (4) small-scale pilot study and item selection; (5) large-scale field test and revisions; (6) validation and technical documentation; and (7) documentation and dissemination of the final measures. During these stages, researchers will assemble an expert panel to further define reading motivation constructs for adolescents and to write assessment specifications. The expert panel, along with reading educators, will write the items. Items will then be piloted and validated with students in grades 5–12 throughout Kansas and in Alameda, California. Researchers will develop the assessment based on motivation constructs and will complete a pilot study that represents an initial effort to collect representative responses from the items and to describe the item characteristics including item difficulties. Researchers will then conduct a large-scale field test and incorporate revisions based on the pilot study findings. Validity of the ARMM will be determined and in addition, a series of interviews with both teachers and students will be conducted in four participating schools. These interviews, along with other data and analysis, will provide evidence of the validity of the assessment. Reliability of the ARMM will also be assessed using various approaches.

Control Condition: There is no control condition.

Key Measures: Questionnaires and think-aloud procedures will be used to assess the perceived usability of the ARMM. Criterion-related validity will be assessed using reading scores on state standardized tests, individually administered diagnostic reading scores, and survey-reported reading behaviors identified by students, teachers, and parents.

Data Analytic Strategy: Multiple techniques, including confirmatory multi-dimensional Item Response Theory (IRT) and structural equation modeling will be used to verify the hypothesized factor and score reporting structures of the assessment. This occurs both within and across subgroups of students. Reliability will be assessed using several approaches, including test information functions provided by IRT and conditional standard errors of measurement. Validation of the assessment will include analysis of information gleaned from the following: (a) test administration feedback from students and teachers; (b) questionnaires about student reading profiles administered to and correlated with students, parents, and teachers; (c) relationships between reading motivation scores and reading achievement as measured by the reading achievement scores; (d) relationships between motivation and diagnostic aspects of reading achievement; and (e) studies of differential item functioning.