Project Activities
The ARMM will be developed using a seven-stage development framework. The assessment development process will begin with construct definition and assessment specification. An expert panel will be assembled to help define reading motivation constructs for adolescents and to write assessment specifications. Following this, the expert panel and reading educators will write the items. Items will then be piloted and validated with students in grades 5–12 throughout Kansas. A small-scale pilot study will also be conducted in Alameda, California. The findings of this pilot study will be used to ascertain item selection. Once this process is complete, the team will carry out a large-scale field test, prepare validation and technical documentation, and complete the documentation and dissemination of the final measures.
Structured Abstract
Setting
School districts in both Kansas and California will participate in the project.
Sample
Participants include 40 teachers and 16,800 students from diverse racial and ethnic backgrounds across grades 5-12.
Researchers will develop and validate the ARMM, a new measure of reading motivation for use with adolescents in grades 5–12. The Adaptive Reading Motivation Measures (ARMM) will use an online computer adaptive format to measure a variety of constructs related to adolescents' reading motivation. These constructs will include self-efficacy, interest, social motivation, and extrinsic and intrinsic motivation. Assessment score reports will be automatically generated for teachers to guide instruction. For this adaptive testing format, approximately 50 items will be developed to measure each of the reading motivation constructs to be targeted. Each student will receive only 4 to 6 items in order to identify their score for that construct. Teachers could use the ARMM in multiple ways, including assessing the overall motivation of their class to guide lesson plans or measuring individual levels of reading motivation to guide the types of books assigned.
Research design and methods
The ARMM will be developed using a seven-stage development framework. The seven stages are: (1) construct definition; (2) assessment specification; (3) item development; (4) small-scale pilot study and item selection; (5) large-scale field test and revisions; (6) validation and technical documentation; and (7) documentation and dissemination of the final measures. During these stages, researchers will assemble an expert panel to further define reading motivation constructs for adolescents and to write assessment specifications. The expert panel, along with reading educators, will write the items. Items will then be piloted and validated with students in grades 5–12 throughout Kansas and in Alameda, California. Researchers will develop the assessment based on motivation constructs and will complete a pilot study that represents an initial effort to collect representative responses from the items and to describe the item characteristics including item difficulties. Researchers will then conduct a large-scale field test and incorporate revisions based on the pilot study findings. Validity of the ARMM will be determined and in addition, a series of interviews with both teachers and students will be conducted in four participating schools. These interviews, along with other data and analysis, will provide evidence of the validity of the assessment. Reliability of the ARMM will also be assessed using various approaches.
Control condition
There is no control condition.
Key measures
Questionnaires and think-aloud procedures will be used to assess the perceived usability of the ARMM. Criterion-related validity will be assessed using reading scores on state standardized tests, individually administered diagnostic reading scores, and survey-reported reading behaviors identified by students, teachers, and parents.
Data analytic strategy
Multiple techniques, including confirmatory multi-dimensional Item Response Theory (IRT) and structural equation modeling will be used to verify the hypothesized factor and score reporting structures of the assessment. This occurs both within and across subgroups of students. Reliability will be assessed using several approaches, including test information functions provided by IRT and conditional standard errors of measurement. Validation of the assessment will include analysis of information gleaned from the following: (a) test administration feedback from students and teachers; (b) questionnaires about student reading profiles administered to and correlated with students, parents, and teachers; (c) relationships between reading motivation scores and reading achievement as measured by the reading achievement scores; (d) relationships between motivation and diagnostic aspects of reading achievement; and (e) studies of differential item functioning.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: The product of this project will be a fully developed and validated assessment of reading motivation for adolescents in grades 5-12 entitled the Adaptive Reading Motivation Measures (ARMM). Products will also include published reports of research findings.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
