Project Activities
The proposed research involves three phases. Phase One includes the development of the NAP, Phase Two provides psychometric validation of the NAP, and Phase Three creates standardized, easy to use training materials. In Phase One, the researchers will build upon a prototype version of the NAP, and will extend this prototype to create a carefully designed tool that has improved implementation procedures, multiple forms, a more comprehensive focus, a broadened age range, enhanced scoring procedures, and improved training procedures. Once the revisions are complete, the team will carry out a psychometric validation of the measure, administering a battery of tests including the NAP, additional narrative measures, a language task, and reading measures to approximately 300 children longitudinally with 4 assessments over 2 years. In the final phase of the research project, the team will work closely with a team of early childhood educators to develop end-user training materials, and will carry out usability testing to ensure ease of use of the tool.
Structured Abstract
Setting
The research study will be conducted in areas of Ohio and Michigan.
Sample
Over the 4 years of the project, at least 800, 3- to 6-year-old children will participate in the measurement development and validation process. In addition, a group of early childhood educators will participate in the usability testing of the NAP.
The researchers will build upon a prototype version of the NAP that has good concurrent and predictive validity and can be implemented reliably. The new version of the NAP will: (a) provide a more comprehensive focus; (b) include a short and long form; (c) adhere to rigorous methods in item development and validation; and (d) undergo usability testing to ensure its scalability. The resulting tool is intended to reach large numbers of educators at very low cost, and will help educators' to differentiate language instruction within early childhood settings to enhance children's narrative growth.
Research design and methods
Researchers will first develop the NAP, a useable and standardized measure of 3- to -6-year-old children's narrative skills. Drawing from our prior development work and the extant literature, the research team will develop, review, and revise a large assessment item pool with input from narrative experts and early childhood professionals. Once the item pool is finalized, researchers will collect and score the narratives of approximately 500 children with the full item pool. The researchers will then select items for inclusion on short and full-form versions of the NAP based on measurement analyses intended to yield optimal internal structure.. Researchers will also determine the psychometric characteristics of the NAP by establishing the validity of the NAP with a multipronged approach. Researchers will examine construct validity by examining relations of the NAP to other measures of narrative skills using both concurrent and longitudinal approaches. The research team will examine criterion-referenced validity by determining the concurrent and predictive relations of the NAP and its longitudinal trajectory to measures of language ability and reading comprehension. Lastly, researchers will ensure the usability of the NAP for a range of end users and will promote its scalability through web-based dissemination. The research team will develop easy-to-use scoring materials using item response modeling methods, and will involve a range of constituents (e.g., early childhood educators) in development of end-user training materials and inter-rater reliability tests to promote usability. The team will support the scalability of this tool by making the tool itself, along with training materials, available on the web at no cost.
Control condition
There is no control condition.
Key measures
The project team will administer four main types of measures: (1) assessment of narrative using the NAP, (2) assessment of narrative using alternative measures of narrative skills, (3) assessment of general language ability using established measures, and (4) assessment of literacy outcomes using established measures. The final version of the NAP (from Phase One) will be administered to each child. Two additional narrative assessments will be administered to each child, namely the Renfrew Bus Story and the Culatta Task as used in the Iowa epidemiological study. Three subtests of the Clinical Evaluation of Language Fundamentals–Preschool: 2 will be administered: Sentence Structure, Word Structure, and Expressive Vocabulary. Finally, children's literacy outcomes will be included for children who are 5 years and older using the Woodcock-Johnson III Tests of Cognitive Abilities.
Data analytic strategy
The design of the NAP gives careful consideration to ensuring the usability and scalability of the tool. Researchers will examine equivalence of the NAP scores across age using Differential Item Functioning (DIF) analysis. Consistent with a typical practice in DIF analysis with Rasch models, researchers will assume that the average category threshold for each factor is equal across groups. The researchers will then compare each item's category threshold parameters across groups in an analysis of variance framework.
People and institutions involved
IES program contact(s)
Products and publications
Products: The products of this project will be a fully developed and validated NAP for use by educational professionals as a measure of 3 to 6-year-old children's narrative skills. Published reports will also be produced.
Journal article, monograph, or newsletter
Bowles, R.P., Pentimonti, J.M., Gerde, H.K., and Montroy, J.J. (2014). Item Response Analysis of Uppercase and Lowercase Letter Name Knowledge. Journal of Psychoeducational Assessment, 32(2): 146-156.
Supplemental information
Co-Principal Investigators: Laura Justice and Shayne Piasta, (Ohio State University), Lori Skibbe and Mark Reckase (Michigan State University)
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
