Description:

Co-Principal Investigators: Randall Penfield (University of North Carolina); Ximena Dominguez (SRI)

Purpose: The purpose of this project is to develop and validate an Item Response Theory (IRT) – based, computer adaptive test (CAT) to assess Latino preschoolers’ science knowledge and skills. The research team will extend current IES-funded work (Lens on Science: Development and Validation of a Computer-Administered, Adaptive, IRT-Based Science Assessment for Preschool Children) that is in the final stages of developing and validating a touch screen computer-adaptive child assessment of science readiness, called Lens on Science, for use with English-speaking, preschool children.  Researchers will create a parallel anchor linked Spanish version of the assessment, called Enfoque en Ciencia, which is valid for use with Latino children.  In this 4-year measurement project, the Enfoque en Ciencia measure will be used to assess Spanish-speaking children’s knowledge in three science content domains (Life Science, Earth and Space Science, and Physical Science) and eight science practices (observing, describing, comparing, questioning, predicting, experimenting, reflecting, and cooperating).

Project Activities: Specific activities of the project are: (1) to develop a culturally relevant and linguistically comparable (across different dialects) preschool science assessment for use with Spanish-speaking preschool children; (2) equate Enfoque en Ciencia (Spanish version) and Lens on Science (English version) assessments on a common scale; (3) establish criterion-related validity; and (4) generate a user-friendly test manual, scoring procedures, and documentation for administering the science assessment in either Spanish or English. In the first year of the project, researchers will translate the Lens on Science audio material into three different Spanish dialects and create a “consensus” translation to be used during the first stage of pilot testing. Then, in the second and third years, researchers will follow an iterative process of data collection both in Miami, Florida and San Mateo, CA. In the final year of the project, the research team will administer Spanish language and mathematics measures that evaluate the reliability and validity of the fully adaptive Spanish assessment. Finally, researchers will generate a user-friendly test manual, scoring procedures, and documentation for administering the science assessment in either Spanish or English.

Products: The products for this project will be a parallel Spanish version of Lens of Science. Researchers will produce a fully developed and validated adaptive Spanish assessment (Enfoque en Ciencia) of Spanish-speaking children’s knowledge in three science content areas and eight science practices. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: This project will take place in Miami, FL and San Mateo, CA.

Sample: Participants include parents, teachers, Spanish-speaking children enrolled in preschool programs. Initial Phase 2 pilot testing includes 1,500 Spanish-speaking children (1,000 in Miami and 500 in San Mateo). The sample for the Phase 3 pilot testing is estimated to be 340 children in Miami and 160 children in San Mateo.   Phase 4 pilot testing will involve a sample of 300 Spanish-speaking children from Miami and California.

Assessment: The Enfoque en Ciencia measure will be used to assess Spanish-speaking children’s knowledge in three science content domains (life Science, Earth and Space Science, and Physical Science) and eight science practices (observing, describing, comparing, questioning, predicting, experimenting, reflecting, and cooperating).  The Enfoque en Ciencia measure will be equated with Lens on Science, the English version of the science assessment. At the end of the study, the fully developed Enfoque en Ciencia measure will include the following components:  (1) a computer adaptive test with a test bank of 400 items covering the three science content domains and the eight science practices; and (2) a technical manual with a description of psychometric properties for both Lens on Science and Enfoque en Ciencia, administration instructions, considerations for reporting, information about score reliability and validity, and evidence of the accuracy of translation for Enfoque en Ciencia. The measure will be developed as a computer adaptive test.  The test will be administered to children via a touch screen computer.

Research Design and Methods: The measurement development work will occur in four phases. The first year/Phase 1 of this 4-year project will include translating the Lens on Science audio material into three different Spanish dialects followed by the work of an expert advisory panel to create a “consensus” translation to be used during the first stage of pilot testing. In Year 2/Phase 2, an initial pilot test will occur with a sample of 1,500 children recruited from preschool programs in Miami and San Mateo. To pilot test the initial bank of 400 items developed during Year 2, children will receive a non-adaptive Enfoque en Ciencia assessment where a random set of 80 of the total 400 items will be administered to each examinee in the pilot test sample.  In Years 2 and 3, the project will involve an iterative process of data collection both in Miami, FL and San Mateo, CA with a Spanish-speaking, preschool population, evaluating item properties within an IRT framework including item bias by gender and dialect, and revising items as needed. At the end of Year 2/Phase 2, the researchers will conduct analyses to examine the item-level parameters of the test items and equate the Enfoque en Ciencia test items with the Lens on Science test items. In Year 3/Phase 3, additional pilot testing will continue as items are revised. Seventy new items will be included in Phase 3 pilot testing.  Additional item-level analyses will be conducted and final equating of test items across the Spanish and English versions of the science assessment will be completed.

The final year/Phase 4 of the project includes final pilot testing and the collection of child assessment data to evaluate the validity of a fully adaptive Spanish assessment.  To examine criterion-related validity, researchers will administer Spanish assessments across the content areas of mathematics, language, and emergent literacy to participating children. Additionally, teacher reports of classroom behavior will be obtained. Finally, the research team will generate a user-friendly test manual, scoring procedures, and documentation for administering the science assessment in either Spanish or English. The technical manual for Enfoque en Ciencia will be developed and this manual will contain a description of the properties of both Lens on Science and Enfoque en Ciencia, administration instructions, considerations for reporting, information concerning score reliability and validity.  Evidence of the accuracy of translation for Enfoque en Ciencia will also be included.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: The research team will ask parents and teachers to report on children's language exposure and language use in the home, preschool classroom, and community contexts. Parents and teachers will be asked to complete the CECR-DLL Family and Teacher Questionnaire.
The research team will use two subtests from the PreLas (Simon Says and Art Show in both Spanish and English) to screen for language dominance and language proficiency. Researchers will examine criterion-related validity with measures of language and emergent literacy. The four core subtests of the Woodcock- Muñoz Language Survey- Revised, Normative Update will be administered to capture children’s linguistic and emergent literacy abilities, beyond the information provided by the PreLAS screeners about communicative language.  To examine the association between children’s science and mathematics learning, children’s responses will be compared between Enfoque en Ciencia and Problemas Aplicadas, a normed Spanish mathematics subtest of the Woodcock-Muñoz Batería III.  Children's classroom behavior will be assessed using the Adjustment Scales for Preschool Intervention.

Data Analytic Strategy: Analyses will be conducted to examine the psychometric properties of the measure and equate the Enfoque en Ciencia measure with the Lens on Science measure.The psychometric properties of the 400 items will be evaluated through consideration of (a) the estimated Rasch item difficulty parameter; (b) the fit of the data to the Rasch model (using infit and outfit measures); (c) an index of item discrimination provided by the correlation between the item response and the estimated ability value (as provided by the Winsteps computer program); and (d) indices of differential item functioning, which (under the Rasch model) concerns whether the item difficulty parameter varies across different sub-populations. The non-equivalent groups anchor test equating design will be used to equate the Spanish (Enfoque en Ciencia) and English (Lens on Science) versions of this science assessment tool.

Researchers will use validity analyses and the correlations among the three subscale scores (Life Sciences, Earth and Space Sciences, and Physical and Energy Sciences), with the total score, will be evaluated for uni-dimensionality and evidence of construct validity.  Correlations with the math language and behavior measures will be examined. Multilevel modeling analyses will be conducted to examine the amount of variance at the child level (with gender, age and dialect serving as predictors) versus the amount of variance at the classroom level. The estimated standard error of the ability estimates obtained in the Phase 4 Pilot Testing of Enfoque en Ciencia will be used to inform the measurement error and reliability of the ability estimates.

Related IES Projects:Lens on Science: Development and Validation of a Computer-Administered, Adaptive, IRT-Based Science Assessment for Preschool Children (R305A090502)

Products and Publications

Book chapter

Greenfield D.B. (2015). Assessment in Early Childhood Science Education. Research in Early Childhood Science Education (pp. 353–380).