Co-Principal Investigator: Adams, Ashley

Purpose: The purpose of this project is to create a complementary Spanish version of the psychometrically robust English Assessment-to-Instruction (A2i) computer adaptive, progress monitoring assessments. The new instrument will be called A2i-Adquisición de Lectura en Español (A2i-ALE) and it will be 1) validated with a gold standard measure of Spanish literacy skills; 2) psychometrically comparable to the A2i English assessments, monitoring language and literacy learning in both languages; 3) able to monitor students' progress in language and literacy within a single school year and across school years from PK to 3rd grade; and 4) able to be used validly and reliably by practitioners (e.g., teachers, instructional coaches). The A2i-ALE will allow teachers to track literacy growth in dual language learners (DLLs) in grades PK–3rd.

Project Activities: There are three phases of the project: development, pilot study, and longitudinal study. In the development phase, the research team will iteratively develop and refine items, create two testlets for each grade, and prepare the Spanish assessment platform. In the pilot study phase, the team will collect A2i English and A2i-ALE data from 200 students at each grade level (PreK-3rd) to initially examine item difficulty and dimensionality. At the end of the pilot study, the goal is to have initial item difficulty parameter estimates estimated with a 1PL model that will allow for the creation of an adaptive algorithm for the CAT. In addition, participating teachers will also review the items administered and complete a questionnaire to gather feedback related to their appropriateness, perception of alignment between item difficult and target grade, and perception of utility of information to be gained from having comparable English and Spanish literacy assessments. In the longitudinal study phase (Years 2-4), the research team will conduct a three-year validity, linking, and scaling study.

Products: The research team will produce the Spanish A2i-ALE progress monitoring assessments for students in PreK–3rd using a computerized adaptive assessment platform. The assessments will be appropriate for use independently or as part of the larger A2i system. In addition, the research team will disseminate findings via conference presentations, peer-reviewed publications, and professional development.

Structured Abstract

Setting: The research team will pilot A2i-ALE with DLL students from the East (New York) and West (Southern California) coasts, to ensure a sample that is diverse in both region and Spanish dialect. In the longitudinal phase, the team will test the assessments with students in Southern California.

Sample: Participants include a piloting sample of 1,000 DLLs from PreK–3rd grade classrooms in California and New York for the item development and calibration work. In the subsequent longitudinal phase, the team will recruit an additional 700 DLLs from schools in Southern California and will follow them longitudinally for 1-2 years depending on grade and year of entry into the study. The longitudinal sample will allow for continued calibration, vertical scaling, and validation. A subset of teachers will also participate in questionnaires and focus groups to determine face validity, and a fidelity of administration study in Year 4.

Instrument: The proposed A2i-ALE will consist of three assessments to measure Spanish language and literacy—Palabras en Parejas [Word Matching]; Conocimiento de Letras y Ortografía [Letter Knowledge and Phonics]; and Leyendo para Comprensión [Reading for Comprehension]. A2i-ALE will validly measure Spanish language and literacy skills within and across school years and be psychometrically comparable to the English A2i assessments. Given its computerized adaptive platform, A2i-ALE will be quick to administer (7-12 minutes per subtest), and students' results will be immediately available to teachers, easily shared with parents, and presented in graphs and tables that have been co-designed with our school partners.

Research Design and Methods: The research team will initially develop items that are psychometrically comparable to English items in terms of estimated difficulty and letter, word, and sound frequency. A content area expert panel will approve items and screen for dialect neutrality. A pilot study in Year 1 will involve 200 students each in PreK-3rd grade taking a version of the test that is not yet adaptive. The research team will distribute new items across two testlets that they will randomly assign to students and include items of systematically varying difficulties. The research team will use a longitudinal cohort sequential design in Years 2-4. Longitudinal data collection will involve three administrations of the computer-adaptive A2i-ALE and A2i-English assessments per year. In Year 2, the team will administer an additional gold standard validation battery. They will iteratively improve the adaptive test algorithm, the precision of item difficulty estimates, and practitioner score reports during Years 2-4. In Year 4, in addition to the final year of longitudinal data, the research team will complete a small practitioner fidelity study to ensure teachers can administer the assessments validly and obtain reliable results without researcher support.

Control Condition: There is no control condition in this study.

Key Measures: Measure development will focus on A2i-ALE Spanish item development and item level statistics including Rasch difficulty and standard error of measurement. The validation battery consists of the Vocabulario Sobre Dibujos [Picture Vocabulary] from the Woodcock- Johnson Tests of Oral Language and the Identificación de letras y palabras [Letter-Word ID] and Comprensión de Textos [Passage Comprehension] subtests from the Woodcock-Munoz Batería IV. Teacher questionnaires will gather formal feedback on face validity. A fidelity rubric will allow for tracking of valid assessment administration by practitioners.

Data Analytic Strategy: The research team will conduct initial estimation of item difficulty and dimensionality via a one-parameter logistic model after the pilot study. In addition, they will iteratively estimate the adaptive algorithm using the two-parameter logistic model with additional item responses gathered during the longitudinal study. They will use a 4-parameter non-linear regression equation to link A2i-ALE and the gold standard measures, allowing for derivation of age and grade equivalents. They will accomplish vertical scaling using an item-anchored design by examining change in ability estimates using growth models. In addition, the research team will assess psychometric comparability to A2i-English assessments by comparing test information curves and ranges of item difficulties.

Cost Analysis: The research team will conduct a comprehensive analysis taking into account various factors, such as the cost to the school to purchase as well as potential expenditures for personnel, facilities, equipment, materials, and training.

Related Projects: Making Individualized Literacy Instruction Available to All Teachers: Adapting the Assessment to Instruction (A2i) Software for Multiple Real-World Contexts (R305A160404)