Co-Principal Investigators: Barr, Christopher; Phillips Galloway, Emily; Miciak, Jeremy

Purpose: These researchers will develop and validate a computer-adaptive version of Core Academic Language Skills Instrument (CALS-I.1) for use with both English proficient (EP) and English learner (EL) students in grades 4 through 8. CALS-I.1 is an existing assessment that has been validated to predict EP students' reading comprehension skills based on their performance with academic vocabulary. However, CALS-I.1 was not developed for use with EL students. Research indicates that ELs typically become proficient in basic reading skills and in colloquial language skills at rates similar to their native English-speaking peers, but they lag behind in developing the academic vocabulary necessary in the upper elementary years and beyond to understand increasingly complex academic texts. This project will develop an assessment tool to help teachers identify possible vocabulary difficulties for both EP and EL students, in order to address potential reading comprehension barriers.

Project Activities: The researchers will iteratively develop additional assessment items and a computer-adaptive version of CALS, called CALS-I.2cat. They will conduct a series of validity and reliability studies to ensure that the new assessment items and computer-adaptive version assess the skills of both EP and ELs with at least intermediate English proficiency.

Products: The goal of this project is to develop and validate a computer-adaptive test (CAT) version of the Core Academic Language Skills Instrument (CALS-I.2cat) for use with both EP and EL students at intermediate or higher levels of English proficiency in grades 4 through 8. The research team will also publish peer-reviewed articles.

Structured Abstract

Setting: The research will take place in schools in Texas, Massachusetts, New York, and Tennessee.

Sample: The project will include roughly 4,300 ELL and EP students in grades 4 through 8. The project will target three non-English language groups, representing the most spoken languages in schools: Spanish, Chinese, and Arabic.

Assessment: Core academic language skills include a set of high-utility language skills that rely on linguistic features prevalent in academic discourse across content areas, such as knowledge of logical markers that connect ideas (nevertheless, consequently, etc.) and knowledge of structures for organizing analytic texts. The existing CALS-I.1 tests these skills using a paper-and-pencil group-administered test that takes about 50 minutes. CALS-I uses multiple choice, matching, and short written responses across nine tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentences, Identifying Definitions, Interpreting Epistemic Stance Markers, Understanding Metalinguistic Vocabulary, Writing Academic Definitions. The CAT version, CALS-I.2cat, will use the same constructs as CALS-I.1 but will have multiple advantages. First, the item bank will include more items, all of which will be appropriate and reliable for both EP students and ELL students at the intermediate level or above. Also, because it will be computer-adaptive, it will be a shorter test and allow for real-time score generation.

Research Design and Methods: The research will occur in three broad phases: the design phase, the final item election phase, and the validation and reliability testing phase. During the design phase, the researchers will develop item specifications and generate a 150-item test bank while also iteratively developing the computer-adaptive testing (CAT) application. They will conduct small-scale studies with students to gather information on the assessment items and CAT usability to inform further refinement. During the final item selection phase, researchers will select the final item set for the CALS-I.2cat and study initial evidence of reliability and predictive validity, and the dimensionality of the CALS-I.2cat final items. They will also explore how well CALS-I.2cat predicts outcomes for students with varying levels of English proficiency and different home languages. This final study will gather data at the beginning and end of the school year to generate reliability estimates and determine content, predictive, and convergent and discriminant validity, as well as equate and norm CALS-I.2cat against other assessments.

Control Condition: There is no control condition. However, the researchers will compare scores for EP and EL students with differing levels of English proficiency and different home languages.

Key Measures: To validate CALS-I.2cat, the researchers will leverage data from CALS-I, English proficiency tests (such as the Texas English Language Proficiency Assessment System, known as TELPAS), reading fluency and comprehension tests (such as the Gates-MacGinitie), and statewide English language arts and math tests.

Data Analytic Strategy: The researchers will use Classical Test Theory and Item Response Theory to validate the CAT version of the assessment. They will also conduct differential item functioning analyses for all items.

Related IES Projects: Catalyzing Comprehension Through Discussion and Debate (R305F100026)