Project Activities
The researchers will iteratively develop additional assessment items and a computer-adaptive version of CALS, called CALS-I.2cat. They will conduct a series of validity and reliability studies to ensure that the new assessment items and computer-adaptive version assess the skills of both EP and ELs with at least intermediate English proficiency.
Structured Abstract
Setting
The research will take place in schools in Texas, Massachusetts, New York, and Tennessee.
Sample
The project will include roughly 4,300 ELL and EP students in grades 4 through 8. The project will target three non-English language groups, representing the most spoken languages in schools: Spanish, Chinese, and Arabic.
Core academic language skills include a set of high-utility language skills that rely on linguistic features prevalent in academic discourse across content areas, such as knowledge of logical markers that connect ideas (nevertheless, consequently, etc.) and knowledge of structures for organizing analytic texts. The existing CALS-I.1 tests these skills using a paper-and-pencil group-administered test that takes about 50 minutes. CALS-I uses multiple choice, matching, and short written responses across nine tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentences, Identifying Definitions, Interpreting Epistemic Stance Markers, Understanding Metalinguistic Vocabulary, Writing Academic Definitions. The CAT version, CALS-I.2cat, will use the same constructs as CALS-I.1 but will have multiple advantages. First, the item bank will include more items, all of which will be appropriate and reliable for both EP students and ELL students at the intermediate level or above. Also, because it will be computer-adaptive, it will be a shorter test and allow for real-time score generation.
Research design and methods
The research will occur in three broad phases: the design phase, the final item election phase, and the validation and reliability testing phase. During the design phase, the researchers will develop item specifications and generate a 150-item test bank while also iteratively developing the computer-adaptive testing (CAT) application. They will conduct small-scale studies with students to gather information on the assessment items and CAT usability to inform further refinement. During the final item selection phase, researchers will select the final item set for the CALS-I.2cat and study initial evidence of reliability and predictive validity, and the dimensionality of the CALS-I.2cat final items. They will also explore how well CALS-I.2cat predicts outcomes for students with varying levels of English proficiency and different home languages. This final study will gather data at the beginning and end of the school year to generate reliability estimates and determine content, predictive, and convergent and discriminant validity, as well as equate and norm CALS-I.2cat against other assessments.
Control condition
There is no control condition. However, the researchers will compare scores for EP and EL students with differing levels of English proficiency and different home languages.
Key measures
To validate CALS-I.2cat, the researchers will leverage data from CALS-I, English proficiency tests (such as the Texas English Language Proficiency Assessment System, known as TELPAS), reading fluency and comprehension tests (such as the Gates-MacGinitie), and statewide English language arts and math tests.
Data analytic strategy
The researchers will use Classical Test Theory and Item Response Theory to validate the CAT version of the assessment. They will also conduct differential item functioning analyses for all items.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: The goal of this project is to develop and validate a computer-adaptive test (CAT) version of the Core Academic Language Skills Instrument (CALS-I.2cat) for use with both EP and EL students at intermediate or higher levels of English proficiency in grades 4 through 8. The research team will also publish peer-reviewed articles.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
