Co-Principal Investigator: Sabatini, John

Purpose: The purpose of this project is to develop a technology-rich instructional program that will improve the language skills and comprehension of content-area texts of English language learners (ELLs). The project is motivated by the awareness that ELLs who may already be struggling to acquire grade-level English language skills may be additionally challenged by the new text demands likely to enter the curriculum as a result of the Common Core State Standards. This project, informed by a prior IES grant (Language Muse — teacher professional development (TPD) project), aims to leverage linguistic analysis tools (Natural Language Processing) to generate student activities appropriate for instructional support of content knowledge acquisition (e.g., science text) and language skills. In this project, researchers will develop and test the Language Muse Activity Palette (Palette), an automated tool that will enable teachers to generate instructional activities designed to support ELLs as they read teacher-selected texts.

Project Activities: The research team will complete three phases of development and testing. During Phase 1, the research team will design and develop the system, building the library of texts that will be incorporated into the Palette. The researchers will then create automatic item-generation scripts which will generate customized and linguistically-relevant activities for teachers to use in their classrooms. Then, the team will ask a variety of users who teach ELL students (ELL experts, ELL specialists, and content-area teachers) to evaluate the viability of incorporating the texts and activities into their classrooms. Concurrently, a sample of both ELL- and non-ELL students will try out the activities. During Phase 2, the researchers will further test the developed materials, with selected texts and activities, along with enhanced teacher professional development (TPD) training materials. In this second pilot study, the researchers will examine the feasibility and fidelity of the intervention. Finally, in Phase 3, researchers will carry out a final pilot study. They will implement the training with a new sample of teachers for a full year in one subject area at one grade level (8th grade science), monitor fidelity of implementation, and evaluate the impact of the intervention on student learning.

Products: Products for this project include the fully developed Language Muse Activity Palette and evidence of promise of the intervention. Researchers will also produce peer reviewed publications.

Structured Abstract

Setting: This study will take place in New Jersey and Texas middle school urban classrooms, and California rural middle school classrooms.

Sample: In Phase 1 and Phase 2 of the project, approximately 10 middle school teachers, and between 300-1000 sixth to eighth grade students will participate. The final pilot will include 30 middle-school teachers and their eighth grade students, both ELLs and others, totaling about 1,200 students, depending on class size. The middle schools in the recruited schools have relatively high populations of ELLs (about 20%).

Intervention: The Language Muse Activity Palette will be a web-based application that supports teachers in the creation of text-based content and language learning activities. Upon uploading a text (either provided by the ETS team, from a teacher's local hard drive, or from the web), linguistic features in that text are automatically analyzed. Based on the prevalence of features in the text across vocabulary, sentence, and discourse categories, language instructional activities are generated and recommended. The Palette then steps the teacher through a structured-interview, offering guidance about selection and development of activities appropriate for her ELLs. The teacher can choose to design activities for different modes of classroom delivery including traditional paper-and-pencil, interactive whiteboard, or electronically-delivered activities, where students interact directly with the system and their activity results can be accessed by the teacher. Teachers will receive professional development and training to use the Palette's activity-generation capability to customize and administer linguistically-relevant activities in their classrooms that support ELLs' content understanding and language skills development.

Research Design and Methods: The research team will complete the design, development, and testing of the Palette following an iterative development process that will unfold over three phases. In Phase 1, the team will focus on the initial design and development of the Palette. The team will design and develop the system, build the text Library amenable to text analyses, and build step-by-step automatic-item generation scripts and associated activities for the text Library. The team will then ask a small sample of ELL experts, ELL specialists, content area teachers, and those who teach ELL students to use the Palette to generate activities. The researchers will also pilot those activities with a target sample of middle school ELL and non-ELL students who have a range of English reading and language ability levels. The outcome of Phase 1 will include a Library of core texts; an associated library of vocabulary, sentence, and discourse activities vetted by experts, teachers, and students; and the technical infrastructure to generate activities using the Palette.

During Phase 2, researchers will select sets of prepared texts and activities for use in Units 1 (vocabulary), 2 (sentence), and 3 (discourse) of the Instructional Program; and develop the enhanced TPD training modules aligned to the Instructional Program and implement the training and program. The team will evaluate the implementation for feasibility and fidelity in the fall semester of the second project year. Based on implementation data, the team will revise the TPD, instruction, or the software application, as necessary, and conduct a second cycle of implementation and revision in spring semester of the second year. The outcome of Phase 2 will include a fully implementable program including TPD and Instructional Program and evidence of feasibility in a trial implementation.

In Phase 3, the team will carry out a formal pilot test of the intervention, implementing the TPD and program for a full year in one subject area at one grade level. The pilot study will use a randomized block design, where a pair of teachers from each of 15 schools will participate. One teacher will be randomly assigned to the treatment condition, and the teacher will receive training in the use of the intervention. Fidelity of implementation will also be monitored. The outcome of Phase 3 will be evidence of promise in promoting student learning outcomes.

Control Condition: Teachers in the control condition will use typical instructional materials and activities.

Key Measures: The main measure of student learning will include scores on the Study Aid and Reading Assessment (SARA) Battery. The SARA is a 45-60 minute, web-administered reading assessment that includes six subtests: word recognition and decoding, vocabulary, morphological awareness, sentence processing, efficiency of basic reading comprehension, and reading comprehension. The Battery includes four parallel forms, and students are tested at both the beginning and end of the academic year, thereby generating both pre- and post-exposure scores. Additionally, the team will examine the science test scores from the state assessments.

Data Analytic Strategy: The analysis will include a hierarchical linear analysis, examining the effect of the intervention on students, nested within classrooms. Possible covariates may include the prior year's state assessment of both English Language Arts and mathematics scores.

Products and Publications

Book chapter

Burstein, J. and Sabatini, J. (2017). The Language Muse Activity Palette: Technology for Promoting Improved Content Comprehension for English Language Learners. Adaptive Educational Technologies for Literacy Instruction. Taylor & Francis eBooks.

Proceeding

Burstein, J., Madnani, N., Sabatini, J., McCaffrey, D., Biggers, K., and Dreier, K. (2017). Generating Language Activities in Real—Time for English Learners using Language Muse. In Proceedings of the Fourth Annual ACM Conference on Learning at Scale (Short Papers).

Madnani, N., Burstein, J., Sabatini, J., Biggers, K., and Andreyev, S. (2016). Language Muse: Automated Linguistic Activity Generation for English Language Learners. In 54th Annual Meeting of the Association for Computational Linguistics. Berlin DE: Association for Computational Linguistics ( ACL ).