Project Activities
The research team developed and pilot tested the Palette, an automated tool that enables teachers to generate instructional activities designed to support ELLs as they read teacher-selected texts. During phase 1 of the project, the research team designed and developed the system, building the library of texts that was incorporated into the Palette. During phase 2, the researchers conducted a usability and feasibility study. Finally, in phase 3, they carried out a randomized pilot study. In this pilot study, the research team implemented the training with a new sample of teachers for a full school year, monitored fidelity of implementation, and evaluated the impact of the intervention on student learning.
Structured Abstract
Setting
The phase 2 usability and feasibility study took place in grade 6 through 8 classrooms in 3 schools across 2 states. The phase 3 pilot study was conducted in 6 middle and 6 high schools (grades 6 through 12) in 3 states.
Sample
The usability and feasibility study included 12 teachers in 2 middle schools. Teachers’ teaching experience ranged from 2 to 22 years, with 10 teaching science, 5 teaching English language arts (ELA), and 2 teaching social studies. No teachers had prior English language learner training in their formal education. The most common first languages among participating students were Spanish and Arabic.
The Language Muse Activity Palette is a web-based application that supports teachers in the creation of text-based content and language learning activities. It uses natural language processing capabilities to automatically generate language activities for classroom texts, targeting the improvement of language and reading skills of middle and high school English learners. The tool has a classroom management component so that teachers can create a class and management the activities and assignments delivered to individual students or the whole class.Teachers use the Palette to create activities and assignments online and the materials can be administered electronically or in traditional paper‐and‐pencil format. Teachers received professional development and training to use the Palette's activity-generation capability to customize and administer linguistically relevant activities in their classrooms that support ELLs' content understanding and language skills development.
Research design and methods
During phase 1 of the project, the research team designed and developed the system, building the library of texts incorporated into the system. They then created automatic item-generation scripts that generated customized and linguistically relevant activities for teacher use in the classroom. The team asked a variety of users who teach ELL students (ELL experts, ELL specialists, and content-area teachers) to evaluate the viability of incorporating the texts and activities into their classrooms. Concurrently, a sample of both ELL- and non-ELL students tested the activities. During phase 2, the researchers conducted a usability and feasibility study. Finally, in phase 3, they carried out a randomized pilot study in which they implemented the training with a new sample of teachers for a full school year, monitored fidelity of implementation, and evaluated the impact of the intervention on student learning.For the usability and feasibility study in phase 2, the team evaluated the implementation for feasibility and fidelity, collecting survey and reflection data from participating teachers and pre/post assessment data for students. In the phase 3 pilot study, the team carried out a formal pilot test of the impact of the intervention. The pilot study used a randomized block design in which a pair of teachers from 15 schools participated. One teacher from each pair was randomly assigned to the treatment condition and received training in the use of the intervention. Fidelity of implementation was also monitored.
Control condition
For the final pilot study, teachers in the control condition used typical instructional materials and activities.
Key measures
The main measure of student learning was the Reading Inventory and Scholastic Evaluation (RISE) reading assessment, which is part of the Study Aid and Reading Assessment (SARA) Battery. The SARA is a 45 to 60 minute, web-administered reading assessment that includes 6 subtests: word recognition and decoding, vocabulary, morphological awareness, sentence processing, efficiency of basic reading comprehension, and reading comprehension. SARA includes four parallel forms, and students are tested at both the beginning and end of the academic year, thereby generating both pre- and post-exposure scores. Other measures included the Palette Training Evaluation, Professional Experiences Survey, teacher reflection logs, and observation protocol.
Data analytic strategy
For the phase 3 pilot study, the researchers used linear regression to analyze each composite posttest score. The regression included an indicator for the student’s teacher’s treatment or control status and indicators for random assignment strata. To account for an unbalanced sample, the researchers ran additional models that controlled for student grade level and interactions between grade level and pretest scores.
Key outcomes
- The Language Muse Activity Palette was developed to assist teachers in generating instructionally appropriate student activites to support language learning for English language learners. (Madnani et al 2016, Burstein et al 2017).
- The usability and feasibility study found that all teachers completed the intervention and that most (86%) reported that activities completely or mostly fulfilled learning objectives (Burstein et al 2017).
- Pilot study results suggest that the Palette is a promising technology that teachers can implement. The intervention had mixed results on language outcomes, with a positive and statistically significant impact on comprehension and no impact on lexical skills.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Publications:
Select Publications
Burstein, J. (2020). Natural Language Processing and the Literacy Challenge. Application of Artificial Intelligence to Assessment, 77. Jiao, H. & Lissitz, R.W., IAP: Information Age Publishing: Charlotte, North Carolina.
Burstein, J., Madnani, N., Sabatini, J., McCaffrey, D., Biggers, K., and Dreier, K. (2017). Generating Language Activities in Real—Time for English Learners using Language Muse. In Proceedings of the Fourth Annual ACM Conference on Learning at Scale (Short Papers).
Burstein, J. and Sabatini, J. (2017). The Language Muse Activity Palette: Technology for Promoting Improved Content Comprehension for English Language Learners. Adaptive Educational Technologies for Literacy Instruction. Taylor & Francis eBooks.
Madnani, N., Burstein, J., Sabatini, J., Biggers, K., and Andreyev, S. (2016). Language Muse: Automated Linguistic Activity Generation for English Language Learners. In 54th Annual Meeting of the Association for Computational Linguistics. Berlin DE: Association for Computational Linguistics ( ACL ).
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
