Differential effects of English language learner training and materials—On Our Way to English (OWE) and Responsive Instruction for Success (RISE)
Data collection approach
Student data. The primary objective of this research is to examine the effects of ELL-specific interventions on the English language achievement of English language learner students in vocabulary, fluency, reading comprehension, grammar, writing, and listening. Because REL Central is conducting the study in three states that use different statewide assessments of student achievement, a common metric was needed. This study will assess student progress using the revised IDEA Proficiency Tests, or IPT (Ballard and Tighe 2005).
Students in grades 1–4 will be tested first in the fall of year 1. Students will then be tested the following spring and tested again in the spring of year 2 (when the original sample is in grades 2–5) to assess performance after exposure to the intervention. Students entering treatment and control schools after the start of the study will also be tested on the same schedule. Because the IPT is vertically equated, it is appropriate for assessing school-level outcomes. Student demographic information on race/ethnicity, language, eligibility for free and reduced-price lunch, and primary language spoken at home will be collected by site coordinators at the beginning of each school year using a data collection summary sheet. Site coordinators will be asked to collect this information for all English language learner students, regardless of when they enter the study.
Teacher data. Three data sources will be used to assess teacher practices in teaching English language learner students: an online log, a classroom observation protocol, and an interview protocol. The online log will be used to assess pedagogical practices for teaching English language learner students as reported by teachers. The logs include checklists and rating scales derived from research findings on best practices for teaching English language learner students and require that teachers detail classroom practices, activities, and any ELL accommodations or ELL-specific strategies used during classroom instruction. REL Central researchers created the online log using checklists and rating scales derived from research findings on best practices for teaching English language learner students (see citations below). Log items address the following constructs identified through the literature on teaching English language learner students.
- One log section addresses the organization of curriculum and pedagogy. It allows for collecting data on such issues as whether the curriculum is organized around "big" questions (or themes) relevant to the students' lives (Freeman and Freeman 2000; National Council of Teachers of English 2006); whether students work collaboratively or in small groups for instruction and how such work is accomplished (Henderson and Landesman 1992; Klingler and Vaughn 2000); the frequency with which teachers expose students to oral language and the types of oral language being developed.
- A second log section addresses a teacher's use of authentic and meaningful experiences. It allows for the collection of data on whether students are involved in authentic fiction and nonfiction reading and writing experiences (National Council of Teachers of English 2006); what types of pedagogical strategies a teacher employs for reading instruction, such as reading with or for students; whether a teacher considers the student's background knowledge and interests (Darder 1993; Godina 1998; Krashen 1996; Smith 1983); and whether the teacher provides instruction and activities that serve to advance academic proficiency as well as develop English language skills (Freeman and Freeman 2000; National Council of Teachers of English 2006).
- A third log section addresses issues of assessment and examines whether teachers use student assessment data to inform or guide their teaching.
- A final log section allows for the collection of data on teacher beliefs. It covers whether teachers have high expectations for their students and faith in their abilities to be successful (Collier 1995).
Completion will take approximately 25 minutes per log entry, with teachers completing logs once per quarter.1 All teachers (treatment and control) in grades 1–5 during the study period will be asked to complete these logs electronically. The software used for the log entries will automatically enter the data into a database.
A classroom observation and interview protocol will be used to assess changes in teachers' practices and skills related to the instruction of English language learner students. It will corroborate information collected through the teacher online logs, which are self-reported. REL Central researchers adapted this observation and interview protocol by modifying the English Language Learner Classroom Observation Instrument (ELCOI; Haager et al. 2003), a 30-item moderate inference instrument that uses a Likert scale. ELCOI consists of six empirically derived subscales adapted from observational, cognitive learning, and sheltered instruction research on effective reading instruction (Englert 1984; Brophy and Good 1986; Tikunoff et al. 1991). High internal consistency (median subscale alpha of .89), moderate item-by-item interobserver agreement of 74 percent, and criterion-related validity in the high/moderate range (median coefficient of .60) are reported in the literature. These findings, which have been replicated (Baker et al. 2004), lend credibility to the use of ELCOI in the proposed intervention. Additional items were incorporated into the treatment and control group ELCOI protocols relative to the instructional setting (such as availability of ELL reading and display materials and room and space arrangements). The treatment group protocol also includes practices specific to the OWE intervention that may be observed, such as the use of the thematic unit teacher's guide and chant posters.
Researchers pilot tested adapted instruments with a small sample of teachers before use with study participants. Observations began in November 2008 for a sample of treatment and control classrooms. Trained observers observed classroom activities such as grouping arrangements, teacher pedagogy, and student activities during a language arts period two times throughout the course of the study. Depending on the grade (1–5), class periods and thus observations last between 30 and 50 minutes. Brief interviews with the classroom teacher conducted on the day of the observation focus on teachers' lesson plans, materials, and instruction during the observation. Questions on the RISE and OWE interventions will also be asked as appropriate to treatment group teachers. Teachers will be observed twice in their classrooms—once in the fall of year 1 and once in the spring of year 2.
Teacher logs, surveys, observation protocols, and interview protocols were developed and pilot tested during 2006–08 and modified following the pilot tests.
REL Central researchers collected information on teacher characteristics during study orientation sessions. Teachers were asked to complete an information sheet requesting general demographic information, such as gender and ethnicity, and also teaching experience (such as years teaching and teaching certification status) and experience teaching English language learner students (such as years teaching English language learner students, knowledge of other languages, and prior exposure to professional development for teaching English language learner students). Category definitions for these characteristics are consistent with the national consensus and available databases.
Implementation data. As this study is one of intervention "effectiveness" rather than intervention "efficacy," the examination of implementation fidelity is not of primary interest. To monitor and examine potential issues related to fidelity of implementation of the OWE and RISE programs, REL Central researchers will rely on information from the online logs, observations, and interviews, RISE training artifacts, and school-level artifacts (such as student attendance).