Differential effects of English language learner training and materials—On Our Way to English (OWE) and Responsive Instruction for Success (RISE)

Regional need and study purpose

English language learner students (ELLs) are the fastest growing segment of the student populations in many states. Projections suggest that they will make up more than 40 percent of school-age children by 2030 (Thomas and Collier 2001). This growth has boosted demand for teachers to address the needs of English language learner students (Hill and Flynn 2004) and to ensure that they have the same opportunity to learn as their native English-speaking peers (Herman and Abedi 2004). Yet the majority of teachers from urban fringe/large towns (67 percent), central cities (58 percent), and rural locales (82 percent) report that they have never participated in professional development for addressing the needs of students learning English as a second language (Lewis et al. 1999).

This randomized control trial study by Regional Educational Laboratory (REL) Central responds to these identified needs for teacher professional development and ELL-specific teaching strategies. It will examine the impact on student achievement of using ELL-specific materials in the classroom, in combination with teacher professional development aligned with those materials.

The programs chosen for this study are Responsive Instruction for Success in English (RISE), a professional development program for teachers of English language learners, and the ELL-specific instructional materials On Our Way to English (OWE). The study will provide an unbiased estimate of the impact of RISE and OWE on student academic achievement. It will also determine whether training in RISE and the use of OWE is effective in raising teachers' knowledge about and skill in teaching English language learners.

The study addresses a primary and secondary research question, both of them confirmatory. The primary research question is:

• Does implementation of RISE in conjunction with the use of OWE have a significant impact on the acquisition of English language skills in vocabulary, fluency, reading comprehension, grammar use, writing, and listening?

The secondary research question is:

• Does the use of OWE in combination with RISE participation change teacher pedagogical practices, reflected in teacher behaviors and skills related to English language learner students?
• Do teachers connect principles of learning theory to second language development, and if so, how?
• Do teachers use small group instruction for reading and writing with their English language learner students and if so, how?
• Do teachers use student assessment data to inform or guide their teaching, and if so, how?

The study will contribute to understanding the effects of professional development aligned with curricular interventions on the academic achievement of English language learner students. Although this study will provide an unbiased estimate of the effects of the professional development intervention and the curricular intervention under examination, these findings will relate only to the combined effect of two particular interventions.

Intervention description

RISE is a professional development program of training and support for teachers of English language learner students. Its content aligns with Fillmore and Snow's (2000) proposal for what teachers need to know about language to help English language learner students meet the high levels of language proficiency required for success in school. OWE is a comprehensive, three-component program for elementary classrooms (kindergarten through grade 5) designed to provide English language learner students with comprehensive literacy instruction and standards-based content area information and to advance their English oral language development. It provides teachers with a structure, sequence, and set of materials for language and literacy instruction for English language learner students. RISE was designed to complement the classroom program by providing teachers with sustained adult learning opportunities to become familiar with and understand the content of the OWE classroom program, the rationale for its structure, and practical strategies for its use.

OWE and RISE have been widely used as standalone products. State-level data from their publisher reveals that RISE professional development courses have been offered in every state in the country and that since 2000 about 3,400 schools in more than 1,400 districts across the United States have purchased RISE, OWE, or both.

The research-based evidence on the effectiveness of OWE materials is mixed. In a four-month quasi-experimental study the pre-to-post reading achievement gains for second and fourth graders in OWE and non-OWE classrooms were compared using the Language Assessment Scales (LAS; Educational Research Institute of America 2004). Grade 2 students in OWE classrooms scored significantly higher on most sections of the LAS than students in other classrooms did. But grade 4 students in OWE classrooms scored the same or only slightly higher on different sections of the assessment. An addendum report on a Nevada school found similar results (Marketing Works 2005).

In a randomized controlled trial researchers examined reading achievement gains for second and fourth graders in bilingual and English Immersion ELL programs using the Standardized Test for Assessment of Reading (STAR; Renaissance Learning 2003) and the oral IDEA Proficiency Tests (IPT; Ballard and Tighe 2001). There were no statistically significant differences in STAR achievement between students who received OWE curriculum and those who did not in either grade 2 or grade 4. The researchers noted, however, larger overall gains in scores for students from the English immersion classroom using OWE than for students in bilingual classrooms using OWE. This indicates a potential interaction with the structure of instruction delivery. The oral IPT yielded encouraging results: 16 percent of students were considered fluent in English at pretest in OWE classrooms, compared with 44 percent at posttest; 26 percent of fourth graders were fluent at pretest in OWE classrooms, compared with 74 percent at posttest.

Little empirical evidence has been collected on the effectiveness of RISE. When this study was proposed, it was anticipated that the results of other research examining changes in teacher knowledge and skills as a result of RISE participation, under way at the time, would be available to inform this study. But that research was discontinued because of lack of funding. Although not produced specifically to support OWE, RISE can be used in conjunction with OWE. Harcourt Achieve, the developer of both interventions, is encouraging this combination, as illustrated through the use of videotaped lessons of teachers and students using OWE materials as part of RISE training. No research has been conducted to examine the impact of using the OWE program materials in conjunction with the RISE professional development course.

Study design

The impact of the RISE and OWE interventions is being evaluated through a randomized control trial, with schools randomly assigned to a treatment or control condition. Random assignment at the school level was considered appropriate for this study for three reasons:

• All teachers in a school participate in the RISE professional development program.
• The study spans two years, and assignment at the teacher level could have resulted in students being assigned to teachers who were not part of the study in year 2 and were therefore neither trained nor using the materials.
• Teacher assignment within a school posed a potential threat of contamination, since it was possible for treatment and control teachers to work together.

A total of 53 elementary schools were recruited and randomly assigned to the study treatment and control conditions at a ratio of 2:1. The schools, distributed across 13 districts in three states, represent a diverse sample of rural, suburban, and urban schools—and small and large schools. Teachers and their ELL students in grades 1–5 from each of the schools are the study participants. The sample of 53 schools is projected to provide sufficient statistical power (›.80) to detect an impact of approximately .35 standard deviation on student achievement and .50 standard deviation on teacher outcomes.

Each school has a site coordinator, responsible for on-site monitoring of data collection activities and coordinating site observation logistics. In treatment schools this person becomes a "trained trainer," who receives RISE professional development training from Harcourt Achieve and delivers the training to colleagues. Site coordinators and teachers in treatment schools are expected to complete the RISE training during the first school year. In addition, treatment school teachers are trained in the OWE program during year 1 of the study. During the half-day OWE training teachers receive all necessary OWE materials, which they are expected to use in their classrooms for at least 30 minutes a day. Because the study spans two years, REL Central asks treatment group teachers to become accustomed to using the materials in their classrooms in the first year and to fully implement them in the second year. Teachers in the control group participate in regular professional development activities and teach their students in the same manner as in previous years. Control school teachers will receive the RISE professional development and OWE materials and training at the end of the study.

Key outcomes and measures

This study is expected to provide unbiased estimates of the impact of the combination of RISE, professional development intervention, and OWE, a curriculum intervention on teacher and student outcomes. The primary student outcome is academic achievement on the revised IDEA Proficiency Tests. Teacher outcomes include changes in teaching practices relative to English language learner students.

Student data. The primary objective of this research is to examine the effects of ELL-specific interventions on the English language achievement of English language learner students in vocabulary, fluency, reading comprehension, grammar, writing, and listening. Because REL Central is conducting the study in three states that use different statewide assessments of student achievement, a common metric was needed. This study will assess student progress using the revised IDEA Proficiency Tests, or IPT (Ballard and Tighe 2005).

Students in grades 1–4 will be tested first in the fall of year 1. Students will then be tested the following spring and tested again in the spring of year 2 (when the original sample is in grades 2–5) to assess performance after exposure to the intervention. Students entering treatment and control schools after the start of the study will also be tested on the same schedule. Because the IPT is vertically equated, it is appropriate for assessing school-level outcomes. Student demographic information on race/ethnicity, language, eligibility for free and reduced-price lunch, and primary language spoken at home will be collected by site coordinators at the beginning of each school year using a data collection summary sheet. Site coordinators will be asked to collect this information for all English language learner students, regardless of when they enter the study.

Teacher data. Three data sources will be used to assess teacher practices in teaching English language learner students: an online log, a classroom observation protocol, and an interview protocol. The online log will be used to assess pedagogical practices for teaching English language learner students as reported by teachers. The logs include checklists and rating scales derived from research findings on best practices for teaching English language learner students and require that teachers detail classroom practices, activities, and any ELL accommodations or ELL-specific strategies used during classroom instruction. REL Central researchers created the online log using checklists and rating scales derived from research findings on best practices for teaching English language learner students (see citations below). Log items address the following constructs identified through the literature on teaching English language learner students.

• One log section addresses the organization of curriculum and pedagogy. It allows for collecting data on such issues as whether the curriculum is organized around "big" questions (or themes) relevant to the students' lives (Freeman and Freeman 2000; National Council of Teachers of English 2006); whether students work collaboratively or in small groups for instruction and how such work is accomplished (Henderson and Landesman 1992; Klingler and Vaughn 2000); the frequency with which teachers expose students to oral language and the types of oral language being developed.
• A second log section addresses a teacher's use of authentic and meaningful experiences. It allows for the collection of data on whether students are involved in authentic fiction and nonfiction reading and writing experiences (National Council of Teachers of English 2006); what types of pedagogical strategies a teacher employs for reading instruction, such as reading with or for students; whether a teacher considers the student's background knowledge and interests (Darder 1993; Godina 1998; Krashen 1996; Smith 1983); and whether the teacher provides instruction and activities that serve to advance academic proficiency as well as develop English language skills (Freeman and Freeman 2000; National Council of Teachers of English 2006).
• A third log section addresses issues of assessment and examines whether teachers use student assessment data to inform or guide their teaching.
• A final log section allows for the collection of data on teacher beliefs. It covers whether teachers have high expectations for their students and faith in their abilities to be successful (Collier 1995).

Completion will take approximately 25 minutes per log entry, with teachers completing logs once per quarter.¹ All teachers (treatment and control) in grades 1–5 during the study period will be asked to complete these logs electronically. The software used for the log entries will automatically enter the data into a database.

A classroom observation and interview protocol will be used to assess changes in teachers' practices and skills related to the instruction of English language learner students. It will corroborate information collected through the teacher online logs, which are self-reported. REL Central researchers adapted this observation and interview protocol by modifying the English Language Learner Classroom Observation Instrument (ELCOI; Haager et al. 2003), a 30-item moderate inference instrument that uses a Likert scale. ELCOI consists of six empirically derived subscales adapted from observational, cognitive learning, and sheltered instruction research on effective reading instruction (Englert 1984; Brophy and Good 1986; Tikunoff et al. 1991). High internal consistency (median subscale alpha of .89), moderate item-by-item interobserver agreement of 74 percent, and criterion-related validity in the high/moderate range (median coefficient of .60) are reported in the literature. These findings, which have been replicated (Baker et al. 2004), lend credibility to the use of ELCOI in the proposed intervention. Additional items were incorporated into the treatment and control group ELCOI protocols relative to the instructional setting (such as availability of ELL reading and display materials and room and space arrangements). The treatment group protocol also includes practices specific to the OWE intervention that may be observed, such as the use of the thematic unit teacher's guide and chant posters.

Researchers pilot tested adapted instruments with a small sample of teachers before use with study participants. Observations began in November 2008 for a sample of treatment and control classrooms. Trained observers observed classroom activities such as grouping arrangements, teacher pedagogy, and student activities during a language arts period two times throughout the course of the study. Depending on the grade (1–5), class periods and thus observations last between 30 and 50 minutes. Brief interviews with the classroom teacher conducted on the day of the observation focus on teachers' lesson plans, materials, and instruction during the observation. Questions on the RISE and OWE interventions will also be asked as appropriate to treatment group teachers. Teachers will be observed twice in their classrooms—once in the fall of year 1 and once in the spring of year 2.

Teacher logs, surveys, observation protocols, and interview protocols were developed and pilot tested during 2006–08 and modified following the pilot tests.

REL Central researchers collected information on teacher characteristics during study orientation sessions. Teachers were asked to complete an information sheet requesting general demographic information, such as gender and ethnicity, and also teaching experience (such as years teaching and teaching certification status) and experience teaching English language learner students (such as years teaching English language learner students, knowledge of other languages, and prior exposure to professional development for teaching English language learner students). Category definitions for these characteristics are consistent with the national consensus and available databases.

Implementation data. As this study is one of intervention "effectiveness" rather than intervention "efficacy," the examination of implementation fidelity is not of primary interest. To monitor and examine potential issues related to fidelity of implementation of the OWE and RISE programs, REL Central researchers will rely on information from the online logs, observations, and interviews, RISE training artifacts, and school-level artifacts (such as student attendance).

Analysis plan

The primary emphasis of data analysis is to estimate the effects of the combined RISE and OWE programs on the academic achievement of students in year 2 of the study. Additional analyses will be conducted to estimate the effects of the two interventions on teacher outcomes. The proposed cluster-randomized trial involves randomization at the school level and collection of outcome data at the student and teacher level. Estimation of treatment effects at the level of the cluster (school) is considered the most appropriate methodology, given the nesting of the data.

This nested design necessitates analysis through multilevel modeling. Two separate multilevel models will be estimated: one to address the primary research question about student achievement and one to address the secondary research question about changes in teacher pedagogy. The effects of RISE and OWE on student outcomes will be analyzed through a three-level hierarchical model. Level 1 will nest students within classrooms and include the students' ELL status as a predictor. Level 1 will also include a student level covariate to account for baseline achievement. Level 2 will include the percent of students in each school on free or reduced-price lunch and a measure of teaching experience with English language learner students as covariates. Level 3 will include an indicator for assignment to intervention or control as a predictor of mean school achievement to estimate the effect of the intervention on student achievement. It will also include a covariate for school district to improve the power of the estimation of the intervention's effect.

Effects of RISE and OWE on teacher outcomes (changes in teacher pedagogical practices, as determined by the quarterly online implementation log) will be analyzed through a two-level hierarchical model. For this model, level 1 nests teachers within schools and uses teacher practices (a function of general teaching experience) and experience teaching English language learner students as covariates. Level 2 will include a school-level covariate at the teacher level from baseline measures of teacher knowledge and experience as well as a variable to indicate group assignment. In addition, the district in which the school is located will be included as a covariate.

As recommended by Shadish, Cook, and Campbell (2002), the researchers will analyze attrition—descriptive information on the overall rate of attrition, the different rate for treatment and control conditions, and whether completers differ from noncompleters. The percentage of students and teachers in each group for whom outcome data could not be obtained and the number of schools in each group that did not complete the study will be reported. Analyses will compare baseline results for the initial sample in each group with baseline results for the final sample to determine whether the final sample differs from the initial sample. Model-based multiple imputation methods will be used to handle missing data.

Principal investigators

Sheila A. Arens, PhD
Mid-continent Research for Education and Learning

Contact information

Sheila Arens, PhD
Mid-continent Research for Education and Learning
4601 DTC Blvd., Ste. 500
Denver, Colorado 80237
Phone: (303) 632-5625
Fax: (303) 337-3007
sarens@mcrel.org

Region: Central

References

Baker, S., Gersten, R., Haager, D., Dingle, M., and Goldenberg, C. (2004). The relationship between observed teaching practice and growth in reading in 1st graders who are English learners. (Technical Report 2004-1). Eugene, OR: Pacific Institutes for Research.

Ballard and Tighe Publishers. (2001). Overview and advantages of IPT Oral Tests. Retrieved September 16, 2005, from http://www.ballard-tighe.com/IPTOnlineInserviceTraining/IPTOral/IPTIOral_OverviewAndAdvantages.htm.

Ballard and Tighe Publishers. (2005). IPT Testing System. Retrieved September 13, 2005, from http://www.nclb.ballard-tighe.com/system.html.

Brophy, J., and Good, T.L. (1986). Teacher behavior and student achievement. In M. Witrock (Ed.), The third handbook of research on teaching. New York: Macmillan.

Collier, V. P. (1995). Acquiring a second language for school. Directions in Language and Education, 1 (4). Retrieved September 3, 2005, from http://www.ncela.gwu.edu/pubs/directions/04.htm.

Darder, A. (1993). How does the culture of the teacher shape the classroom experience of Latino students? The unexamined question in critical pedagogy. In Rothstein, S. (Ed.), Handbook of Schooling in Urban America. Westport, CT: Greenwood Press.

Educational Research Institute of America. (2004, June). A Study of the instructional effectiveness of On Our Way to English. Brewster, NY: Educational Research Institute of America.

Englert, C. (1984). Effective direct instruction practices in special education settings. Remedial and Special Education, 5 (2), 38–47.

Fillmore, L. W., and Snow, C. E. (2000). What teachers need to know about language. ERIC Clearinghouse on Languages and Linguistics. Retrieved December 6, 2006, from http://faculty.tamu-commerce.edu/jthompson/Resources/FillmoreSnow2000.pdf.

Freeman, D., and Freeman, Y. (2000). Meeting the needs of English Language Learners. Talking Points, 12 (1), 2–7.

Godina, H. (1998). Mexican-American high school students and the role of literacy across home-school-community settings. (Doctoral dissertation, University of Illinois at Urbana-Champaign, 1990). Dissertation Abstracts International, 59 (8), 2825.

Haager, D., Gersten, R., Baker, S., and Graves, A. W. (2003). The English-language learner classroom observation instrument for beginning readers. In S. Vaughn and K. L. Briggs, (Eds.) Reading in the classroom: Systems for the observation of teaching and learning. Baltimore, MD: Brookes.

Henderson, R. W., and Landesman, E. M. (1992). Mathematics and middle school students of Mexican descent: The effects of thematically integrated instructions (Research Report No. 5). Santa Cruz, CA: University of California, National Center for Cultural Diversity and Second Language Learning.

Herman, J. L., and Abedi, J. (2004, April). Issues in assessing English language learners opportunity to learn mathematics. Paper presented at the meeting of the American Educational Research Association, San Diego, CA.

Hill, J. D., and Flynn, K. (2004). English language learner resource guide: A guide for rural districts with a low incidence of ELLs. Aurora, CO: The Mid-continent Research for Education and Learning.

Klingler, J. K., and Vaughn, S. (2000). The helping behaviors of fifth graders while using collaborative strategic reading during ESL content classes. TESOL Quarterly, 34 (1), 69–98.

Krashen, S. (1996). Under attack: The case against bilingual education. Culver City, CA: Language Education Associates.

Lewis, L., Parsad, B., Carey, N., Bartfai, N., Farris, E., Smerdon, B., and Green, B. (1999). Teacher quality: A report on the preparation and qualification of public school teachers (NCES 1999-080). Washington, DC : National Center for Education Statistics. Retrieved September 5, 2005, from http://nces.ed.gov/pubs99/1999080.pdf.

Marketing Works. (2005). The instructional effectiveness of On Our Way to English on English immersion and bilingual students. Yardley, PA: Marketing Works.

National Council of Teachers of English. (2006). NCTE position paper on the role of English teachers in educating English language Learners (ELLs). National Council of Teachers of English. Retrieved February 1, 2007, from http://www.ncte.org/positions/statements/teacherseducatingell.

Renaissance Learning. (2003). STAR Reading-Overview. Renaissance Learning. Retrieved September 16, 2005, from http://www.renlearn.com/sr/overview.aspx.

Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

Smith, F. (1983). Essays into literacy: Selected papers and some afterthoughts. Portsmouth, NH: Heinemann.

Tikunoff, W. J., Ward, B. A., van Broekhuizen, L. D., Romero, M., Castaneda, L. V., Lucas, T., and Katz, A. (1991). Final Report: A descriptive study of significant features of exemplary special alternative instructional programs. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, Regional Educational Laboratory Southwest.

Thomas, W. P., and Collier, V. P. (2001). A national study of school effectiveness for language minority students' long-term academic achievement. Berkely, CA: University of California, Center for Research on Education, Diversity and Excellence. Retrieved April 14, 2004, from http://www.crede.ucsc.edu/research/llaa/1.1pdfs/1.1complete.pdf.

Return to Index