The Effects of Opening the World of Learning (OWL) on the Early Literacy Skills of At-Risk Urban Preschool Students
Regional need and study purpose
Many children from low-income households are less likely to have the early literacy knowledge and skills needed to achieve success in reading due to differences in home and school experiences (Craig and Washington 2006; McGee and Richgels 2003). Although the literature suggests that high-quality preschool programs greatly improve children's school readiness, particularly in cognitive development and early literacy skills, states are not providing children with access to such programs (National Institute for Early Education Research 2004).
A report by the National Institute for Early Education Research notes that all Appalachian Region states (Kentucky, Tennessee, Virginia, and West Virginia) failed to meet the federal benchmark for a comprehensive preschool curriculum and that three states (Kentucky, Virginia, and West Virginia) failed to meet the federal benchmark for teacher education and training. Preschool teachers in the Appalachia Region need training in three areas: the importance of early literacy, the role of childcare in supporting children's early literacy development, and effective scientific practices that support early literacy development.
The purpose of this randomized controlled longitudinal study is to determine whether a research-based preschool curriculum—Opening the World of Learning (OWL)—has an impact on the literacy achievement of at-risk children and, if so, whether such effects vary by types of children, families, schools, and preschool and program experiences. The primary research questions are:
- How does the OWL program affect children's literacy skills and school readiness, particularly those from low-income families, before entering kindergarten?
- What are the impacts over time, as children progress through kindergarten and grade 1?
- How does the OWL program change teacher practices in literacy instruction and classroom environments?
Exploratory research questions include:
- To what extent are teachers successful in implementing the OWL program fully and accurately?
- To what extent have teachers been provided training and support to implement the program?
- Under what conditions is the OWL program effective?
- Does teacher behavior align with scientific research-based practices?
- To what extent are the OWL program effects on student achievement associated with variations in student characteristics, teacher background, and instructional practice?
The relationships between research-based instructional practice in preschool and literacy achievement in preschool through grade 1 will inform policymakers about the characteristics of effective preschool instruction. The findings of this study might be generally applied to other urban settings with similar populations implementing a research-based program.
Intervention description
Developed by Judith Schickedanz, PhD, and David Dickinson, EdD, in collaboration with Charlotte-Mecklenberg Schools, OWL is a comprehensive research-based preschool curriculum designed to support all aspects of development, including self-regulation and social development. Program elements are based on research that correlates language development with the quantity and quality of conversations children have with adults, particularly through daily interactions with teachers (Dickinson and Tabors 2001). Phonological awareness and letter knowledge are integrated with language development throughout the curriculum.
Since 2003 OWL has been used in more than 150 school districts in 39 states, mostly in the south and central regions. In 2007 OWL was the research-based curriculum used in 8 of the 32 Early Reading First grants awarded across the country; in 2008 it was the most frequently used curriculum—chosen by 12 of the 31 grantees. Aligned with Tennessee Early Learning Standards, OWL was approved as a research-based program for state preschool classrooms by the Tennessee Department of Education. Preliminary results from studies evaluating the effectiveness of OWL indicate significant benefits for early literacy, with moderate to large effect sizes across the early literacy skills assessed (Dickinson, McCabe, and Essex 2006).
Study design
A randomized, pretest-posttest, treatment-control group design is used. The random assignment used in the study minimizes selection, history, and maturation threats, thus strengthening the validity of study conclusions. Because random assignment occurs at the school level, teacher contamination is minimal. In addition, a low attrition rate is expected because all students who remain in the district are available for follow-up assessment in kindergarten and grade 1.
In year 1, 41 schools were randomly assigned to treatment or control groups using a stratification based on the funding source (district or Title I), which resulted in 26 treatment and 24 control classrooms. Because the researchers were late to begin the intervention in year 1, they adopted a second cohort of students in year 2. Due to school closings and classroom relocations, 59 preschool classrooms at the 38 cohort 2 schools were randomly assigned to the treatment (32) and control (27) groups (table 1). The research team determined, however, that low implementation fidelity with cohort 1 could confound findings if the data from both cohorts were pooled for analyses. So, only cohort 2 participates in the longitudinal aspect of the study and the testing in kindergarten and grade 1.
Table 1. Participants by cohort
| Participants | Cohort 1 | Cohort 2 | ||
| Treatment | Control | Treatment | Control | |
| Schools | 21 | 20 | 20 | 18 |
| Teachers | 26 | 24 | 32 | 27 |
| Pretest students | 452 | 447 | 612 | 489 |
| Posttest students | 412 | 410 | 569 | 461 |
| Student attrition (percent) | 9 | 8 | 7 | 6 |
Source: Authors' compilation based on data from the study.
The study takes place in a large urban district in Tennessee with district- or Title I-funded preschools. The sample includes mostly Black students from low-income families because of the school district's priority enrollment policy for students the most at risk. All participating schools were interested in implementing OWL if selected but they agreed to continue current (traditional) programs until the completion of the study if not chosen. Treatment schools receive the benefits of the district-provided OWL curriculum and accompanying professional development. Monetary compensation was provided to the district at the beginning of the study for each control school for supplying data (allowing site researchers to conduct assessments and classroom observations).
The study was originally designed to have sufficient power to detect effect sizes ranging from small (.30 standard deviation) to moderate (.40 standard deviation) for student achievement data from the first cohort. The minimum power is about .80 when the intraclass correlation is assumed to be .10. The intraclass correlation represents the "clustering" effect—in which 10 percent of outcome variation lies between clusters (classrooms). In addition, cluster-level covariates include sex, age, pretest scores, and time between testing. Although a second power analysis was not performed for cohort 2, researchers expect that the study still has sufficient power because the sample size increased.
The study design supports a more realistic representation of curriculum implementation in a large urban district serving disadvantaged students. It also encourages the support and cooperation of the district administration and teachers. Ongoing interaction with the district administration and preschool teachers has been positive and suggests they are supportive of the study.
Key outcomes and measures
Key outcomes are measured through observations of preschool classrooms, surveys of preschool teachers and parents, and student assessments in preschool, kindergarten, and grade 1. Classroom observations provide information on the literacy environment, literacy activities, use of research-based literacy instructional strategies, and implementation of the intervention. Preschool teachers and paraprofessionals provide demographic data through surveys to evaluate their perceptions of the program, including professional development and pedagogical change. Parent surveys provide information on perceptions of the program. And student assessments evaluate student literacy development. In preschool the researchers assess receptive vocabulary, emergent writing skills, alphabet knowledge, phonological awareness, and print/word awareness. In kindergarten and grade 1 they assess basic reading, writing, and oral language skills.
Data collection approachBaseline and follow-up data are collected by experienced site researchers through direct assessments of individual children, classroom observations, and surveys of parents, teachers, and paraprofessionals. All site researchers undergo comprehensive training and are monitored by research staff for quality assurance. In preschool each student assessment lasts about 30 minutes, and each site researcher tests an average of five students a day. Classroom observations take at least half a school day, so researchers complete only one observation a day. Each survey also requires about 15 minutes to complete. Teacher and paraprofessional surveys are online, while the parent survey is on paper.
In year 1 preschool student achievement data were collected twice, surveys once, and observations intermittently throughout the school year. Baseline assessments were given in the fall of year 1, and follow-up data were collected in the spring of that year. Student achievement data were collected using the Peabody Picture Vocabulary Test-Third Edition (PPVT-III) and the Phonological Awareness Literacy Screening-PreK (PALS). In addition, classroom observations were conducted using the Early Literacy Observation Tool (E-LOT) and the Early Language and Literacy Classroom Observation (ELLCO). Student assessments, classroom observations, and surveys of teachers, paraprofessionals, and parents occurred in all participating classrooms.
Data for the second cohort came from baseline and follow-up student achievement testing using the PPVT-III and PALS in the fall and spring of year 2. Classroom observations using the E-LOT and ELLCO also occurred in the fall and spring for all classrooms and included a supplementary observation of implementation fidelity, the Holistic OWL Observation Tool (HOOT), in treatment classrooms. The researchers collected surveys of teachers, paraprofessionals, and parents in the spring.
To evaluate the program's long-term impact on student literacy skills, Woodcock-Johnson Tests of Achievement-Third Edition are administered once in the spring of year 3 (kindergarten) and year 4 (grade 1) to cohort 2 students. Observations and surveys are not administered beyond the preschool year.
Analysis plan
The researchers use both qualitative and quantitative methods to analyze the data. Inferential analyses focus on determining statistically significant differences between groups (treatment and control) at the end of preschool and beyond. They also compare preintervention and postintervention observations and student achievement when data are available. In addition, they calculate and examine descriptive data—frequencies, means, standard deviations, effect sizes—for differences.
Classroom observations and surveys. To evaluate differences between groups for the E-LOT observations and surveys, the researchers use the Wilcoxon-Mann-Whitney test, a nonparametric method for comparing two independent groups. Due to the normal distribution of the ELLCO observation data, an unpaired t-test is used. To evaluate the differences between the preintervention (fall) and postintervention (spring) observations, the Wilcoxon signed-rank test for matched pairs is used for the E-LOT, and a paired t-test for the ELLCO. Observations and demographic profiles of teachers with high student achievement are examined descriptively.
Student achievement. The researchers perform factor analyses, using principal components extraction and an oblique rotation, on both pretest and posttest variables to reduce the set of variables for estimation of treatment effects. Results from preliminary analyses indicate that two factors—alphabetic principles and emergent reading skills—account for around 70 percent of the variance in subtest scores. Cases with missing scores are deleted. Missing data are analyzed to determine whether a relationship exists between respondent characteristics and study attrition due to missing data, with a particular focus on differential attrition by treatment status.
To address the research question for the effects of OWL on student literacy readiness, the researchers estimate two-level hierarchical linear models (HLM) for both alphabetic principles and emergent reading skills using HLM 6 (Raudenbush et al. 2004). To control for experimentwise alpha inflation, they apply the Holme's sequential Bonferroni procedure in testing treatment effects on each outcome. In the level 1 model the posttest scores on alphabetic principles and emergent reading skills are treated as outcomes, while sex, age (in months), days between test administrations, and the corresponding pretest score are incorporated as covariates. All student-level predictors center on their grand means, so the level 1 intercept estimates are analogous to the covariate-adjusted means obtained through analysis of covariance. In the level 2 model the researchers model variation in the classroom intercepts as a function of treatment status. They compute treatment effect size estimates by transforming the t-test statistic associated with the test of the treatment effect into Cohen's d.
Two-level HLMs are also estimated to explore whether the effects of OWL are associated with teacher and student characteristics. HLM analyses similar to those performed for the confirmatory impact analysis are performed for alphabetic principles and emerging reading skills, except that cross-level interaction effects between treatment and sex, pretest, and age are estimated. Finally, relationships between classroom-level student outcomes, implementation variables, teacher background characteristics, and instructional practice are examined.
Anna Grehan, PhD
Education Innovations
Contact information
Anna W. Grehan, PhDResearch Associate Professor
The University of Memphis
(901) 678-4222
awgrehan@memphis.edu
Rachel L. Peterman, MA
Research Associate
The University of Memphis
(901) 678-3850
r.peterman@memphis.edu
Region: Appalachia
References
Craig, H.K., and Washington, J.A. (2006). Recent research on the language and literacy skills of African-American students in the early years. In D.K. Dickinson and S.B. Neuman (Eds.), Handbook of Early Literacy Research, Vol. 2. New York: Guilford Press.
Dickinson, D.K., and Tabors, P.O. (2001). Beginning literacy with language: Young children learning at home and school. Baltimore, MD: Brookes Publishing.
Dickinson, D.K, McCabe, A.A., and Essex, M.J. (2006). A window of opportunity we must open to all: The case for preschool with high-quality support for language and literacy. In D.K. Dickinson and S.B. Neuman (Eds.), Handbook of Early Literacy Research, Vol. 2. New York: Guilford Press.
McGee, L.M., and Richgels, D.J. (2003). Designing early literacy programs: Strategies for at-risk preschool and kindergarten children. New York: Guilford Press.
National Institute for Early Education Research. (2004). The State of Preschool: 2004 State Preschool Yearbook. Rutgers, The State University of New Jersey. Retrieved November 15, 2008, from http://nieer.org/yearbook2004/.
Raudenbush, S., Bryk, A., Cheong, Y. F., and Congdon, R. (2004). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
Schickedanz, J.A. and Dickinson, D.K. with Charlotte-Mecklenburg Schools. (2005). Opening the World of Learning: A Comprehensive Early Literacy Program. Parsippany, NJ: Pearson Early Learning.