Partners: WestEd, Compton USD, Fresno County group, Pasadena USD, Rialto USD, San Joaquin County group, Tehama County Department of Education

Purpose: In this implementation science project researchers and local public pre-kindergarten administrators and teachers worked as teams to improve teachers' implementation of a What Works Clearinghouse-reviewed curriculum, Pre-K Mathematics. Previous IES-funded research (see Related Projects below) has found this curriculum to be effective at improving low-income students' readiness for elementary school mathematics. Anecdotal evidence has suggested that some teachers, who have learned to implement Pre-K Mathematics with fidelity and adequate curriculum dosage, subsequently adapt their implementation over time. Some adaptations, however, could potentially reduce active ingredients of the curriculum, which in turn could reduce the curriculum's effectiveness. This project had two principal purposes. The first was to obtain systematic evidence of whether teachers adapted their implementation over time, evidence as to whether adaptations reduced teachers' fidelity or dosage delivery, and evidence of whether the curriculum was beneficial to children's mathematics learning, despite adaptations made by teachers. The second purpose was to use an empirically based improvement process to provide guidance to public preschool administrators and teachers about ways to adapt implementation to meet local needs without reducing the effectiveness of the curriculum.

The partnership enhanced supports for implementation of Pre-K Mathematics, with the intent of sustainably embedding the curriculum within teachers' instructional practice and within a public pre-k program's system of instruction. Through a "scaling in" process, researchers preserved and enhanced delivery of the essential features of the intervention that are necessary to produce positive math outcomes and modify non-essential features judged unsustainable by the pre-k programs. By improving the pre-kindergarten math curriculum, the partnership intended to increase student proficiency in math at kindergarten entry, thus reducing the gap in math knowledge across income groups at the beginning of elementary school. At the end of the project, the partnership had an improved implementation of Pre-K Mathematics by tailoring it to the needs of local programs, and each local program had greater capacity for implementing this curriculum sustainably and effectively.

Project Activities: The partnership coordinated activities of the research team, program administrators, and teachers. The partnership relied on input from teachers regarding necessary conditions for implementing the intervention and upon program administrators for supporting and monitoring implementation. The partnership first modified implementation of Pre-K Mathematics in ways chosen by the partnership. Through two short semester-long improvement cycles followed by a single annual cycle (the Sustainability Study), conducted over two cohorts, the partnership adapted, tested, and then evaluated the implementation model used for the Pre-K Mathematics curriculum. In this way programs could implement the curriculum in ways that addressed organizational (e.g., resource) constraints of local programs but preserved the active ingredients (i.e., essential features) of the curriculum. The partnership is communicating its findings to practitioners, researchers, and policymakers.

Key Outcomes:

• The Sustainability Study provided systematic evidence that teachers, when left to implement without monitoring or guidance (the Unguided Adaptation condition), often adapt their implementation over time, and that some adaptations can reduce curriculum fidelity and dosage delivery.
• Researchers tested a Guided Implementation condition in which teachers implemented for an entire school year and program administrators monitored and supported implementation while allowing for three agreed-upon program modifications. Researchers collected and analyzed implementation data and child outcome data.

Structured Abstract

Setting: The project was conducted in classrooms in six sets of local public pre-kindergarten programs in California. These LEAs varied structurally (Head Start as well as California State Preschool programs operated by school districts and a county department of education), geographically (urban and rural), and in racial/ethnic composition of the communities served.

Sample: Study participants were two cohorts of lead teachers in a total of 67 classrooms (36 Guided Adaptation and 31 Unguided Adaptation) located in public pre-kindergarten programs. The 669 pre-kindergarten children comprising the research sample for the sustainability study were randomly selected within each participating classroom. Twenty-one lead teachers/classrooms and 176 pre-kindergarten children from the prior RCT comprised the Business-As-Usual (no Pre-K Mathematics) control condition.

Intervention: The standard form of the Pre-K Mathematics intervention combines professional development for teachers with content aligned with the Common Core State Standards. Through workshops and onsite training, teachers learn pedagogical content knowledge and strategies for delivering the curriculum with fidelity, including the use of small-group activities and engaging parents in home-based math activities. Based on local needs and organizational capacity within each of the public pre-kindergarten programs, the partnership adapted the standard way teachers implement the curriculum by

• Providing instructional assistants complete training in Pre-K Mathematics so that they can support delivery of the curriculum;
• Allowing teachers more flexibility in the number of children to be included in small group math activities;
• Conducting an outreach event (e.g., a family math event) using Pre-K Mathematicshome activities to engage parents;
• Revising the Pre-K Mathematics curriculum plan to provide more time to review math activities at regular intervals throughout the school year, especially with children who need extra support.

In addition, program administrators actively monitored and supported Pre-K Mathematics implementation. These modifications were then tested in classrooms in short cycles lasting a half school year. By the end of short cycle testing, one modification — use of flexible group sizes — was found to reduce fidelity and dosage delivery. Consequently, teams decided to revert to the original group size (4–5) for small group activities. The other three modifications worked well and were incorporated into the implementation model used by teachers. Next, a quasi-experimental sustainability study was conducted in the year following short cycle testing. This study was designed to determine whether local programs can sustain an effective implementation of Pre-K Mathematics when they use the modified implementation model developed through short cycle testing.

Research Design and Methods: Participating teachers had previously been trained, to implement Pre-K Mathematics in a standard way as part of a Randomized Controlled Trial. After participating in the RCT, teachers continued implementing Pre-K Mathematics on their own for the next 3-4 years. At the beginning of the present project, these teachers were re-randomized to one of two conditions: (1) the Guided Adaptation(treatment) condition and (2) the Unguided Adaptation (comparison) condition. In the Guided Adaptation condition, teachers worked as part of a collaborative team to make data-guided decisions about scaling-in their implementation of Pre-K Mathematics. Teams focused on adaptations intended to sustain effective math instruction without reducing implementation fidelity or dosage delivery. In the Unguided Adaptation condition, teachers continued adapting their implementation in whatever way they wished without guidance.

Control Condition: Control classrooms from the original RCT comprised a Business-As-Usual (no Pre-K Mathematics) control condition for some comparisons.

Key Measures: Trainers used the Fidelity of Implementation Record Sheet during bi-weekly classroom observations of small-group math sessions to evaluate teachers' fidelity to the intervention. Trainers and data collectors used an adaptation checklist to document implementation of modifications by teachers. Teachers used the Assessment Record Sheet (ARS) to record each small group math session, including an assessment of each child's performance during the session. Teachers collected Parent & Family Feedback Forms (PFFs) from parents to track families' use of Pre-K Mathematics home math activities. Researchers used the Early Mathematics Classroom Observation tool to measure teachers' instructional practices. The research team used the Math Mastery instrument to measure children's learning of teacher-administered math activities. The Child Math Assessment (CMA) was used to measure annual growth in children's mathematics knowledge.

Data Analytic Strategy: Researchers used chi-square statistics to compare deviations from benchmark fidelity and dosage levels within classrooms. Researchers used a generalized linear regression model to assess whether implementation differed significantly between guided adaptation and unguided adaptation classrooms, and a two-level generalized linear regression model of children within classrooms to estimate whether mathematical achievement differs significantly among children in guided adaptation, unguided adaptation, and control classrooms.

Project Website: https://prekmath.wested.org/current-projects/continuous-improvement/

Related IES Projects: Scaling Up the Implementation of a Pre-Kindergarten Mathematics Curriculum in Public Preschool Programs (R305K050186), Closing the SES Related Gap in Young Children's Mathematical Knowledge (R305A080188), A Randomized Study of the Efficacy of a Two-Year Mathematics Intervention for At-Risk Pre-Kindergarten and Kindergarten Students (R305A120262), A Longitudinal Study of the Effects of a Pre-Kindergarten Mathematics Curriculum on Low-Income Children's Mathematical Knowledge (R305J020026)

Products and Publications

This project produced three principal products. First, early childhood programs that currently implement or plan to implement Pre-K Mathematics, now have a set of effective implementation models from which to choose. Second, this project utilized an empirically based process to guide LEAs' efforts to adapt interventions to their local context. This process can potentially be used to adapt other interventions, such as early literacy interventions, without reducing their effectiveness. Third, this project produced knowledge for implementation science.

ERIC Citations: Publications from this project available in ERIC can be found here.