WWC review of this study

The sequential scale-up of an evidence-based intervention: A case study. [Pre-K Mathematics vs. business as usual]

Thomas, J., Cook, T. D., Klein, A., Starkey, P., & DeFlorio, L. (2018). Evaluation Review, 42(3), 318–357. Retrieved from: https://eric.ed.gov/?id=ED587225

  • Randomized Controlled Trial
     examining 
    1,313
     Students
    , grade
    PK

Reviewed: June 2022

At least one finding shows strong evidence of effectiveness
At least one statistically significant positive finding
Meets WWC standards without reservations
General Mathematics Achievement outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment

Pre-K Mathematics vs. Business as usual

0 Days

Full sample;
1,313 students

30.84

29.09

Yes

 
 
12
 
More Outcomes
Show Supplemental Findings

Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment

Pre-K Mathematics vs. Business as usual

0 Days

African American;
69 students

32.54

29.13

Yes

 
 
22

Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment

Pre-K Mathematics vs. Business as usual

0 Days

Hispanic or Latino;
1,001 students

30.26

28.56

Yes

 
 
11


Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Characteristics of study sample as reported by study author.


  • Female: 52%
    Male: 48%

  • Rural, Suburban, Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    California
  • Race
    Asian
    2%
    Black
    6%
    Other or unknown
    79%
    White
    13%
  • Ethnicity
    Hispanic    
    75%
    Not Hispanic or Latino    
    25%

Study sample

The sample was entirely comprised of low-income students (does not mention free or reduced-price lunch status). Seventy-six percent of students identified as Hispanic, 11% identified as White, and 6% identified as African American.

Intervention Group

This study seeks to examine the effect of exposing high-need students to one year of tier-1 mathematics curricula, Pre-K Mathematics, in pre-K classrooms in Northern and Southern California. Pre-K Mathematics is a multicomponent supplementary math curriculum, consisting of a carefully sequenced set of small-group math activities with concrete manipulatives and key mathematical terms that teachers implement in the pre-K classroom, as well as parent-child home math activities. The study aims to change math instruction through classroom teacher training and implementation of curriculum materials, and in turn to improve students' math knowledge, general school achievement, and decoding skills in first grade.

Comparison Group

Students and teachers in the business-as-usual comparison condition did not participate in the intervention. Students were exposed to typical instruction.

Support for implementation

Teachers in the intervention attend multi-day professional development workshops in which they learn about the philosophy and key features of the program as well as how to implement the math activities. Teachers get hands-on experience with implementation tools in the workshops, and learn how to explain at-home activities and parent feedback forms to parents.

Reviewed: February 2022

At least one finding shows strong evidence of effectiveness
At least one statistically significant positive finding
Meets WWC standards without reservations
Mathematics outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Early Childhood Longitudinal Study, Birth Cohort (ECLS-B) Math assessment

Pre-K Mathematics vs. Business as usual

0 Days

Full sample;
1,313 students

30.84

29.09

Yes

 
 
12
 
More Outcomes

Test of Early Mathematics ability, Third Edition, (TEMA-3)

Pre-K Mathematics vs. Business as usual

0 Days

Full sample;
1,256 students

16.07

14.33

Yes

 
 
9
 


Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Characteristics of study sample as reported by study author.


  • 33% English language learners

  • Female: 52%
    Male: 48%

  • Rural, Suburban, Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    California
  • Race
    Asian
    2%
    Black
    5%
    Other or unknown
    80%
    White
    13%
  • Ethnicity
    Hispanic    
    75%
    Not Hispanic or Latino    
    25%

Setting

This study is a scale-up of previous studies on the Pre-K Mathematics intervention. In contrast to prior studies, this study takes place over a wide geographical area (the state of California). The study includes 140 pre-kindergarten classrooms: 17 participating classrooms in the Bay Area, 31 in rural Northern California, 13 in the rural Central Valley, and 79 in Southern California (including some in the Los Angeles area). Overall, 18 percent of the classrooms were in urban areas, 51 percent in suburban, and 31 percent in small towns or rural areas. The pre-K classrooms came from public pre-K and Head Start programs that serve large proportions of low-income families from diverse racial/ethnic backgrounds.

Study sample

One-hundred forty pre-kindergarten classrooms were randomly assigned to conditions: 70 into the intervention group and 70 into the comparison group. There were 1,373 students in the study at the time of random assignment: 687 in the intervention group and 686 in the comparison group. In the analytic sample for the ECLS-B, there was no cluster attrition. There was some individual non-response, with 1,313 analytic sample students (653 in the intervention group and 660 in the comparison group). In the TEMA-3 analytic sample, there was low cluster attrition; 135 classrooms remained in the sample (67 intervention and 68 comparison). Individual non-response was also a bit higher than in the first outcome, with 1,256 children remaining in the analytic sample for TEMA-3 (621 treatment, 635 control). Cluster-level attrition was low for both the ECLS-B and TEMA-3 measures. Cluster attrition was 0 percent for ECLS-B. For TEMA-3, overall cluster attrition was 3.6 percent and differential cluster attrition was 1.4 percent. Individual-level attrition was also low under the optimistic boundary for both the ECLS-B and the TEMA-3 measures. Overall attrition rates were 4.4 percent for the ECLS-B and 8.5 percent for the TEMA-3. Differential attrition rates were 1.2 percent for the ECLS-B and 2.2 percent for the TEMA-3.

Intervention Group

Pre-K Mathematics (Klein & Starkey, 2002, 2004) is a multicomponent supplementary math curriculum for pre-kindergarten children. Pre-K Mathematics focuses on the pre-kindergarten classroom and home learning environments of young children, especially those from families experiencing economic hardship. Its activities are designed to support mathematical development by providing learning opportunities to increase children’s informal mathematical knowledge. The intervention consists of a sequence of small-group math activities with concrete manipulatives that teachers implement in the pre-kindergarten classroom. The program also includes home math activities in the form of picture strips for parents to use with their children. The curriculum includes mathematical concepts and skills, including number, operations, geometry, pattern knowledge, and measurement. Units and activities within Pre-K Mathematics were designed to prepare children for the Common Core State Standards for Mathematics at kindergarten.

Comparison Group

Comparison group classrooms continued their business-as-usual curriculum and teaching practices for the duration of the study.

Support for implementation

Teachers in the intervention group attended multi-day professional development workshops to learn about the philosophy, key features, and activities in Pre-K Mathematics. They learned how to implement the activities, track children's learning, and explain at-home activities to parents.

Reviewed: August 2018

At least one finding shows strong evidence of effectiveness
At least one statistically significant positive finding
Meets WWC standards without reservations
General Mathematics Achievement outcomes—Statistically significant positive effects found
Outcome
measure
Comparison Period Sample Intervention
mean
Comparison
mean
Significant? Improvement
    index
Evidence
tier

Early Childhood Longitudinal Study- Kindergarten (ECLS-K) Math Assessment

Pre-K Mathematics vs. Business as usual

0 Days

Full sample;
1,313 students

30.84

29.09

Yes

 
 
12
 
More Outcomes

Test of Early Mathematics Ability (TEMA-3)

Pre-K Mathematics vs. Business as usual

0 Days

Full sample;
1,256 students

16.07

14.33

Yes

 
 
9
 


Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.

Characteristics of study sample as reported by study author.


  • Female: 52%
    Male: 48%

  • Rural, Suburban, Urban
    • B
    • A
    • C
    • D
    • E
    • F
    • G
    • I
    • H
    • J
    • K
    • L
    • P
    • M
    • N
    • O
    • Q
    • R
    • S
    • V
    • U
    • T
    • W
    • X
    • Z
    • Y
    • a
    • h
    • i
    • b
    • d
    • e
    • f
    • c
    • g
    • j
    • k
    • l
    • m
    • n
    • o
    • p
    • q
    • r
    • s
    • t
    • u
    • x
    • w
    • y

    California
  • Race
    Asian
    2%
    Black
    6%
    Other or unknown
    79%
    White
    13%
  • Ethnicity
    Hispanic    
    75%
    Not Hispanic or Latino    
    25%

Setting

Pre-K school sites were recruited from the greater Los Angeles area, the Central Valley, the Bay Area, and rural Northern California. The purposive selection included public pre-K and Head Start programs located in urban, suburban, and rural areas with large proportions of low-income families from diverse racial/ethnic backgrounds who plausibly represent all low-income families in California with a child of pre-K age.

Study sample

There were 17 pre-K classrooms in the Bay Area, 31 in rural Northern California, 13 in the rural Central Valley, and 79 in various parts of Southern California, including the greater Los Angeles area. Overall, 18% of pre-K classrooms were in urban areas, 51% in suburban areas, and 31% in towns or rural areas. Seventy-five percent were Hispanic, 13% were White, 6% were African American, 4% were of mixed race, and 2% were Asian. Most of the sample were exclusively English speakers (68%). For 25%, Spanish was the dominant language, and 7% spoke both English and Spanish. On average, children were 4.4 years old at baseline (fall of the pre-K year), and 48% were male.

Intervention Group

The intervention consisted of a sequence of small-group math activities that teachers implemented in the pre-K classroom. The program also included home math activities in the form of picture strips for parents to use with their children. The content of the activities is based on developmental research about the nature and extent of early mathematical knowledge and targets a range of mathematics concepts and skills. Units within Pre-K Mathematics are designed to prepare children for standards included in the Common Core State Standards for Mathematics at kindergarten.

Comparison Group

The comparison group received pre-K as usual.

Support for implementation

Teachers attend multiday professional development workshops to learn about the philosophy and key features of the program as well as how to implement the math activities. In addition, Pre-K Mathematics employed several implementation tools, which teachers got hands-on experience with in the workshops. Teachers also learned how to keep track of each child’s learning over the course of the year, using recording sheets that accompany each math activity and a progress monitoring tool. During the workshops, teachers also learned how to explain at-home activities to parents and how to use a parent feedback form to document parents’ use of these activities. To monitor fidelity, authors collected two implementation measures during the school year. Local trainers made visits to teachers in the treatment classrooms and observed teachers conducting a small-group math activities, giving them feedback afterward about any departures from fidelity. Additionally, the Early Mathematics Classroom Observation (EMCO) observation tool was used to determine the nature and amount of mathematics instruction that preschool teachers provided in their classrooms. For each teacher–participant activity involving mathematical content, trained observers recorded the type of mathematical content, number of children present, and the duration of the activity. This provided data on the number of minutes of math instruction, on average, to a child during an observation session.

 

Your export should download shortly as a zip archive.

This download will include data files for study and findings review data and a data dictionary.

Connect With the WWC

loading
back to top