The Sequential Scale-Up of an Evidence-Based Intervention: A Case Study [Pre-K Mathematics vs. business as usual]
Thomas, Jaime; Cook, Thomas D.; Klein, Alice; Starkey, Prentice; DeFlorio, Lydia (2018). Retrieved from: https://eric.ed.gov/?id=ED587225
-
examining1,313Students, gradePK
Pre-K Mathematics Intervention Report - Preparing Young Children for School
Review Details
Reviewed: July 2023
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Pre-K Mathematics.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Early Childhood Longitudinal Study, Birth Cohort (ECLS-B) Math assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
30.84 |
29.09 |
Yes |
|
|
|
Test of Early Mathematics Ability, Third Edition, (TEMA-3) |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
16.07 |
14.33 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
32% English language learners -
Female: 52%
Male: 48% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Asian 2% Black 6% Other or unknown 79% White 13% -
Ethnicity Hispanic 75% Other or unknown 25% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
This study takes place in 140 pre-kindergarten classrooms: 17 participating classrooms in the Bay Area in California, 31 in rural Northern California, 13 in the rural Central Valley, and 79 in Southern California (including the Los Angeles area). Overall, 18% of the classrooms were in urban areas, 51% in suburban areas, and 31% in small towns or rural areas. The classrooms were public pre-kindergarten and Head Start programs.
Study sample
One-hundred forty pre-kindergarten classrooms were randomly assigned to conditions: 70 into the intervention group and 70 into the comparison group. There were 1,373 students in the study at the time of random assignment: 687 in the intervention group and 686 in the comparison group. Approximately 52% of the students were female and 32% were English learners. Seventy-five percent were Hispanic or Latino.
Intervention Group
Pre-K Mathematics is a multicomponent supplementary math curriculum for pre-kindergarten children. Pre-K Mathematics focuses on the pre-kindergarten classroom and home learning environments of young children, especially those from families experiencing economic hardship. Its activities are designed to support mathematical development by providing learning opportunities to increase children’s informal mathematical knowledge. The intervention consists of a sequence of small-group math activities with concrete manipulatives that teachers implement in the pre-kindergarten classroom. The program also includes home math activities in the form of picture strips for parents to use with their children. The curriculum includes mathematical concepts and skills, including number, operations, geometry, pattern knowledge, and measurement. Units and activities within Pre-K Mathematics were designed to prepare children for the Common Core State Standards for Mathematics at kindergarten.
Comparison Group
Comparison group classrooms continued their business-as-usual curriculum and teaching practices for the duration of the study.
Support for implementation
Teachers in the intervention group attended multi-day professional development workshops to learn about the philosophy, key features, and activities in Pre-K Mathematics. They learned how to implement the activities, track children's learning, and explain at-home activities to parents.
Practice Guide
Review Details
Reviewed: June 2023
- Practice Guide (findings for Pre-K Mathematics)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Early Childhood Longitudinal Study, Birth Cohort (ECLS-B) Math assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
30.84 |
29.09 |
Yes |
|
|
|
Test of Early Mathematics ability, Third Edition, (TEMA-3) |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
16.07 |
14.33 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
32% English language learners -
Female: 52%
Male: 48% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Asian 2% Black 6% Other or unknown 79% White 13% -
Ethnicity Hispanic 75% Other or unknown 25% -
Eligible for Free and Reduced Price Lunch Other or unknown 100%
Study Details
Setting
This study takes place in 140 pre-kindergarten classrooms: 17 participating classrooms in the Bay Area in California, 31 in rural Northern California, 13 in the rural Central Valley, and 79 in Southern California (including the Los Angeles area). Overall, 18% of the classrooms were in urban areas, 51% in suburban areas, and 31% in small towns or rural areas. The classrooms were public pre-kindergarten and Head Start programs.
Study sample
One-hundred forty pre-kindergarten classrooms were randomly assigned to conditions: 70 into the intervention group and 70 into the comparison group. There were 1,373 students in the study at the time of random assignment: 687 in the intervention group and 686 in the comparison group. Approximately 52% of the students were female and 32% were English learners. Seventy-five percent were Hispanic or Latino.
Intervention Group
Pre-K Mathematics is a multicomponent supplementary math curriculum for pre-kindergarten children. Pre-K Mathematics focuses on the pre-kindergarten classroom and home learning environments of young children, especially those from families experiencing economic hardship. Its activities are designed to support mathematical development by providing learning opportunities to increase children’s informal mathematical knowledge. The intervention consists of a sequence of small-group math activities with concrete manipulatives that teachers implement in the pre-kindergarten classroom. The program also includes home math activities in the form of picture strips for parents to use with their children. The curriculum includes mathematical concepts and skills, including number, operations, geometry, pattern knowledge, and measurement. Units and activities within Pre-K Mathematics were designed to prepare children for the Common Core State Standards for Mathematics at kindergarten.
Comparison Group
Comparison group classrooms continued their business-as-usual curriculum and teaching practices for the duration of the study.
Support for implementation
Teachers in the intervention group attended multi-day professional development workshops to learn about the philosophy, key features, and activities in Pre-K Mathematics. They learned how to implement the activities, track children's learning, and explain at-home activities to parents.
Department-funded evaluation
Review Details
Reviewed: June 2022
- Department-funded evaluation (findings for Pre-K Mathematics)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a cluster randomized controlled trial with low cluster-level attrition and individual-level non-response.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|
|---|---|---|---|---|---|---|---|---|---|
|
Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
30.84 |
29.09 |
Yes |
|
|
|
| Show Supplemental Findings | |||||||||
|
Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
African American;
|
32.54 |
29.13 |
Yes |
|
||
|
Early Childhood Longitudinal Study- Birth Cohort (ECLS-B) Mathematics Assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
Hispanic or Latino;
|
30.26 |
28.56 |
Yes |
|
||
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Asian 2% Black 6% Other or unknown 79% White 13% -
Ethnicity Hispanic 75% Not Hispanic or Latino 25%
Study Details
Study sample
The sample was entirely comprised of low-income students (does not mention free or reduced-price lunch status). Seventy-six percent of students identified as Hispanic, 11% identified as White, and 6% identified as African American.
Intervention Group
This study seeks to examine the effect of exposing high-need students to one year of tier-1 mathematics curricula, Pre-K Mathematics, in pre-K classrooms in Northern and Southern California. Pre-K Mathematics is a multicomponent supplementary math curriculum, consisting of a carefully sequenced set of small-group math activities with concrete manipulatives and key mathematical terms that teachers implement in the pre-K classroom, as well as parent-child home math activities. The study aims to change math instruction through classroom teacher training and implementation of curriculum materials, and in turn to improve students' math knowledge, general school achievement, and decoding skills in first grade.
Comparison Group
Students and teachers in the business-as-usual comparison condition did not participate in the intervention. Students were exposed to typical instruction.
Support for implementation
Teachers in the intervention attend multi-day professional development workshops in which they learn about the philosophy and key features of the program as well as how to implement the math activities. Teachers get hands-on experience with implementation tools in the workshops, and learn how to explain at-home activities and parent feedback forms to parents.
Grant Competition
Review Details
Reviewed: August 2018
- Grant Competition (findings for Pre-K Mathematics)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Early Childhood Longitudinal Study- Kindergarten (ECLS-K) Math Assessment |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
30.84 |
29.09 |
Yes |
|
|
|
Test of Early Mathematics Ability (TEMA-3) |
Pre-K Mathematics vs. Business as usual |
0 Days |
Full sample;
|
16.07 |
14.33 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 52%
Male: 48% -
Rural, Suburban, Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
California
-
Race Asian 2% Black 6% Other or unknown 79% White 13% -
Ethnicity Hispanic 75% Not Hispanic or Latino 25%
Study Details
Setting
Pre-K school sites were recruited from the greater Los Angeles area, the Central Valley, the Bay Area, and rural Northern California. The purposive selection included public pre-K and Head Start programs located in urban, suburban, and rural areas with large proportions of low-income families from diverse racial/ethnic backgrounds who plausibly represent all low-income families in California with a child of pre-K age.
Study sample
There were 17 pre-K classrooms in the Bay Area, 31 in rural Northern California, 13 in the rural Central Valley, and 79 in various parts of Southern California, including the greater Los Angeles area. Overall, 18% of pre-K classrooms were in urban areas, 51% in suburban areas, and 31% in towns or rural areas. Seventy-five percent were Hispanic, 13% were White, 6% were African American, 4% were of mixed race, and 2% were Asian. Most of the sample were exclusively English speakers (68%). For 25%, Spanish was the dominant language, and 7% spoke both English and Spanish. On average, children were 4.4 years old at baseline (fall of the pre-K year), and 48% were male.
Intervention Group
The intervention consisted of a sequence of small-group math activities that teachers implemented in the pre-K classroom. The program also included home math activities in the form of picture strips for parents to use with their children. The content of the activities is based on developmental research about the nature and extent of early mathematical knowledge and targets a range of mathematics concepts and skills. Units within Pre-K Mathematics are designed to prepare children for standards included in the Common Core State Standards for Mathematics at kindergarten.
Comparison Group
The comparison group received pre-K as usual.
Support for implementation
Teachers attend multiday professional development workshops to learn about the philosophy and key features of the program as well as how to implement the math activities. In addition, Pre-K Mathematics employed several implementation tools, which teachers got hands-on experience with in the workshops. Teachers also learned how to keep track of each child’s learning over the course of the year, using recording sheets that accompany each math activity and a progress monitoring tool. During the workshops, teachers also learned how to explain at-home activities to parents and how to use a parent feedback form to document parents’ use of these activities. To monitor fidelity, authors collected two implementation measures during the school year. Local trainers made visits to teachers in the treatment classrooms and observed teachers conducting a small-group math activities, giving them feedback afterward about any departures from fidelity. Additionally, the Early Mathematics Classroom Observation (EMCO) observation tool was used to determine the nature and amount of mathematics instruction that preschool teachers provided in their classrooms. For each teacher–participant activity involving mathematical content, trained observers recorded the type of mathematical content, number of children present, and the duration of the activity. This provided data on the number of minutes of math instruction, on average, to a child during an observation session.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).