Implementation study of The Real Reasons for Seasons (2003–2004): SCALE-uP Report No. 4.
Pyke, C., Lynch, S., Kuipers, J., Szesze, M., & Watson, W. (2004). Washington, DC: George Washington University, SCALE-uP.
-
examining2,408Students, grade7
Great Explorations in Math and Science (GEMS) The Real Reasons for Seasons Intervention Report - Science
Review Details
Reviewed: January 2013
- Randomized controlled trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Please see the WWC summary of evidence for Great Explorations in Math and Science (GEMS) The Real Reasons for Seasons.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Concept Assessment - CSA score |
Great Explorations in Math and Science (GEMS) The Real Reasons for Seasons vs. Business as usual |
Posttest |
Grade 7: Cohort 2;
|
38.54 |
42.26 |
Yes |
|
|
|
Concept Assessment - RSA score |
Great Explorations in Math and Science (GEMS) The Real Reasons for Seasons vs. Business as usual |
Posttest |
Grade 7: Cohort 1;
|
27.80 |
35.39 |
Yes |
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
6% English language learners -
Female: 48%
Male: 52% -
Suburban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
Maryland
-
Race Asian 14% Black 22% White 46% -
Ethnicity Hispanic 18% Not Hispanic or Latino 82%
Study Details
Setting
The study took place in 10 schools in Maryland’s Montgomery County School District. The student population of this large suburban district was 43% White, 22% African American, 14% Asian American, and 20% Hispanic. The study was part of a multiyear research project called “Scaling up Curriculum for Achievement, Learning, and Equity Project” (SCALE-uP)
Study sample
In this randomized controlled trial, researchers created a sampling frame consisting of five school profile categories, with approximately seven schools in each category. The sampling frame was based on achievement and demographic factors. Each school category had a similar profile determined by: the percentage of students eligible for free and reduced-price meals, math and reading achievement scores, ethnicity, eligibility for English for Speakers of Other Languages (ESOL) services, and eligibility for special education (SPED) services. Two schools were randomly selected from each category to participate in the study. In each category, one school of the matched pair was then randomly chosen to implement the intervention, and the other served as the comparison school. The study school sample consisted of five schools implementing GEMS® The Real Reasons for Seasons and five schools not implementing it. The analysis is based on two cohorts of seventh-grade students that attended the study schools during two consecutive school years. Cohorts 1 and 2 consisted of seventh-grade students in the 2003–04 and 2004–05 school years, respectively. The Cohort 1 analysis sample included 1,318 seventh-grade students who received GEMS® The Real Reasons for Seasons and 1,051 seventh-grade students who received the regular science curriculum. Cohort 2 included 1,287 seventh-grade students who received GEMS® The Real Reasons for Seasons and 1,121 seventh-grade students who received the regular science curriculum. Overall and differential attrition rates of students were low for Cohort 1 (9% and 3%, respectively) and Cohort 2 (13% and 6%, respectively). The study reported student outcomes for the two cohorts after the completion of the unit; these findings are included in the WWC effectiveness rating and can be found in Appendix C. Additional findings for Cohort 1 subgroups by gender, race/ethnicity, eligibility for ESOL services, and eligibility for SPED services are considered supplemental findings by the WWC and can be found in Appendix D.
Intervention Group
The intervention teachers implemented the eight activities of the GEMS® The Real Reasons for Seasons curriculum unit over a period of three weeks. Each activity required about 30–90 minutes of class time. The curriculum unit addressed common misconceptions about seasons and was designed to either validate students’ accurate ideas about seasons or to address common problems students experienced when learning about seasons. The curriculum came with a teacher’s guide, student lab materials, and master copies for duplication or electronic presentation. Montgomery County Public Schools purchased and distributed to teachers all student lab materials needed for use with the unit. The GEMS® The Real Reasons for Seasons curriculum was embedded in a larger astronomy unit using the district-approved curriculum.
Comparison Group
Comparison group teachers used regular curriculum materials normally available to Montgomery County Public Schools’ teachers. The district materials addressed the same instructional benchmarks as the GEMS® The Real Reasons for Seasons curriculum unit.
Outcome descriptions
Students took a concept assessment test for both the pretest and posttest. For Cohort 1, the authors used the Reasons for the Seasons Assessment (RSA). For Cohort 2, the authors used the Causes for the Seasons Assessment (CSA). Although named differently, essentially the same concept test was used for data analysis for both cohorts of students. For a more detailed description of this outcome measure, see Appendix B. Study authors also assessed each student’s personal orientation toward learning using the Science Learning Orientation and Engagement for Students Questionnaire. This outcome measure is outside the scope of the Science review protocol and this review.
Support for implementation
The study did not describe any information about training provided to teachers or staff.
Additional Sources
In the case of multiple manuscripts that report on one study, the WWC selects one manuscript as the primary citation and lists other manuscripts that describe the study as additional sources.
-
Pyke, C., Lynch, S., Kuipers, J., Szesze, M., & Watson, W. (2005). Implementation study of The Real Reasons for Seasons (2004–2005): SCALE-uP Report No. 7. Washington, DC: George Washington University, SCALE-uP.
-
Rethinam, V., Pyke, C., & Lynch, S. (2008). Using multi-level analyses to study the effectiveness of science curriculum materials. Evaluation & Research in Education, 21(1), 18–42.
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).