Grant Closed

Summer School and Summer Learning: An Examination of Selection, Implementation, and Program Effects in a Multiyear Randomized Trial

NCER

Program: Education Research Grants

Program topic(s): Improving Education Systems

Award amount: $1,176,686

Principal investigator: Keith Zvoch

Awardee:

University of Oregon

Year: 2009

Project type:

Efficacy

Award number: R305A090369

Purpose

The purpose of this study is to conduct a rigorous evaluation of a multiyear summer school literacy program initiative delivered to kindergarten, first, and second grade students identified as at-risk for future reading difficulty and aimed at closing the performance gap between strong and struggling readers and to ensure that struggling readers gain the skills requisite to meet reading proficiency targets.

Project Activities

By assigning eligible students to a summer school literacy program or the control condition (without a summer school program) based on whether they scored immediately above or below a cut point, the research team will be able to use a modified regression discontinuity model to develop unbiased estimates of the effect of the summer school intensive literacy program.

Structured Abstract

Setting

The study is being conducted in a moderately-sized school district located in the Pacific Northwest, serving approximately 5,000 students each year distributed across seven elementary schools, two middle schools, and two high schools with a focus on the 2009, 2010, and 2011 summer program implementations.

Sample

Summer school attendees are those who completed kindergarten, first, or second grade in the preceding academic year and did not meet an established proficiency cut point on the spring administration of a grade appropriate progress monitoring assessment. Approximately half of the students receive a free or reduced price lunch.

Intervention

The intervention is an academically intensive summer literacy program for students who completed their kindergarten, first grade or second grade year but did meet a proficiency standard. It is delivered to approximately 250 students a year in one dozen mixed grade (K–2) classrooms at one central school site. The program runs over a five week period during the middle of the three month summer vacation period. Instruction occurs 3.5 hours per day four mornings per week in small class (less than 20 students). Students receive a minimum of two hours of teacher-directed daily reading instruction in both whole and small group settings in the beginning reading skills of phonemic awareness (oral blending and segmentation), alphabetic understanding (letter-sounds, decoding, phonic analysis), and fluency.

Research design and methods

The study includes three designs: an interrupted time series design, a regression discontinuity design, and an experimental design. The summer school program began in the summer of 2004. Baseline learning rates will be estimated using the 2000–2004 data. Using the cut point at which students become eligible for the program, a non-equivalent group interrupted time series (ITS) design can be modeled to test for changes in the intercept and slope of strong (above the cut) and struggling (below the cut) readers using data from 2005 to 2011 with a change in slope only expected for the struggling readers.

Control condition

For each design the comparison group will vary. For the interrupted time series design, the comparison group is the cohorts of students attending kindergarten through second grade before the summer program was offered. For the regression discontinuity design, students who scored above the lower cut off point and did not attend the summer program will be the comparison group. For the experimental design, students who scored in the cut off interval but were not assigned to summer school will be the control group.

Key measures

Literacy outcomes will be progress monitoring scores obtained on district interim assessments and scores obtained on the third grade state proficiency test. During kindergarten and 1st, the district administers the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) three times over the year. In 1st and 2nd grade, the district administers the Test of Oral Reading Fluency three times a year. Reading scores on the third grade Oregon Assessment of Knowledge and Skills (OAKS) will serve as the final outcome measure.

Data analytic strategy

Multilevel techniques will be applied in many analyses as data will be nested to varying degrees and as longitudinal cohort models will be estimated.

People and institutions involved

IES program contact(s)

Corinne Alfeld

IES Education Research Analyst

NCER

Products and publications

Products: Products from this project include descriptive and analytic reports of study findings.

Journal article, monograph, or newsletter

Zvoch, K. (2012). How Does Fidelity of Implementation Matter? Using Multilevel Models to Detect Relationships Between Participant Outcomes and the Delivery and Receipt of Treatment. American Journal of Evaluation, 33 (4): 541-559.

Zvoch, K., and Stevens, J.J. (2011). Summer School and Summer Learning: An Examination of the Short- and Longer Term Changes in Student Literacy. Early Education and Development, 22 (4): 649-675.

Zvoch, K., and Stevens, J.J. (2013). Summer School Effects in a Randomized Field Trial. Early Childhood Research Quarterly, 28 (1): 24-32.

Zvoch, K., and Stevens, J.J. (2015). Identification of Summer School Effects by Comparing the In- and Out-of-School Growth Rates of Struggling Early Readers. Elementary School Journal, 115 (3): 433-456.

Zvoch, K. (2016). The Use of Piecewise Growth Models to Estimate Learning Trajectories and RTI Instructional Effects in a Comparative Interrupted Time-Series Design. The Elementary School Journal, 116 (4), 699-720.

Zvoch, K., and Robertson, M. C. (2017). Multivariate Summer School Effects. Studies in Educational Evaluation, 55 , 145-152.

Nongovernment report, issue brief, or practice guide

Zvoch, K., and Stevens, J.J. (2010). Elementary Students' Attitudes About Reading. Bethel School District: Eugene, OR.

Zvoch, K., Melton, J.A., Fukuda, E., and Stevens, J.J. (2012). The Status and Change in Bethel Elementary Students Summer Reading Activities. Eugene, OR: Bethel School District.

** This project was submitted to and funded under Education Policy, Finance, and Systems in FY 2009.

Supplemental information

For the summers of 2000, 2010, and 2011, a cut off interval will be used set by a lower and upper cut score. Students falling below the lower score will be assigned to summer school and will be compared with students scoring above that score and not attending summer school under a regression discontinuity design. Students scoring in the cut off interval will be randomly assigned to summer school or no summer school creating an experimental design.

Fidelity of implementation will be monitored through a combination of classroom observation, student attendance, homework completion, and a survey of treatment and control students regarding the summer's activities given at the start of the next school year.

Two-level models will be specified to account for the nesting of test scores within students and the nesting of students within classrooms. Two-level longitudinal growth models (observations within students) will be estimated when data from an entire student cohort is considered as students change academic year classrooms each year and as some students (i.e., non-participants) do not experience a nested classroom structure in the summer. When the focus is estimating literacy outcomes for students receiving treatment (i.e., implementation-by outcome), two-level difference score models will be specified to account for the nesting of students in summer classrooms and to allow for examination of implementation effects on summer literacy change. Piecewise growth models will be applied in instances where student literacy progress is being tracked across one or more academic years and the intervening summers. Single-level regression models will be applied when only changes in literacy across the summer months for treatment and control students are to be estimated.

Questions about this project?

To answer additional questions about this project or provide feedback, please contact the program officer.

Summer School and Summer Learning: An Examination of Selection, Implementation, and Program Effects in a Multiyear Randomized Trial

Purpose

Project Activities

Structured Abstract

Setting

Sample

Research design and methods

Control condition

Key measures

Data analytic strategy

People and institutions involved

IES program contact(s)

Corinne Alfeld

Products and publications

Supplemental information

Questions about this project?

Questions about this project?

You may also like

Meta-Analysis Training Institute (MATI)

Summer Research Training Institute on Cluster-Rand...

Data Science for Education (DS4EDU)