REL Midwest Ask A REL Response

June 2019

Afterschool Alliance. (2008). Evaluations backgrounder: A summary of formal evaluations of the academic impact of afterschool programs. Washington, DC: Author. Retrieved from https://eric.ed.gov/?id=ED502307.

From the ERIC abstract: “Although afterschool programs for children have been operating for decades in some communities, the afterschool movement—the great national awakening to the opportunity afterschool offers—is just a few years old. As public demand for afterschool has grown, so has the demand for accountability. That is particularly true in afterschool programs that spend public dollars. After all, where tax dollars flow, so must accountability to taxpayers. Fortunately for afterschool advocates, a steady stream of afterschool evaluations are documenting gains for children, especially those who regularly participate in afterschool programs and those at highest risk of academic failure. This updated evaluations backgrounder includes summaries of: (1) a meta-analysis covering 35 studies, by Mid-continent Research for Education and Learning (McREL); (2) a meta-analysis covering 73 studies, by the Collaborative for Academic, Social, and Emotional Learning (CASEL); (3) a study of 35 high-quality afterschool programs in California, Colorado, Connecticut, Michigan, Montana, New York, Oregon and Rhode Island, by researchers at the University of California, Irvine, the University of Wisconsin-Madison and Policy Studies Associates, Inc.; (4) a new study of Chicago’s After School Matters, by researchers at the Chapin Hall Center for Children at the University of Chicago; (5) a study of academically focused New Hampshire programs by the RMC Research Corporation; and (6) a new study of the five-city, California-based Communities Organizing Resources to Advance Learning (CORAL), conducted by Public/Private Ventures. Summaries of other evaluations cover studies of LA’s BEST, Citizen Schools, the YMCA of Greater New York’s Virtual Y Program, the Young Scholars Program, Generacion Diez, 21st Century Community Learning Centers throughout Texas, the Massachusetts After-School Research Study, The After-School Corporation (TASC), Foundations Inc., Project Learn, San Diego’s ‘6 to 6’ and more.”

Beckett, M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., et al. (2009). Structuring out-of-school time to improve academic achievement (NCEE 2009-012). Princeton, NJ: What Works Clearinghouse. Retrieved from https://eric.ed.gov/?id=ED505962.

From the ERIC abstract: “Out-of-school time programs can enhance academic achievement by helping students learn outside the classroom. The purpose of this practice guide is to provide recommendations for organizing and delivering school-based out-of-school time (OST) programs to improve the academic achievement of student participants. The five recommendations in this guide are intended to help district and school administrators, out-of-school program providers, and educators design out-of-school time programs that will increase learning for students. These recommendations are: (1) Align the OST program academically with the school day; (2) Maximize student participation and attendance; (3) Adapt instruction to individual and small group needs; (4) Provide engaging learning experiences; and (5) Assess program performance and use the results to improve the quality of the program. The guide also describes the research supporting each recommendation, how to carry out each recommendation, and how to address roadblocks that might arise in implementing them. The scope of this practice guide is limited to programs that (1) serve elementary and middle school students; (2) are organized by or conducted in partnership with a school or school district; and (3) aim to improve academic outcomes.”

Deschenes, S. N., Arbreton, A., Little, P. M., Herrera, C., Grossman, J. B., & Weiss, H. B. (2010). Engaging older youth: Program and city-level strategies to support sustained participation in out-of-school time. Cambridge, MA: Harvard Family Research Project. Retrieved from https://eric.ed.gov/?id=ED509442. Full text available from https://archive.globalfrp.org/out-of-school-time/publications-resources/engaging-older-youth-program-and-city-level-strategies-to-support-sustained-participation-in-out-of-school-time.

From the ERIC abstract: “Out-of-school time (OST) programs represent a vital opportunity and resource for learning and development for children and youth. Given the potential of city-level OST initiatives to support participation, and against the national backdrop of inequitable access to quality OST programs for older youth from disadvantaged communities, The Wallace Foundation commissioned this research study. To understand how to promote sustained participation in OST programs, this study examined the program characteristics—both program practices and structural features—associated with high participation and retention that were employed by OST programs, primarily serving disadvantaged youth, in six cities that have worked toward building OST initiatives. In particular, this report addresses how OST programs keep middle and high school youth engaged over time (i.e., the duration of participation) and how the supports that city initiatives provide can help foster youth participation, with the assumption that programs can have a potentially greater impact if they are able to work with these youth over an extended period of time.”

Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45(3–4), 294–309. Retrieved from https://link.springer.com/article/10.1007/s10464-010-9300-6.

From the abstract: “A meta-analysis of after-school programs that seek to enhance the personal and social skills of children and adolescents indicated that, compared to controls, participants demonstrated significant increases in their self-perceptions and bonding to school, positive social behaviors, school grades and levels of academic achievement, and significant reductions in problem behaviors. The presence of four recommended practices associated with previously effective skill training (SAFE: sequenced, active, focused, and explicit) moderated several program outcomes. One important implication of current findings is that ASPs should contain components to foster the personal and social skills of youth because youth can benefit in multiple ways if these components are offered. The second implication is that further research is warranted on identifying program characteristics that can help us understand why some programs are more successful than others.”

Note: REL Midwest was unable to locate a link to the full-text version of this resource. Although REL Midwest tries to provide publicly available resources whenever possible, it was determined that this resource may be of interest to you. It may be found through university or public library systems.

Herrera, C., Grossman, J. B., & Linden, L. L. (2013). Staying on track: Testing Higher Achievement’s long-term impact on academic outcomes and high school choice. New York, NY: MDRC. Retrieved from https://eric.ed.gov/?id=ED545471.

From the ERIC abstract: “One crucial decision that middle schoolers (and their families) make is where they will attend high school. Many districts employ school choice systems designed to allow students to pick a high school that will meet their needs and interests. Yet most students prefer high schools that are close to home, and for youth in low-income neighborhoods, this often means attending a more disadvantaged, lower performing school (Nathanson et al. 2013). Youth who defy these odds and choose a competitive high school instead have much to gain. Cullen et al. (2005), for instance, found that Chicago public middle school students who chose to attend a higher-achieving high school were substantially more likely to graduate. However, even as eighth graders, these students already differed in many ways from their peers who chose a neighborhood school—they had better self-reported grades and higher expectations for the future, felt more prepared for high school, and were more likely to have spoken with their parents about what school to attend. These findings raise the question of how we can prepare more disadvantaged students to take the many steps necessary-throughout the middle school years-to successfully transition to a competitive, high-quality high school that can ultimately launch them toward college and careers. The Washington, DC-based Higher Achievement program is taking on this challenge. Higher Achievement targets rising fifth and sixth graders from ‘at-risk communities’ and serves them throughout the middle school years. Its goal is to strengthen participants’ academic skills, attitudes and behaviors, reinforce high aspirations and help students and their families navigate the process of applying to and selecting a high-quality high school. In 2006, the authors began a comprehensive multi-year evaluation of Higher Achievement to test its impact on participants’ academic performance, attitudes and behaviors and on their high school enrollment. The evaluation used random assignment-the most rigorous design available to researchers-to assess program impacts. This brief summarizes the study’s findings. Findings suggest that the program does appear to expand the options available to its students by making them more likely to apply to and attend private schools and less likely to apply to and attend weaker public magnet and charter schools. This, in turn, may position youth for better outcomes in high school and beyond.”

Hurd, N., & Deutsch, N. (2017). SEL-focused after-school programs. The Future of Children, 27(1), 95–115. Retrieved from https://eric.ed.gov/?id=EJ1145092.

From the ERIC abstract: “After-school programs offer young people opportunities for self-expression, exploring their talents, and forming relationships with supportive adults. That is, after-school programs promote young people’s social and emotional learning (SEL) skills—whether the programs use that term or not. Despite these programs’ potential, Noelle Hurd and Nancy Deutsch write, they have yet to make a big impact on the field of SEL. One reason is that studying them poses many problems for researchers—for example, attendance is not mandatory, meaning that it can be hard to separate a program’s effects from young people’s personal characteristics that led them to choose the program in the first place. Still, research shows that after-school programs can promote many desirable SEL outcomes, and Hurd and Deutsch outline the factors that make high-quality programs stand out. How could policy help after-school programs promote SEL more effectively? First, positive youth-staff relationships are crucial to effective programs, and competent adult staff are the linchpin of effective after-school programs targeting SEL outcomes. Yet the after-school work force is poorly paid, and turnover is high. Hurd and Deutsch suggest several ways to professionalize after-school work—for example, by boosting professional development and creating more opportunities for career advancement. Second, as schools have become more focused on standardized test scores, funders and policymakers have pushed after-school programs, too, to demonstrate their academic impact. Hurd and Deutsch write that this approach is misguided: overemphasizing academic outcomes leads to neglect of SEL outcomes that can help young people become productive and engaged citizens. They argue for expanding the criteria used to determine whether after-school programs are effective to include SEL. More broadly, they write, high-stakes evaluations create a disincentive for programs to undertake the difficult work of assessing and improving their own practices. A better approach to evaluation would focus less on whether programs ‘work’ and instead seek ways to make them work better.”

Kidron, Y., & Lindsay, J. (2014). The effects of increased learning time on student academic and nonacademic outcomes: Findings from a meta-analytic review (REL 2014-015). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Appalachia. Retrieved from https://eric.ed.gov/?id=ED545233.

From the ERIC abstract: “REL Appalachia conducted a systematic review of the research evidence on the effects of increased learning time. After screening more than 7,000 studies, REL Appalachia identified 30 that met the most rigorous standards for research. A review of those 30 studies found that increased learning time does not always produce positive results. However, some forms of instruction tailored to the needs of specific types of students were found to improve their circumstances. Specific findings include: (1) Increased learning time promoted student achievement in mathematics and literacy when instruction was led by a certified teacher and when teachers used a traditional instructional style (i.e., the teacher is responsible for the progression of activities and students follow directions to complete tasks); (2) Increased learning time improved literacy outcomes for students performing below standards; and (3) Increased learning time improved social-emotional skills of students with attention deficit/hyperactivity disorder.”

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76(2), 275–313. Retrieved from https://eric.ed.gov/?id=EJ751154.

From the ERIC abstract: “Schools and districts are adopting out-of-school-time (OST) programs such as after-school programs and summer schools to supplement the education of low-achieving students. However, research has painted a mixed picture of their effectiveness. To clarify OST impacts, this synthesis examined research on OST programs for assisting at-risk students in reading and/or mathematics. Researchers analyzed 35 OST studies that employed control or comparison groups and met other inclusion criteria. Meta-analyses indicated small but statistically significant positive effects of OST on both reading and mathematics student achievement and larger positive effect sizes for programs with specific characteristics such as tutoring in reading. Whether the OST program took place after school or during the summer did not make a difference in effectiveness.”

Note: REL Midwest was unable to locate a link to the full-text version of this resource. Although REL Midwest tries to provide publicly available resources whenever possible, it was determined that this resource may be of interest to you. It may be found through university or public library systems.

Leos-Urbel, J. (2015). What works after school? The relationship between after-school program quality, program attendance, and academic outcomes. Youth & Society, 47(5), 684–706. Retrieved from https://eric.ed.gov/?id=EJ1070820.

From the ERIC abstract: “This article examines the relationship between after-school program quality, program attendance, and academic outcomes for a sample of low-income after-school program participants. Regression and hierarchical linear modeling analyses use a unique longitudinal data set including 29 after-school programs that served 5,108 students in Grades 4 to 8 over 2 years. Program quality measures, based on activity observations, include supportive environment, opportunities for purposeful engagement, and structured interactions. Findings suggest that middle school students attend programs with greater purposeful engagement less often, while attendance for younger students is less sensitive to program quality. Greater purposeful engagement is associated with lower test scores for elementary and middle school students. In contrast, a more supportive environment and greater opportunities for structured interactions relate to improvements in test scores. Findings are discussed in light of ongoing policy debate regarding the proper focus of after-school programs and concerns about poor program attendance.”

Note: REL Midwest was unable to locate a link to the full-text version of this resource. Although REL Midwest tries to provide publicly available resources whenever possible, it was determined that this resource may be of interest to you. It may be found through university or public library systems.

Neild, R. C., Wilson, S. J., & McClanahan, W. (2019). Afterschool programs: A review of evidence under the Every Student Succeeds Act. Philadelphia, PA: Research for Action. Retrieved from https://eric.ed.gov/?id=ED593600.

From the ERIC abstract: “This review summarizes virtually all available evidence on the effectiveness of specific afterschool programs, based on a comprehensive literature search and review of studies published in 2000 or later. The review is motivated, in part, by a growing emphasis on using rigorous assessment of program impacts to inform decision-making in education and youth development programs. This review uses the Every Student Succeeds Act (ESSA) evidence framework to assess the evidence of afterschool program effectiveness. ESSA describes a tiered evidence framework of four levels, or tiers: Strong (Tier I), Moderate (Tier II), Promising (Tier III), and a fourth category that has been titled Demonstrates a Rationale (Tier IV) in September 2016 guidance from the U.S. Department of Education. In this review, the authors use ESSA’s evidence requirements and the Department’s guidance to identify afterschool programs that are judged to be supported by evidence at each tier. The authors also provide further definition for Tiers III and IV, which are described only generally in both ESSA and the guidance. Key findings include: (1) There is a substantial set of afterschool programs with evidence of effectiveness meeting ESSA Tiers I-III, including branded and unbranded programs; (2) Effective afterschool programs can be found at each grade level and within almost every program type; and (3) There are more afterschool program options for improving mathematics and reading/ELA achievement than for improving other outcomes.”