Ask a REL Response

Afterschool STEM programs — March 2015

Question

Could you provide references on afterschool STEM programs that have demonstrated their impact through rigorous evaluation?

Response

We have prepared the following memo with references on afterschool STEM program evaluation. The literature is quite scarce, though more is known about the impact of afterschool programs in general (see, for example, the Harvard Family Research Project’s page on Out of School Time: http://www.hfrp.org/out-of-school-time).

Citations include a link to a free online version, when available. They are accompanied by an abstract, excerpt, or summary written by the author or publisher of the document. We have not done an evaluation of the methodological rigor of these resources, but provide them for your information only.

Research References

Afterschool Alliance. (2011). STEM learning in afterschool: An analysis of impact and outcomes. Washington, DC: Author. Retrieved from http://www.afterschoolalliance.org/STEM-Afterschool-Outcomes.pdf

Excerpt: This document summarizes evaluation reports from afterschool STEM programs across the United States and identifies common trends and strengths that afterschool learning brings to STEM education. Evaluations presented here were collected by casting a wide net to solicit reports from afterschool programs across the country through various communication channels as well as searching through evaluation databases. Several of the programs used pre- and post-program surveys and focus groups to measure change in students while a few continued tracking their students after they left the programs. There were also some that measured academic achievement through administering pre- and post-program tests as well as recording grades and standardized test scores. Some evaluation studies also surveyed parents and program staff.

Afterschool Alliance. (2013). Partnerships with STEM-rich institutions. Afterschool Alert, Issue Brief No. 61. Washington, DC: Author. Retrieved from http://www.afterschoolalliance.org/issue_briefs/issue_STEM_61.pdf

Abstract: The Afterschool Alliance is proud to present the first in a series of two issue briefs on afterschool STEM programs, generously supported by the Noyce Foundation. The issue brief topics represent emerging discussions within the afterschool field and are drawn from the two award categories of the 2013 Afterschool STEM Impact Awards: (1) partnerships with STEM-rich institutions, and (2) computing & engineering in afterschool. The briefs feature the award winners and other exemplary afterschool programs that are having a significant impact on participants and demonstrate the potential of the afterschool space to contribute to broader national STEM education goals. Afterschool programs around the nation have enthusiastically embraced science, technology, engineering, and math (STEM). Some major afterschool providers, like 4-H and Girls Inc., have long made STEM a priority and in recent years, the vast majority of providers have also come to value providing STEM learning opportunities as an important part of their programming. An Afterschool Alliance poll of afterschool programs conducted in 2010–2011 showed that 99 percent of respondents thought that offering some sort of STEM programming was important, even if that wasn’t the focus of their program. Afterschool STEM programs are proving to be highly effective and they deliver important outcomes. Youth in high-quality afterschool STEM programs show (1) improved attitudes toward STEM fields and careers; (2) increased STEM capacities and skills; and (3) a higher likelihood of graduation and pursuing a STEM career. As interest and commitment to STEM learning in afterschool grows, there is an increased need for support to build the capacity of afterschool programs to offer innovative and robust STEM programming. Afterschool programs have a long and rich history of leveraging community resources to best meet the needs of the youth they serve. They recognize that STEM-rich institutions—science centers and museums, universities and colleges, corporations and businesses, and government agencies—have a lot to offer. All of these institutions in turn have keen interests, and sometimes a major stake, in K–12 STEM education. Partnerships leverage the STEM expertise and resources of the institution—and in combination with the youth development expertise of afterschool professionals, they open up the possibility for incredibly impactful STEM learning opportunities for youth. This issue brief illustrates the power of strong, successful partnerships between afterschool programs and STEM-rich institutions. Additionally, the partnerships described offer promising and innovative models that can have a significant impact on both students and their instructors.

Afterschool Alliance. (2013). Defining youth outcomes for STEM learning in afterschool. Washington, DC: Author. Retrieved from www.afterschoolalliance.org/STEM_Outcomes_2013.pdf

Excerpt: Afterschool programs are increasingly recognized as playing a valuable role in improving science, technology, engineering, and mathematics (STEM) education. However, the expectations for how such programs support young people’s STEM engagement and learning are varied. The Defining Youth Outcomes for STEM Learning in Afterschool study aimed to identify what STEM learning outcomes these program leaders and supporters believe that afterschool programs could contribute to, what the indicators of progress toward such outcomes might be, and what types of evidence could be collected by afterschool programs, without regard to whether or not appropriate data collection tools currently exist…The consensus set of outcomes and indicators produced through this study is not intended to represent a set of mandatory goals for all afterschool STEM programs, as the afterschool STEM field is diverse and impacts are entirely dependent upon the particular circumstances (age of participants, resources, goals, community context) of each program. Rather, the outcomes, indicators and sub-indicators identified through this study are intended to help provide a common framework and language for programs to utilize as they define appropriate goals for their programs and then describe the impact of their afterschool STEM program. This will allow for aggregation of impacts across programs so that we may better describe the contributions of afterschool programs to the larger issues in STEM education.

Bevan, B., Michalchik, V., Bhanot, R., Rauch, N., Remold, J., Semper, R., & Shields, P. (2010). Out-of-school time STEM: Building experience, building bridges: Trends, Questions, and Findings from the Field. San Francisco: Exploratorium, Learning and Youth Research and Evaluation Center (LYREC) and Menlo Park, CA: SRI, International. Retrieved from http://www.integratingengineering.org/after_school/Out%20of%20School%20Time%20STEM_NSF%20AYS%20report.pdf

Excerpt: This report reviews patterns, challenges, and questions developing in the field of out-of-school-time (OST) science, technology, engineering, and mathematics (STEM) programs by drawing on the efforts of some two dozen federally funded programs that participated and presented their work at a conference held in Washington, DC in October of 2009. Reflecting the questions and concerns of both practitioners and researchers in the OST STEM field, the report is intended to inform the work of OST educators, researchers, and funders. [REL West note: see page 13 for research and evaluation frameworks.]

Chi, B., Dorph, R., & Reisman, L. (2008). Evidence & impact: Museum-managed STEM programs in out-of-school settings. Berkeley, CA: University of California, Lawrence Hall of Science. Retrieved from http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_089887.pdf

Excerpt: Commissioned by the Committee on Successful Out-of-School STEM Learning, The National Research Council’s Committee on Successful Out-of-School STEM (Science, Technology, Engineering, and Mathematics) Learning has been tasked with writing a consensus report on the value of and evidence for out-of-school STEM learning programs. As part of its charge, the committee has posed the following questions to be addressed in this paper: What evidence is there for the impact of museum- (and other designed setting) managed programs on STEM learning and interest? What is known about the impact and value of such programs on school-age children’s understanding of STEM concepts and practices as well as their interest and engagement in STEM?

What is known about the characteristics of successful programs? Does the relevant literature provide enough evidence to point to design principles for such programs? This paper responds to these questions focusing on the evidence of the impacts and features of STEM learning programs run by museums and other designed environments, such as science and technology centers, planetariums, zoos, aquaria, etc… To examine evidence of STEM learning programs managed by museums, we conducted a literature review of peer-reviewed journals and books. To build on existing work of the NRC, we focused on articles that have been published since the writing of

Learning Science in Informal Environments (2009) and on a limited set of literature written before 2009 that were not included in LSIE. Given the limitations of the published body of evidence in this area, we also draw from a selected set of evaluation reports and emerging research to enrich our treatment of this topic. We have organized the paper in five main sections:

1. Introduction
2. Evidence
3. Impact
4. Features of Successful Programs, and
5. A Way Forward.

Chi, B., Snow, J. Z., Goldstein, D., Lee, S., & Chung, J. (2010). Project Exploration 10-year retrospective program evaluation summative report. Berkeley, CA: University of California, Lawrence Hall of Science. Retrieved from https://projectexploration.org/wp-content/uploads/2013/04/lhs-final-report-10-year-eval.pdf

Excerpt: This report summarizes findings from a retrospective study of 10 years of project alumni experiences since the founding of Project Exploration (PE), an organization that has fostered youth interest and engagement in science through a framework of youth development principles… The evaluation goals were twofold: (1) describe PE’s influence on its past participants, and (2) explain the organizational practices that support science learning for traditionally underrepresented students in science… The study findings also directly address recommendations that have emerged from a recent report of trends, questions, and findings from the field of out-of-school time STEM to identify ways that OST programs can build the capacities of youth to engage in science, and to examine what features promote quality programming that could be scalable. Although scalability was not a specific focus of this study, the current study helps to illuminate which program features are present in PE programming. Finally, applying a social theory of learning such as “communities of practice” to youth development science programs for underrepresented students helps to frame and further illuminate the outcomes and practices of such programs in preparing science learners and future scientists as well as productive individuals in society.

Chun, K., & Harris, E. (2011). Research update 5: STEM out-of-school time programs for girls. Cambridge, MA: Harvard Family Research Project, Harvard Graduate School of Education. Retrieved from https://eric.ed.gov/?id=ED517977

Abstract: Increasing interest in science, technology, engineering, and mathematics (STEM) has become part of education reform efforts in recent years in order to prepare students for the challenges of the twenty-first century global economy. Out-of-school time (OST) programs that focus on girls’ involvement in STEM can play an essential role in improving female representation in these traditionally male-dominated fields. OST programs offer girls a non-threatening and non-academic environment for hands-on learning that is collaborative, informal, and personal. However, barriers to quality implementation and outcome-based evaluation present challenges for STEM programs serving girls. This Research Update highlights findings from the evaluations and research studies in the OST Database that focus on STEM programs for girls. The six STEM programs covered in this Research Update reflect the diversity of approaches to STEM education, including structured afterschool modules, recreational activities, intensive summer trips, and mentoring. These programs mainly serve youth in middle school and high school, with some targeting girls only, and others available to both boys and girls but with a particular focus on girls.

Fancsali, C. (2003). What we know about girls, STEM, and afterschool programs: A summary. New York: Educational Equity Concepts, Inc. Retrieved from https://secwww.jhuapl.edu/MESA/Content/Docs/whatweknow.pdf

Excerpt: This paper summarizes key research and literature related to what we know about girls’ participation in science, technology, engineering, and mathematics education at the primary, secondary, and postsecondary levels, as well as their participation in afterschool programs with a gender-equity focus and in STEM careers. The summary is organized around five areas:

1. learning styles, teaching strategies, and educational environments promoting girls’ STEM participation and achievement;
2. characteristics of effective STEM and afterschool programs;
3. afterschool program access issues for girls;
4. STEM afterschool program outcomes that have been measured; and
5. examples of STEM programs with evidence of success.

This summary, while not an exhaustive review of the literature and research, provides a basis for discussions related to the Science, Gender and Afterschool working conference funded by the National Science Foundation’s Program for Gender Equity. As a field, afterschool education is growing rapidly and plays a significant role in the development of young people in terms of providing opportunities and promoting positive behavior to foster their academic, vocational and social success. The conference will address two critical issues in terms of how afterschool programs can engage and support all girls’ interest in STEM: 1) how to use available research and evaluation to inform program development and 2) how to ensure the high quality of current and future programs.

Ferreira, M. M. (2001). The effect of an after-school program addressing the gender and minority achievement gaps in science, mathematics, and engineering. ERS Spectrum, 19(2), 11–18.

Abstract: Examines effects of an afterschool science program on minority female students’ attitudes toward science, engineering, and mathematics. The program, which incorporated cooperative learning, hands-on activities, mentoring, and role models, positively influenced participants’ attitudes to (and subsequent participation in) these subject areas.

Hu, X., Craig, S. D., Bargagliotti, A. E., Graesser, A. C., Okwumabua, T., Anderson, C., . . . Sterbinsky, A. (2012). The effects of a traditional and technology-based after-school program on 6th grade student’s mathematics skills. Journal of Computers in Mathematics and Science Teaching, 31(1), 17–38. Retrieved from http://eric.ed.gov/?id=EJ973329

Abstract: This study investigated the effectiveness of the Assessment and Learning in Knowledge Spaces (ALEKS) system as a method of strategic intervention in after-school settings to improve the mathematical skills of struggling 6th grade students. Students were randomly assigned to after-school classrooms in which they either worked with ALEKS to improve their math skills or to classrooms where instruction was provided by certified teachers. Students’ performance on the Tennessee Comprehensive Assessment Program (TCAP), administered annually to all Tennessee students, indicated that students assigned to the ALEKS condition performed at the same level as those taught by expert teachers (d = 0.09). Also, students participating in our after-school program outperformed non-participating students.

Huerta Migus, L. (2008). Broadening access to STEM learning through out-of-school learning environments. Washington DC: National Science Foundation, Committee on Successful Out-of-School STEM Learning. Retrieved from http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_089995.pdf

Excerpt: This paper will explore evidence-based strategies developed in OST STEM programs for successfully engaging youth from underrepresented demographics in STEM learning.

Halpern, D. F., Aronson, J., Reimer, N., Simpkins, S., Star, J. R., & Wentzel, K. (2007). Encouraging girls in math and science. IES practice guide. Washington, DC: National Center for Education Research, U.S. Department of Education. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/20072003.pdf

Abstract: The objective of this guide is to provide teachers with specific recommendations that can be carried out in the classroom without requiring systemic change. Other school personnel having direct contact with students, such as coaches, counselors, and principals, will also find the guide useful.

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K–12 education. Washington, DC: Committee on Integrated STEM Education; National Academy of Engineering; National Research Council. Available for purchase from http://www.nap.edu/catalog/18612/stem-integration-in-k-12-education-status-prospects-and-an

Book description: STEM Integration in K–12 Education examines current efforts to connect the STEM disciplines in K–12 education. This report identifies and characterizes existing approaches to integrated STEM education, both in formal and after- and out-of-school settings. The report reviews the evidence for the impact of integrated approaches on various student outcomes, and it proposes a set of priority research questions to advance the understanding of integrated STEM education. STEM Integration in K–12 Education proposes a framework to provide a common perspective and vocabulary for researchers, practitioners, and others to identify, discuss, and investigate specific integrated STEM initiatives within the K–12 education system of the United States. STEM Integration in K–12 Education makes recommendations for designers of integrated STEM experiences, assessment developers, and researchers to design and document effective integrated STEM education. This report will help to further their work and improve the chances that some forms of integrated STEM education will make a positive difference in student learning and interest and other valued outcomes.

Krishnamurthi, A., Ballard, M., & Noam, G. G. (2014). Examining the impact of afterschool STEM programs. Washington, DC: Afterschool Alliance, and Cambridge, MA: Harvard University. Retrieved from http://files.eric.ed.gov/fulltext/ED546628.pdf

Abstract: Afterschool programs that provide strong science, technology, engineering, and math (STEM) learning experiences are making an impact on participating youth not only become excited and engaged in these fields but develop STEM skills and proficiencies, come to value these fields and their contributions to society, and—significantly—begin to see themselves as potential contributors to the STEM enterprise. This paper begins by presenting some of the research findings about the importance of afterschool and other out-of-school-time (OST) experiences for STEM learning. It then proceeds to summarize evaluation data from a selection of strong afterschool STEM programs and describes the types of substantive impacts these programs are having on participating youth. The paper concludes with recommendations for how an integrated approach to education that includes in-school and OST experiences can be constructed. Appendices provide details of the programs whose outcomes are featured in the narrative. They include: (1) Framework of Youth Outcomes; (2) Descriptions of selected afterschool STEM programs with strong youth outcomes; and (3) Selected evaluation results of afterschool STEM programs with strong youth outcomes.

Lyon, G. H., Jafri, J., & St. Louis, K. (2012). Beyond the pipeline: STEM pathways for youth development. Chicago, IL: After School Matters. Retrieved from http://www.projectexploration.org/wp-content/uploads/2013/04/ASM_2012_16_fall.pdf

Abstract: This report describes how Project Exploration, a STEM pipeline alternative, has identified a disproportionate exit of minority and female students from STEM programs. The authors used a mix of quantitative and qualitative data to assess efficacy of Project Exploration’s efforts to engage minority and female students who traditionally are under-represented in the STEM pipeline. Based on the success of their alternative approach, the authors propose an alternative model of a STEM pipeline that is based on a matrix of progressive competencies.

McClure, P., Rodriguez, D. A., Cummings, F., Falkenberg, K., & McComb, E. M. (2007). Factors related to advanced course-taking patterns, persistence in science, technology, engineering and mathematics, and the role of out-of-school-time programs: A literature review. Greensboro, NC: SERVE Center at University at North Carolina. Retrieved from http://www.gb.nrao.edu/~sheather/new%20lit/Colilition%20for%20Science%20after%20School.pdf

Excerpt: This paper seeks to review the research, evaluation, and experiences related to persistence in STEM by U.S. students and the selection of sequential and advanced STEM courses including

Advanced Placement and International Baccalaureate. The literature on persistence and self-efficacy in STEM is reviewed and overlaid with the best practices in after-school and out-of-school-time programs that provide support for students to engage in, continue in, and develop capacity in STEM. The project was guided by the conceptual framework of Engagement, Capacity, and Continuity (the ECC Trilogy) developed by Jolly, Campbell, and Perlman (2004). Each of these factors is necessary but individually is not sufficient to ensure student continuation in the sciences and quantitative disciplines (p. 3). In this paper, Engagement and Self-Efficacy Factors are addressed first in an exploration of what is essential for students to stay on track in advanced courses and of evidence that after-school programs support engagement and persistence in STEM. In this section, literature on self-regulated learning, self-efficacy, social cognitive theory, and agency were reviewed to illuminate reasons that students persist in STEM education and to augment the ECC Trilogy. Next, in Capacity Factors, the science and mathematics content and skills needed for entry into and success in advanced STEM courses are reviewed along with capacity factors related to after-school staffing and professional development. Finally, sociocultural and institutional factors related to Continuity in STEM are reviewed, and the support roles of after-school programs and community organizations are discussed.

Sahin, A. (2013). STEM clubs and science fair competitions: Effects on post-secondary matriculation. Journal of STEM Education: Innovations and Research, 14(1), 5–11. Retrieved from http://eric.ed.gov/?id=EJ1006871

Abstract: As the global economic competition gets tougher, American policymakers and researchers are interested in finding ways to increase the number of students pursuing STEM (Science, Technology, Engineering, and Mathematics)-related majors in order for the United States to continue its role as an economic powerhouse. A survey study was employed to investigate a multi-charter school system’s (Harmony Public Schools [HPS]) after-school program in which doing a science fair project was expected for all 4th–12th grade students, and students were encouraged to participate in STEM-related clubs (MATHCOUNTS, American Mathematics Competition [AMC], Science Olympiad, University Interscholastic League [UIL], and Science DEMO). In particular, the first part of the study focused on how related matriculation of this specific school system was with the national average (n = 230). In the second part, the relationships between students’ science fair and STEM club participation and their STEM major selection was investigated (n = 149). Findings showed that HPS outperformed the national average in terms of post-secondary admissions and STEM major selections. Multiple years of science fair project competition were positively related with students choosing a post-secondary STEM major (x[squared](4) = 5.32, p = 0. 255). There was also a statistically significant relationship between the number of STEM clubs students participated in and their choice of STEM major (x[squared](4) = 34.22, p less than 0.001). Findings are discussed in light of developing STEM-focused after-school clubs.

Sahin, A., Ayar, M. C., & Adiguzel, T. (2014). STEM related after-school program activities and associated outcomes on student learning. Kuram Ve Uygulamada Egitim Bilimleri, 14(1), 309–322. Retrieved on March 20, 2015, from http://files.eric.ed.gov/fulltext/EJ1038710.pdf

Abstract: This study explores the characteristics of after-school program activities at a charter school in the Southeast U.S. highlighting students’ experiences with and gains from these after-school program activities. A qualitative case study design was employed to understand students’ views and opinions regarding the activities and their learning trajectories. Study data were collected through formal and informal observations, one-on-one semi-structured interviews, and field notes. The study’s findings indicated that such activities emphasize open-ended and collaborative scientific investigations in Science, Technology, Engineering, and Mathematics (STEM) fields and provided an arena for students to demonstrate various uses of 21st century skills. We have described and explained: (a) the importance of collaborative learning groups, (b) the popularity of after-school program activities, (c) interest in STEM fields, and (d) activities’ contribution to developing 21st century skills. These findings show that STEM related activities have the potential to promote collaborative learning and inquiry as well as to contribute to the development of 21st century skills. These findings have also been discussed in light of how STEM related after-school program activities support students’ learning.

Wilkerson, S. B., & Haden, C. M. (2014). Effective practices for evaluating STEM out-of-school time programs. Afterschool Matters (19), 10–19. Retrieved from http://files.eric.ed.gov/fulltext/EJ1021960.pdf

Abstract: Science, technology, engineering, and mathematics (STEM) programs in out-of-school time (OST) are designed to supplement school work, ignite student interest, and extend STEM learning. From interactive museum exhibits to summer-long science camps, opportunities for informal student engagement in STEM learning abound. The differences these programs make, and how they can be improved concern educators and funders alike. OST program developers and providers can benefit from understanding why evaluation is critical to the success of STEM OST programs, what data collection methods are appropriate, and how to effectively communicate and report findings. The authors of this article share lessons from their experience in each of these areas and provide examples of how effective practices play out.

Method

Keywords and Search Strings Used in the Search

(“After school program” OR “out of school time”) AND (“STEM” OR “math” OR “science”)

Search of Databases

EBSCO Host; Google; and Google Scholar

Criteria for Inclusion

When REL West staff review resources, they consider—among other things—four factors:

• Date of the Publication: The most current information is included, except in the case of nationally known seminal resources.
• Source and Funder of the Report/Study/Brief/Article: Priority is given to IES, nationally funded, and certain other vetted sources known for strict attention to research protocols.
• Methodology: Sources include randomized controlled trial studies, surveys, self-assessments, literature reviews, and policy briefs. Priority for inclusion generally is given to randomized controlled trial study findings, but the reader should note at least the following factors when basing decisions on these resources: numbers of participants (Just a few? Thousands?); selection (Did the participants volunteer for the study or were they chosen?); representation (Were findings generalized from a homogeneous or a diverse pool of participants? Was the study sample representative of the population as a whole?).
• Existing Knowledge Base: Although we strive to include vetted resources, there are times when the research base is limited or nonexistent. In these cases, we have included the best resources we could find, which may include newspaper articles, interviews with content specialists, organization websites, and other sources.

This memorandum is one in a series of quick-turnaround responses to specific questions posed by educators and policymakers in the Western region (Arizona, California, Nevada, Utah), which is served by the Regional Educational Laboratory West (REL West) at WestEd. This memorandum was prepared by REL West under a contract with the U.S. Department of Education’s Institute of Education Sciences (IES), Contract ED-IES-12-C-0002, administered by WestEd. Its content does not necessarily reflect the views or policies of IES or the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.