Even as many students returned to school buildings in fall 2021, schooling this year is far from a return to normalcy. Educators are faced with the challenge of supporting students who are struggling academically, socially, and emotionally. Trying new strategies and interventions to meet these wide-ranging student needs can be challenging and expensive, and there is no guarantee that these efforts will have their desired effects. What if there was a way to gather information about how well a new strategy or intervention was working, before investing resources into broad implementation? Implementation pilot studies can do just that. Through its coaching and technical support activities with the Tennessee Department of Education, REL Appalachia created a resource that can walk you through the process of conducting one.

What is an implementation pilot study?

An implementation pilot study allows you to “test” out a new program or initiative in a small sample of real-world education settings and gather data about how it is working. State and local education leaders can use the information learned from an implementation pilot study to inform their decisionmaking on whether to scale up the program or initiative. Pilot studies can also provide information about the supports and conditions that are necessary when scaling up implementation. Implementation pilot studies can answer questions such as:

• Do teachers see the initiative as valuable for helping students?
• Do teachers have the time, organizational supports, and technology they need to implement the initiative?
• Did teachers implement the initiative as intended? If not, why not?
• How have teachers adapted the initiative for use in their classrooms?

How do you conduct a pilot study?

REL Appalachia created Learning Before Going to Scale: An Introduction to Conducting Pilot Studies to walk education leaders through a seven-step process for designing, executing, and engaging with the results from a pilot study. The figure below describes each of the seven steps.

What does an implementation pilot study look like in practice?

This year, REL Appalachia has been supporting educators in eastern Tennessee to conduct an implementation pilot study. In response to widespread concerns about learning loss due to the COVID-19 pandemic, the Niswonger Foundation designed an intensive small-group literacy tutoring program. The program, called Project On-Track, makes use of regular progress monitoring, one-on-one activities for each student based on their needs, and a computerized reading program. The Niswonger Foundation plans to implement the program for three years, both in the summer and during the school year. The program launched with a four- to six-week implementation in summer 2021 in about 25 sites in eastern Tennessee, which the program staff referred to as their “soft opening.” While not specifically designed as a pilot, the soft opening provided an ideal opportunity to use the pilot study resource, as project leaders wanted to learn as much as they could from the summer implementation so they could improve their model and the supports they offered when they expanded their implementation to more sites in the fall.

Design, execute, and engage with results

Project On-Track leadership assembled a core working group of tutoring site coordinators who they worked with to design the pilot study. With REL Appalachia support, they worked through the three steps in the design phase, including establishing roles and responsibilities and identifying pilot study questions. The working group also developed a plan for data collection, which included administering a survey to tutors and conducting focus groups with tutoring site coordinators to gather data on implementation. Project On-Track leadership then worked through the two steps to execute the study: identifying who to collect data from and collecting the data. Finally, Project On-Track leadership engaged with the results by analyzing the data with REL Appalachia support and then holding a meeting with the working group to review the results and identify next steps.

Project On-Track leadership learned some valuable lessons from their pilot study that they were able to use to improve implementation in the fall. For example, they learned that site coordinators and tutors liked the reading program and student activities, students found the program engaging, and tutors and site coordinators felt the program complemented other instructional materials used in the school. However, due to the quick start-up of the program in the summer, the timeline for training was very short. Tutors reported that they could have benefitted from additional training before the program began. They also reported that they needed time to plan as they were implementing the program. Project On-Track used this information to inform their implementation this fall by expanding their initial training and adding planning time each week for tutors.

Resources for ongoing learning

The pilot study resource was helpful for Project On-Track leadership to understand what an implementation pilot study is, and how it might be of benefit to them. REL Appalachia met with project leaders regularly to support them in using the resource to conduct their own pilot study.

A pilot study can be conducted for almost any initiative or program. If you have one in mind and want to conduct an implementation pilot study, you can access the REL Appalachia pilot study resource here as well as these additional resources to support you in the process:

• Tips and Recommendations for Successfully Pilot Testing Your Program: A Guide for the Office of Adolescent Health and Administration on Children, Youth, and Families Grantees provides information about why you might want to conduct a pilot study and some recommendations to consider as you plan your pilot study.
• These planning materials from REL Southwest were created to support educators to conduct a pilot study of an early learning inventory to assess children's knowledge and skills when they entered kindergarten.