In 2017, the Colorado School Improvement Research Partnership, which includes Colorado Department of Education (CDE) staff and REL Central researchers, established the Program Evaluation Professional Development Modules project to build the capacity of CDE staff who monitor the implementation and outcomes of school improvement programs. From the project five research-based modules were developed, initially designed to be easily modified for training CDE staff working with different state programs, the modules will now also support district level staff to monitor school improvement programs. Each module covers a specific program evaluation topic and contains presentation slides, guides, reports, and other related materials, along with a web-based activity for staff to practice applying the content that they learn.
Joshua Stewart, a REL Central senior researcher and the project lead, explained that “REL Central has been collaboratively working with CDE to make module materials as user-friendly and accessible as possible for district staff.”
Since the modules were created to be independent of specific CDE programs, their usefulness extends beyond the Colorado context to three other REL Central projects, two of which are in different states.
For example, REL Central is supporting the North Dakota Department of Public Instruction (NDDPI) in evaluating the Teacher Shortage Education Loan Forgiveness Program. As part of this project, formed by members of the Rural Education Research Alliance, REL Central has adapted the CDE modules to help NDDPI develop a program logic model, evaluation questions, and a data matrix. NDDPI will use these modules to better understand if the loan forgiveness program is supporting teacher recruitment and retention as intended.
“Through this work, NDDPI identified that additional survey questions might be needed to fully answer evaluation questions,” said Douglas Gagnon, a REL Central senior researcher and lead for the project. “We were able to flexibly apply the tools in the existing module and adapt our support to include coaching that led to the development of additional survey items.”
In addition, American Indian Education Research Alliance members working on the Data Collection and Analysis Plan for American Indian Education Programs project are using the modules to provide South Dakota Department of Education (SD DOE) staff with an adaptable framework for applying evaluation theories in a culturally sensitive manner.
As project lead Kerry Englert explained, “REL Central is adopting the evaluation modules to provide technical assistance on developing logic models, specifying evaluation questions, and aligning data collection activities. The modules are part of the support that are helping SD DOE to implement specific culturally appropriate activities to address the needs of their students, including Native language instruction, trauma-informed practices, and the indigenization of their curriculum.”
Finally, as part of the Early Childhood Education Research Alliance, REL Central is using three of the modules to support CDE staff in their efforts to better understand the effectiveness of Early Childhood Professional Credential 2.0 system. The logic model, data matrix, and analysis will help CDE create a framework for researching the credential system, which is designed to provide a framework for early childhood educators to document their formal education, professional development activities, professional experience, and demonstrated competencies. CDE plans to use this data to make adjustments to the credential system to support preparation of effective early childhood educators.
As these various projects demonstrate, the program evaluation modules offer education agencies opportunities to develop targeted evaluation plans to address their critical education needs. With the support of REL Central, our stakeholders are not only learning important program evaluation strategies and developing plans for existing projects, but also building capacity to create and apply program evaluation plans beyond those current projects to support further improvement.
