Using data to support the Arkansas Initiative for Instructional Materials (AIIM)

State education agency staff are at the front lines of evaluating educational initiatives. From policy development to student support, staff members navigate competing priorities every day and the data that could be used to evaluate a program's effectiveness may not exist in current information systems. Staff at the Arkansas Department of Education (ADE) are seeking to explore data that can help them understand the extent to which two state initiatives are equipping all students with the skills necessary for success after high school.

Previous research from Regional Educational Laboratory (REL) Southwest found significant disparities in students' performance on indicators of postsecondary readiness and success by race/ethnicity, eligibility for the National School Lunch Program, English learner status, disability status, and age.¹ To help all students prepare for life after high school, the Building Pathways for Student Success (BPSS) partnership is supporting ADE to evaluate the implementation and impact of initiatives that focus on strengthening student success on the path to college and careers.

One such program is the Arkansas Initiative for Instructional Materials (AIIM), which seeks to ensure every student in Arkansas has equitable access to high-quality instructional materials and grade-level instruction.² Arkansas defines high-quality instructional materials, or HQIM, as curriculum materials aligned to state academic standards that include evidence-based strategies, inclusive practices, and embedded teacher supports. These materials are designed to account for the needs and experiences of all learners.³ Additionally, AIIM provides districts and educators with High Quality Professional Learning (HQPL) to support the implementation of HQIM.

Through a series of coaching sessions, REL Southwest is building the capacity of ADE staff to conduct a formative evaluation of AIIM, which includes developing an implementation survey, an evaluation plan, and a structure for continued survey administration. With accurate data on the strengths and areas of growth for the initiative, ADE will be better able to refine the program to improve its implementation and impact, with the goal of ensuring all students have access to HQIM.

Building capacity of ADE staff to evaluate AIIM

Earlier this year, REL Southwest kicked off the coaching series by focusing on logic models and best practices for logic model development to help ADE staff set goals for their evaluation of AIIM. During the session, ADE staff developed a logic model for AIIM with short-, medium-, and long-term outcomes through an interactive activity that allowed the group to refine their vision of the evaluation. The group then formulated research questions aligned with the logic model that focused on both the implementation and impact of AIIM. This work also included a needs sensing activity which Assistant Commissioner of Learning Services, Dr. Pride said, "supported the team's critical thinking around priorities for teaching and learning." By digging deeper into logic model development, ADE staff can better visualize the goals of AIIM and, according to Dr. Pride, "provide a structure for the team to address our problem of practice for AIIM."

After working with ADE staff to set their vision, goals, and foundational logic model for the AIIM evaluation, REL Southwest and ADE staff continued to review best practices for survey development, including recommendations for writing and adapting high-quality survey items. Together, REL Southwest and ADE staff began identifying a clear survey structure and prioritizing survey questions. For example, ADE staff expressed interest in gathering data to understand which districts in Arkansas had adopted HQIM and the kinds of challenges their schools experienced. They also wanted to examine district implementation timelines, whether the timelines differed by grade level or content area, and to what extent districts felt supported by ADE throughout the process. Based on these priorities, REL Southwest and ADE staff reviewed existing surveys for items that could be adapted and co-wrote new items. Ultimately, the survey data will allow ADE to adjust their support for districts adopting HQIM and address common challenges so educators across the state can better implement HQIM in their classrooms.

REL Southwest and ADE staff then collaborated on a draft of the district survey and created a plan for pretesting. REL Southwest proposed the survey item bank and format and worked with ADE to refine items for inclusion in the survey to align with ADE's goals and their need for better data to understand district implementation strategies. Following these conversations, REL Southwest shared best practices for pretesting a survey by gathering systematic feedback from a variety of stakeholders and conducting cognitive interviews to assess the clarity and completeness of the draft survey. ADE staff are using the first draft of the survey to conduct pretesting activities and identify district leaders to complete cognitive interviews. This process is helping ADE staff understand how future respondents may interpret the survey questions. Through the pretesting and feedback gathering activities, REL Southwest and ADE staff are refining the district survey to ultimately gather data that will help ADE staff understand their districts' implementation of HQIM.

Successful Collaboration with ADE

"The partnership with REL Southwest supports ADE with continuous improvement around the implementation of AIIM by providing the structure for evaluation of the initiative," ADE's Dr. Pride said. "The analysis plan that we will generate will provide us with a plan for monitoring and adjusting AIIM to meet the needs of educators, students, and families."

REL Southwest will continue to support ADE's AIIM team with two more coaching sessions. The fourth session in the series will focus on survey administration and evaluation plan development, including working with ADE to plan how the survey data will be analyzed to examine the implementation of AIIM in the 2024/25 school year. During the fifth and final session, REL Southwest and the AIIM team will review and finalize the evaluation plan.

Related Resources

• Blog: Supporting postsecondary readiness and success for all students in Arkansas
• Infographic: Leveraging High-Quality Instructional Materials to Support Student Success