Co-Principal Investigators: Vanek, Jenifer

Related Network Teams: This project will serve as the network research team for the CREATE Adult Skills Network, which conducts research and dissemination to support the millions of U.S. adults who have low foundational skills and includes the following other projects: Adult Skills Network Lead(R305N210014), Adult Numeracy in the Digital Era: Adaptive Technology for Quantitative and Digital Literacy (R305N210029), Writing in Adult Secondary Education Classes (W-ASE) (R305N210030), Adult Skills Assessment Program: Actionable Assessments for Adult Learners (R305N210031), Content-Integrated Language Instruction for Adults with Technology Support(R305N210032), Developing and Implementing a Technology-Based Reading Comprehension Instruction System for Adult Literacy Students (R305A200413)

Kick-off Webinar available for viewing here

Purpose: This project explores the rollout of the Teaching Skills That Matter (TSTM) initiative of the U.S. Department of Education's Office of Career, Technical, and Adult Education (OCTAE) and will develop and pilot test instructional resources for TSTM, delivered through an online platform called SkillBlox. The TSTM initiative seeks to align the skills teachers focus on in the classroom and students' needs and interests in part by providing the TSTM framework. This framework focuses on nine intra/interpersonal competencies (such as communication, critical thinking), five topic areas (civics education, digital literacy, financial literacy, health literacy, workforce preparation), and three teaching approaches (integrated and contextualized learning, problem-based learning, project-based learning). OCTAE has been supporting the rollout of TSTM by using a train-the-trainer model, in which a small set of master teachers in each state receive training and then train other teachers, with the support of state professional development coordinators. However, it is not clear what elements of the rollout improve outcomes, and educators need additional digital resources to sustain and scale their use of TSTM. This project will address these gaps.

Project Activities: The researchers will conduct exploratory work to understand the context of implementation of TSTM to date. They will leverage this information to guide the development of digital educational resources by refining the online SkillBlox platform and working with instructors to increase its content and usability while also aligning content to TSTM. They will then pilot test the fully developed platform and conduct a cost analysis of TSTM-aligned SkillBlox.

Products: The products of this project will include an analysis of and reports on the TSTM rollout; a developed platform and open educational resources (OERs) to support adult education learners, educators, and programs; initial evidence of the promise of the TSTM-aligned OER platform including analysis of its costs; and peer-reviewed publications.

Structured Abstract

Setting: The study will focus on adult education programs in 10 states—5 states where the TSTM initiative was first piloted (Arizona, Arkansas, Illinois, Pennsylvania, and Virginia) and 5 to be sampled from the remaining 18 TSTM states.

Sample: During the exploratory phase, 40 master teachers and 200 classroom teachers will participate. During the development phase of the SkillBlox, up to 200 teachers and 1,000 students will participate. The pilot test of the TSTM-aligned version of SkillBlox will include up to 150 programs, 1000 adult education teachers, and more than 25,000 students in the study states.

Intervention: The TSTM-aligned version of SkillBlox will build off an emerging SkillBlox platform that organizes instructional content according to adult education topics and skill levels as defined by the College and Career Readiness Standards for Adult Education. Additional updates to SkillBlox are underway that provide expanded functionality for both instructors and students. Instructors will be able to locate, organize, share, and assign content by TSTM skills and topic areas, and students will be able to access and save their work on assigned skill blocks. This project will add extensive TSTM content to SkillBlox and make enhancements based on usability testing with instructors. The resulting pilot version of TSTM-aligned SkillBlox will be designed to allow instructors to easily locate a TSTM-aligned skill they want to teach; select relevant lessons and high quality OERs, such as articles, videos, games, simulations, and assessments they wish to use to teach that skill; and then assign skill blocks to students through the platform.

Research Design and Methods: In the first phase of work, the researchers will explore the current rollout of TSTM in 10 study states, using a mixed-methods design that combines (a) qualitative data from interviews and focus groups with (b) quantitative, state-provided administrative data on program participation and learner outcomes used for OCTAE's National Reporting System (NRS). In the second phase, the researchers will collaborate with practitioners and students to iteratively develop the TSTM-aligned SkillBlox platform and associated OER content and supports and conduct usability and feasibility testing. For the iterative development, they will use a design-based process to refine the initial SkillBlox design by testing and revising design assumptions through iterative sprint cycles. During the third phase, the researchers will test the promise of the intervention using a randomized control trial (RCT) in multiple states and program sites. The RCT will also include an implementation and cost study.

Control Condition: The control group will have access to regular professional development opportunities and supports available in the 10 study states, including those related to TSTM, as well as publicly available OER. However, they will not be able to access the TSTM-aligned version of SkillBlox.

Key Measures: The exploratory and pilot study measures include both teacher and student outcomes. Teacher outcomes include measures of instructional practices, use of instructional materials (including OER), self-efficacy, student expectations, and familiarity with the principles of the TSTM initiative. Student outcomes include course completion; measurable skill gains; and post-program credential attainment, postsecondary enrollment, and employment. Measures for the design and development effort include OER usage, teacher feedback on utility and effectiveness of SkillBlox features and OER, student feedback on utility and relevance of TSTM-aligned OER developed via SkillBlox, and teacher and student feedback on SkillBlox content and iterative prototypes.

Data Analytic Strategy: For the exploratory mixed-methods study, the researchers will use a number of approaches, including thematic analysis, summary statistics, and simple regression analyses. For the iterative development, they will rely primarily on qualitative, inductive, open coding and thematic analysis. For the pilot study analysis, the researchers will use multilevel regression methods to measure the impact of TSTM-aligned SkillBlox, controlling for contextual program-, teacher-, and student-level background variables, along with qualitative thematic methods to measure implementation and identify potential barriers in using TSTM-aligned SkillBlox.

Cost Analysis: The researchers will use the "ingredients" method to estimate the cost of using TSTM-aligned SkillBlox and related OER, such as the cost for the time teachers spend learning how to use these digital supports for TSTM.