Co-Principal Investigators: Oliveri, Maria Elena; Sabatini, John; Zenisky, April L.

Related Network Teams: This project will serve as the network research team for the CREATE Adult Skills Network, which conducts research and dissemination to support the millions of U.S. adults who have low foundational skills and includes the following other projects: Adult Skills Network Lead(R305N210014), Teaching Skills That Matter (TSTM)–SkillBlox Research Team(R305N210025), Adult Numeracy in the Digital Era: Adaptive Technology for Quantitative and Digital Literacy (R305N210029), Writing in Adult Secondary Education Classes (W-ASE) (R305N210030), Content-Integrated Language Instruction for Adults with Technology Support(R305N210032), Developing and Implementing a Technology-Based Reading Comprehension Instruction System for Adult Literacy Students (R305A200413)
Kick-off Webinar available for viewing here

Purpose: The Adult Skills Assessment Project (ASAP) research team will create a digital, online repository of assessment tasks in literacy, numeracy, and the intersection of these two domains to provide effective, comprehensive measures of academic constructs important to the success of all adult learners. Current assessments in adult education are not helpful for instructional needs such as placement, diagnosis, or formative assessment, nor are they valid and reliable measures for research in adult education. Moreover, existing assessments are not useful to employers, not customized to the learner, and not available outside adult education programs. This project will address these gaps by developing and validating ASAP, which will provide portable and personalized diagnostic literacy and numeracy assessments available to adult learners, teachers, career counselors, and employers to help improve instruction, job training, and successful academic and occupational outcomes for adult learners.

Project Activities: The researchers will conduct a comprehensive needs assessment across adult education, workforce investment, and other adult programs to identify the most critical skills and competencies learners, adult educators, counselors, and employers need to support adult learners. They will leverage this information and existing assessment items and tasks to the ASAP assessment system that will be personalized to the learner, delivered digitally, and provide diagnostic and actionable information for adult educators, learners, and employers. They will ensure the validity and fairness of the assessments using psychometric procedures.

Products: Products will include the online repository of assessment task modules; the supporting online testing system modularized for different assessment purposes that serve the learner, educator/counselor, or employer; and assessment resources such as online practice tests, test administration guides, and score report interpretation guides. The team also will disseminate research results through presentations at practitioner and research-oriented conferences and by publishing research findings in peer reviewed journals.

Structured Abstract

Setting: This research will take place primarily in Massachusetts and Nebraska but will also involve partners and adult education students across the U.S.

Sample: Approximately 12 experts will participate in the needs assessment. For the course of the development and validation work, approximately 20,000 adult learners from primarily adult education programs and also community college programs will participate. Approximately 100 educators and 60 other stakeholders will provide feedback on the ASAP system.

Innovation: ASAP will be a digital repository of assessment modules that users can electronically assemble and configure to serve specific assessment needs in adult education and workforce development. The assessment "unit" in the ASAP system is a "task module" that represents a cluster of literacy, numeracy, or integrative skills of interest to learners, educators, or employers. ASAP will leverage technology to capture evidence of student learning, using a universal design for learning architecture and combining features to make ASAP a personalized assessment system. ASAP will also leverage content and standards from adult education and competencies from the world of the workplace by linking assessment task to job skills in the O*NET system (which are linked to specific jobs and occupational clusters). ASAP assessment modules will be organized by NRS educational functioning levels and PIAAC achievement levels. ASAP will be available on virtually any digital device in any setting and delivered using relatively small assessment modules that can be taken in isolation or systematically pieced together to form larger assessments tailored to specific assessment goals. The system design will support all adult learners by providing additional opportunities to learn and acquire the skills needed for gainful employment and upskilling.

Research Design and Methods: For the needs assessment, the researchers will conduct a literature review in adult education assessment, survey key stakeholders (adult learners, teachers, career counselors, employers), and conduct focus groups. For the ASAP system, they will use principled design frameworks to ensure the validity, fairness, and reliability of the ASAP and address potential sources of construct-irrelevant variance that may emerge when assessments are administered to diverse adult learners. They will use socio-cognitive extended evidence-centered design to help ensure alignment between the assessment argument, measured skills, tasks designed, scoring, and reporting of evidence collected. In particular, they will be attending to possible bias for subgroup variation such as variation based on to sex, age, race/ethnicity/cultural heritage, native language, educational history, and learning disabilities. Over the course of the project, they will build four modes of ASAP: a manual mode for learner to choose a particular skill to work on, a placement mode and a diagnostic mode for educators or counselors, and a workplace mode for employers and similar stakeholders. After developing the full ASAP, they will conduct studies of its implementation to ensure it functions as intended and studies explore how ASAP links to national and international assessments in adult education to compare ASAP performance relative to other validated assessments in adult education.

Key Measures: Key measures include the Massachusetts Adult Proficiency Test for Mathematics and Numeracy, the Massachusetts Adult Proficiency Test for Reading, the Program for International Student Assessment, and other reading and numeracy adult education achievement test data (e.g., Accuplacer, WorkKeys).

Data Analytic Strategy: The researchers will use item response theory for test development; differential item functioning and other psychometric analyses to evaluate fairness, reliability, and validity; regression analyses to explore construct-relevant and irrelevant factors affecting learners' performance; exploratory analyses of log (response process) data to evaluate motivation and cognitive processes; equi-percentile linking and alignment studies to evaluate performance relative to PIAAC.