Project Activities
The researchers will conduct a series of studies that include using extant process data from the Program for the International Assessment of Adult Competencies (PIAAC) and primary data. Using the PIAAC data, the team will examine how adults with different skill levels behaviorally interact (such as patterns of engagement, efficiency, test-taking strategies) with different types of adaptive problem-solving items. Using both PIAAC and think-aloud data, they will examine cognitive and metacognitive strategies that correspond to the observed process data behaviors as adults with different levels of skill engage with PIAAC adaptive problem-solving items. The researchers will also explore the extent to which the process and think-aloud data are related to adults' foundational literacy skills and background characteristics (e.g., demographics), and they will use a survey and focus groups to understand technology access and use in adult literacy classrooms.
Structured Abstract
Setting
The research team will use a nationally representative U.S. sample of adults from the extant 2022 Cycle 2 PIAAC process data. They will also collect data from adults in Georgia.
Sample
The U.S. 2022 PIAAC process data (released approximately in January 2025) include a nationally representative sample of adults aged 16 to 65 (across three cognitive domains - literacy, numeracy, adaptive problem-solving [APS]). The research team will use the APS domain process files to examine adults with low skills (Level 1 and below) and higher skills (Levels 2 and 3). They will collect data from learners in adult literacy programs (N = 120) who read between 6th and 10th grade levels, college students (120), and adult literacy teachers (N = 7).
The researchers will explore how low-skilled adults engage with adaptive problem-solving items relative to higher-skilled adults. Some of the factors that may affect performance include background characteristics of the learners, such as information and communications technology (ICT) skills (basic computer skills, using web features, clicking functions, mouse dragging, etc.) and foundational reading skills. Another set of relevant factors include features of the assessment modality and item characteristics. The researchers will also consider factors relevant to the frameworks that informed the development of PIAAC items: the PIAAC APS framework and the RESOLV model. The PIAAC framework assumes that the contexts and nature of the task are important aspects of adaptive problem solving, and the PIAAC items reflect this. The RESOLV model also assumes that general problem solving requires cognitive and metacognitive processes (e.g., goal setting, progress monitoring, organizing), which are important to all problem-solving tasks involving texts, and that these processes will vary based on the nature of the task.
Research design and methods
This project includes multiple studies: one set that focuses primarily on the process data and another set that focuses more fully on the process data. In the first study using the PIAAC adaptive problem-solving (APS) process data, the researchers will examine whether task-specific APS dimensions (information environment, problem context, digital tools) create challenges for adults with lower skills compared to those with higher skills. In the APS study 2, they will explore test-takers' distinct behavior patterns (engagement, efficiency, response patterns, strategies) on APS items of successful as compared to unsuccessful respondents. In the APS study 3, they will assess whether distinct behavior patterns (from study 2) relate to literacy performance and respondent characteristics (demographics, information and communications technology [ICT] skill use, problem-solving skills use). In the first think-aloud (TA) study, the researchers will pilot a think-aloud protocol to identify specific cognitive, metacognitive, and problem-solving strategies used by adults with differing levels of skill on linear texts and how these relate to comprehension performance. In TA study 2, they will develop a think-aloud protocol specific to adaptive problem-solving and pilot this with adults of varying skill levels. In TA study 3, they will conduct a virtual survey and focus group with adult literacy instructors on access and use of technology. In TA study 4, they will refine their think-aloud protocol and collect think-aloud and process data from adults with differing skill levels and explore whether think-aloud strategies, process data behaviors, individual differences measures, and background characteristics are predictive of item accuracy for adults with lower or higher skills.
Control condition
Due to nature of this study, there is no control condition. However, the researchers will compare adults based on their skill levels (higher and lower).
Key measures
The primary measure from the PIAAC are process data from the APS domain, which assesses adults' adaptive problem-solving skills in a digital context as a single score (0–500 scale score) and includes interactive and dynamic items that cross contexts. PIAAC typically breaks the scale into three proficiency levels: Level 1, Level 2, and Level 3. The researchers will also include total scores on the literacy and APS domains and respondent background characteristics from the main 2022 U.S. PIAAC dataset. They will develop and collect data from think-aloud protocols and will also collect a comprehensive demographic survey and a battery of reading component skills from their primary-data sample.
Data analytic strategy
Depending on the particular study, the researchers will use one or more analytic strategy. The possible strategies include descriptive analyses, cognitive diagnostic modeling, sequence mining techniques, latent state comparison analyses, supervised machine learning methods, LASSO, regression, cluster analyses, and multi-level modeling.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The researchers will refine theoretical models on the challenges faced by adults with low foundational skills in the context of adaptive problem-solving skills, and they will conduct workshops for researchers interested in analyzing process data from large-scale digital and will produce peer-reviewed publications and other materials for both research and non-research audiences.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigators: He, Qiwei; Magliano, Joseph
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
