Inside IES Research

Notes from NCER & NCSER

Designing Culturally Responsive and Accessible Assessments for All Adult Learners

Dr. Meredith Larson, program officer for adult education at NCER, interviewed Dr. Javier Suárez-Álvarez, associate professor and associate director at the Center for Educational Assessment, University of Massachusetts Amherst. Dr. Suárez-Álvarez has served as the project director for the Adult Skills Assessment Project: Actionable Assessments for Adult Learners (ASAP) grant and was previously an education policy analyst in France for the Organisation for Economic Co-operation and Development (OECD), where he was the lead author of the PISA report 21st-Century Readers: Developing Literacy Skills in a Digital World. He and the ASAP team are working on an assessment system to meet the needs of adult education learners, educators, and employers that leverages online validated and culturally responsive banks of literacy and numeracy tasks. In this interview, Dr. Suárez-Álvarez discusses the importance of attending to learners’ goals and cultural diversity in assessment.

How would you describe the current context of assessment for adult education, and how does ASAP fit in it?

In general, the adult education field lacks assessments that meet the—sometimes competing—needs and goals of educators and employers and that attend to and embrace learner characteristics, goals, and cultural diversity. There is often a disconnect where different stakeholders want different things from the same assessments. Educators ask for curriculum-aligned assessments, learners want assessments to help them determine whether they have job-related skills for employment or promotion, and employers want to determine whether job candidates are trained in high-demand skills within their industries.

Despite these differing needs and interests, everyone involved needs assessment resources for lower skilled and culturally diverse learners that are easy to use, affordable or free, and provide actionable information for progress toward personal or occupational goals. ASAP is one of the first attempts to respond to these needs by developing an assessment system that delivers real-time customizable assessments to measure and improve literacy and numeracy skills. ASAP incorporates socioculturally responsive assessment principles to serve the needs of all learners by embracing the uniqueness of their characteristics. These principles involve ensuring that stakeholders from diverse socioeconomic, cultural, linguistic, racial, and ethnic groups are represented in our test design and development activities.

Why is attending to cultural diversity important to ASAP and assessment, and how are you incorporating this into your work?

U.S. Census projections for 2045 predict a shift in the demographic composition of the population from a White majority to a racially mixed majority. This suggests that we should prepare for cultural shifts and ensure our assessments fully embrace socioculturally responsive assessment practices. Without these practices, assessments limit the ability of adults from varied demographic backgrounds to demonstrate their capabilities adequately. Socioculturally responsive assessments are pivotal for representing the growing diversity in the learner population and for uncovering undetected workforce potential.

In ASAP, we are conducting focus groups, interviews, and listening sessions with learners, educators, and employers to understand their needs. We are also co-designing items in collaboration with key stakeholders and building consensus across adult education, workforce, and policy experts. We are developing use cases to understand hypothetical product users and conducting case studies to establish linkages between instruction and assessment as well as across classroom and workplace settings.

How has your background informed your interest in and contributions to ASAP?

As a teenager growing up in Spain, I saw first-hand the possible negative impact assessments could have when they don’t attend to learner goals and circumstances. When I was 15, my English teacher, based on narrow assessments, told my parents I was incapable of learning English, doubted my academic potential, and suggested I forego higher education for immediate employment. Defying this with the support of other teachers and my family, I pursued my passion. I became proficient in English at the age of 25 when I needed it to be a researcher, and I completed my PhD in psychology (psychometrics) at the age of 28.

Many adult students may have heard similar messages from prior teachers based on assessment results. And even now, many of the assessments the adult education field currently uses for these learners are designed by and for a population that no longer represents most learners. These adult learners may be getting advice or feedback that does not actually reflect their abilities or doesn’t provide useful guidance. Unfortunately, not all students are as lucky as I was. They may not have the support of others to counterbalance narrow assessments, and that shouldn’t be the expectation.

What are your hopes for the future of assessments for this adult population and the programs and employers that support them?

I hope we switch from measuring what we know generally how to measure (such as math and reading knowledge on a multiple-choice test) to measuring what matters to test takers and those using assessment results so that they can all accomplish goals in ways that honor individuals’ circumstances. Knowledge and skills—like the real world—are much more than right and wrong responses on a multiple-choice item. I also hope that as we embrace the latest developments in technology, such as AI, we can use them to deliver more flexible and personalized assessments.

In addition, I hope we stop assuming every learner has the same opportunities to learn or the same goals for their learning and that we start using assessments to empower learners rather than just as a measure of learning. In ASAP, for example, the adult learner will decide the type of test they want to take when to take it, the context within which the assessment will be framed, and when, where, and to whom the assessment result will be delivered.


This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), program officer for adult education at NCER.

 

IES Researchers on the Hill: A Briefing on Adult Education

On August 23, 2023, WestEd research and policy experts presented at a briefing focused on strengthening adult education and career pathways for Senate staff in Washington, DC. hosted by Senator Jack Reed (D-RI) and Senator Susan Collins (R-ME). The briefing drew from research and development WestEd is conducting for both IES (Adult Numeracy in the Digital Era: Adaptive Technology for Quantitative and Digital Literacy, ANDE) and the Office of Career, Technical, and Adult Education (Adult Numeracy Instruction 2.0), as well as WestEd’s experience building and managing statewide adult education and postsecondary longitudinal dashboards and leading technical assistance on building data-informed accelerated pathways to living wage careers for adult learners.

NCER program officer Meredith Larson asked WestEd’s Dr. Ann Edwards about this briefing and how she understands the role of researchers and research in communicating with policymakers. As the principal investigator on the ANDE grant, Dr. Edwards has been involved in recent work in adult education and provided some context for the briefing and the work that was discussed.

How did this briefing come about?

Senators Jack Reed (D-RI) and Todd Young (R-IN) recently signaled their commitment to strengthening adult education by introducing the Strengthening Research in Adult Education Act. To deepen their understanding about the current state of research in the field, the co-chairs of the Senate Adult Literacy Caucus, Senators Reed and Collins (R-ME), co-hosted this briefing and asked WestEd to share what we are learning with a bipartisan group of Senate staff.

What were some of the main points made at the briefing?

We covered many topics, but we presented our interpretation of existing evidence. Some of our comments included the following:

  • Much of the existing research on adult education is outdated or sparse. Existing research tends to focus on adult learning broadly or on adult learners in other contexts, with less attention paid to the unique educational needs and goals of learners in adult education programs. For example, some adult learners enroll in programs to pass a test for a job, continue their education, or improve their English language skills. These learner-centric goals aren’t always the focus of the research.
  • Moving forward, we can build more knowledge of adult education by investing in studies that are situated in the actual adult education settings that adult learners can, and do, access. For example, much of our knowledge about how adults might learn and appropriate instructional practices for them draws from K-12 and higher education systems rather than the adult education system.
  • When thinking about how to improve the adult education system, it’s important to understanding which strategies work for learners in adult education programs who face a wide range of life experiences as they balance jobs and family responsibilities. We need to consider the range of reasons they access programs and how these needs are changing. For example, in addition to foundational skills in numeracy and literacy (reading, writing, language), digital literacy is increasingly important for adult learners as they seek to achieve high school equivalency and look to engage in the workforce.
  • Investing in research could help us modify and strengthen programs. We have seen how effective something like a research network (for example, the IES-funded CREATE Adult Skills Network) can be for rapidly generating new insights that can strengthen adult learning.
  • We also identified a few specific areas where more research is necessary and could be key to adult education:
    • strengthening the adult educator workforce
    • identifying and applying literacy, numeracy, and English as a Second Language instructional practices developed specifically for adult learners
    • understanding promising models and instructional strategies that can be scaled across the adult education system
    • measuring short- and long-term outcomes of adult education programs
    • exploring the use of artificial intelligence to strengthen teaching and learning for adult learners
  • To this end, there is room for strengthening adult education data collections that are connected to federal reporting and accountability systems. The adult education system needs rigorous and reliable data and research to know accurately what works for adult learners and what improvements are needed.

Why do you feel this sort of communication is important?

We hope that briefings can help congressional staff better understand the complexities of the adult education system and the uniqueness of this population. Additionally, we wanted to emphasize for policymakers the limits of existing research on adult learners. Because federal reporting requirements often dictate which data are collected, we wanted to suggest ways the federal government could improve data collection, integration, and reporting. We also wanted to underscore and illustrate how research, policy, and practice are connected in efforts to enhance adult education outcomes. Sharing our research and insights with Senate staffers can help inform their decisionmaking by grounding policy in research, and we believe that can help to improve outcomes and increase opportunities for the economic mobility of this often overlooked population.

Do you have any plans for future conversations with policymakers?

Yes, we plan to stay connected to policymakers and look forward to future opportunities to contribute our insights on research and policy. As Congress takes on new opportunities such as the Strengthening Adult Education Research Act or the reauthorization of the Workforce Innovation and Opportunity Act (WIOA), we hope they will turn to researchers and practitioners to learn about what is working in the field and how research can inform progress.


This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education, NCER.

Adult Foundational Skills Research: Reflections on PIAAC and Data on U.S. Adult Skills

In this blog, NCER program officer, Dr. Meredith Larson, interviews Dr. Holly Xie from NCES about the Program for the International Assessment of Adult Competencies (PIAAC), an OECD-developed international survey  of adult skills in literacy, numeracy, and digital problem solving administered at least once a decade. PIAAC also collects information on adult activities (such as skill use at home or work, civic participation, etc.), demographics (such as level of education, race), and other factors (such as health outcomes). To date, NCER has funded three research grants (herehere, and here) and one training grant that relied on PIAAC data.

NCES has led the U.S. efforts in administering PIAAC and has been sharing results for over a decade. PIAAC in Cycle I (PIAAC I) included three waves of data collection in the United States with the first data released in 2013. From PIAAC I, we learned a wealth of information about the skills of U.S. adults. For example, the 2017 wave of data collection found that the percentages of U.S. adults performing at the lowest levels were 19 percent in literacy, 29 percent in numeracy, and 24 percent in digital problem solving. As we look forward to learning from PIAAC II, Dr. Xie reflects on the products from PIAAC I and possibilities for PIAAC II (release in 2024).

What is your role at NCES and with PIAAC specifically?

I am the PIAAC national program manager and oversee all aspects of the PIAAC study in the United States, including development and design, data collection, analysis and reporting, and dissemination/outreach. I also represent the United States at PIAAC international meetings.

What is something you’re particularly excited about having produced during PIAAC I?

I am most excited about the U.S. PIAAC Skills Map. The Skills Map provides information on adult skills at the state and county levels. Users can explore adult skills in literacy and numeracy in their state or county and get estimates of literacy or numeracy proficiency overall and by age and education levels. Or they can compare a county to a state, a state to the nation, or compare counties (or states) to each other. The map also has demographic and socioeconomic data from the American Community Survey (ACS) to provide context for the state or county estimates. This YouTube video demonstrates what the map can do.

 

 

We also have other PIAAC web products and publications such as national and international reports, Data Points, and PIAAC publications that provide invaluable information on U.S. adult skills and interrelationships of those skills to other social, economic, and demographic factors.

Do you have examples of how information from PIAAC I has been used?

PIAAC data cover results at the national, state, and county levels, and as such, they can be useful for policymakers or decision makers who would like to know where things stand in terms of the skills of their adult population and where they need to allocate resources at these different levels of the system. In other words, PIAAC data can be useful for drafting targeted policies and programs that will benefit their population and constituencies.

For instance, at the national level, former Vice President Biden used information from PIAAC I in his report Ready to Work for the June 2014 reauthorization of the Workforce Innovation and Opportunity Act, known as WIOA. PIAAC was also cited in the discussion of extending the Second Chance Pell experiment as identified in the 2019 report titled Prisoners’ Eligibility for Pell Grants: Issues for Congress.

The Digital Equity Act of 2021 also leveraged the PIAAC. This legislation identifies particular populations that determine the funding formula. The quick guide to these populations uses PIAAC to estimate one of these populations: Individuals with a language barrier, including individuals who are English learners and have low levels of literacy.

Local governments have also used PIAAC products. For example, the Houston Mayor’s Office for Adult Literacy in collaboration with the Barbara Bush Foundation used the PIAAC Skills Map data to inform the Adult Literacy Blueprint.

And the adult education advocacy group, ProLiteracy, also used the PIAAC and the Skills Map to develop a toolkit for local program adult education and adult literacy program advocacy.

When will the results of PIAAC II be available, and how does this cycle differ from PIAAC I?

PIAAC II data collection began in 2022 and results will be released in December 2024 and will include information on the literacy, numeracy, and adaptive problem-solving skills of adults in the United States. The numeracy assessment now includes a measure of “numeracy components,” which focus on number sense, smaller/bigger number values, measurement, etc. This information will help us learn more about the skills of adults who have very low numeracy skills. The adaptive problem-solving component is a new PIAAC module and will measure the ability to achieve one’s goals in a dynamic situation in which a method for reaching a solution is not directly available.

PIAAC II will also include, for the first time, questions about financial literacy in the background questionnaire, using items on managing money and tracking spending and income, savings methods, and budgeting. These additional questions will allow people to explore relationships between foundational skills, financial literacy, and other constructs in PIAAC.

What types of research could you imagine stemming from the PIAAC II?

One of the most unique features of PIAAC (both PIAAC I and II) is the direct assessment of literacy, numeracy, and problem-solving skills (information that no other large-scale assessment of adults provides). Thirty-one countries, including the United States, participated in PIAAC II (2022/23), so researchers will be able to compare the adult skills at the international level and also study trends between PIAAC I and PIAAC II.

It’s worth noting that the data collection took place while we were still experiencing the effects of the COVID-19 pandemic. This may provide researchers opportunities to explore how the pandemic is related to adults’ skills, health, employment, training, and education status.

Where can the public access data from PIAAC?

Researchers can find information about the available data from the national U.S. PIAAC 2017 Household, PIAAC 2012/14 Household, and PIAAC 2014 Prison datasets, and international and trend datasets on the NCES Data Files page. PIAAC restricted-use data files contain more detailed information, such as continuous age and earnings variables, that can be used for more in-depth analysis. Accessing the restricted-use data requires a restricted-use license from NCES.

NCES also has an easy-to-use online analysis tool: the International Data Explorer (IDE). The IDE allows users to work directly with the PIAAC data and produce their own analyses, tables, regressions, and charts. An IDE tutorial video provides comprehensive, step-by-step instructions on how to use this tool. It contains detailed information about the content and capabilities of the PIAAC IDE, as well as how the PIAAC data are organized in the tool.


This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education, NCER.

Adult Ed Grantee Spotlight: Aydin Durgunoglu and Research for Adult English Learners

As part of the IES 20th Anniversary, NCER is reflecting on the past, present, and future of adult education research. In this blog, Dr. Aydin Durgunoglu, Distinguished Global Professor Emeritus at the University of  Minnesota-Duluth, reflects on how her life and training have influenced her work. Dr. Durgunoglu is the principal investigator on Content-Integrated Language Instruction for Adults with Technology Support, one of the six research projects that comprise the CREATE Adult Skills Research Network.  As part of this network, Dr. Durgunoglu and her team are focusing on the needs of adult English learners and on U.S. history and civics education, such as what might be taught as part of Integrated English Literacy and Civics Education programs. Hers is the first grant NCER has funded that is focusing on this area for this population.

Please describe your IES project.

My colleagues and I are developing a curriculum called CILIA-T (Content-Integrated Language Instruction for Adults with Technology Support). We are embedding English instruction into U.S. History and Civics content and providing technology supports for both students and teachers as part of the curriculum. Our goal is to provide a complete and integrated resource that can be used by teachers with varying levels of experience in English as a Second Language (ESL), civics/U.S. history and citizenship classes.

What motivates you to do this work?

Two of my motivations are my background as an English learner and immigrant and my training as a cognitive psychologist.

I started learning English when I was 12 in an immersion-based approach. I recall some of the struggles I had such as misunderstanding that “you may sit down” was a full sentence in English because, in Turkish, the single word oturun has a similar meaning to the English sentence. This realization along with both Turkish and American experiences helped me to see the importance of culture, language, and instruction.

As a cognitive psychologist by training, I am interested in learning, memory, knowledge acquisition, and—most of all—language. One of my research areas has been how literacy develops across different languages—how it may progress differently in Spanish, Turkish, English, Hmong, etc. and whether it involves general cognitive processes that are language independent.

These experiences and interests have long influenced my work. For example, my colleagues and I collaborated on literacy projects for Mother Child Education Foundation (MOCEF) based in Turkey that have evolved and currently include a focus on women’s citizenship and empowerment. This work was based on my theoretical work on literacy development in Turkish. In the United States, I have conducted studies with adults and children on how what they know in their home languages can help them learn English (cross-language transfer). All of these experiences led me to our work as the CILIA-T team.

How are you leveraging your experiences to build CILIA-T?

In addition to my theoretical and applied experiences in adult education, this project is benefiting from the contributions of a group of dedicated adult educators. These colleagues are teaching ESL, citizenship, history, and civics classes. We are collaborating on writing a curriculum that teachers like themselves would like to use. Based on our experiences and findings from the field, we have identified the components that we feel are key for CILIA-T. Three of these main components include

  1. Multimodal input:  Contrary to how I started learning English, providing linguistic input in several different modalities is helpful. Technology provides many opportunities to realize this goal. Learners can interact with and produce content in many forms. For example, they can create and share academic vocabulary sets and review them like a game. Technology can also facilitate deep conceptual understanding of academic topics. For example, learners can share and discuss not only texts but also audios and videos for a deeper analysis and application of civics and history topics.
  2. Build on first languages (L1): Adults already have a well-developed language system or systems, if they know multiple languages. They use the clues from their L1 to understand how English operates. Therefore, we can provide opportunities to bring that existing linguistic knowledge to the forefront and to compare and contrast explicitly. One clear way to leverage L1 is to integrate oral language and help bridge what the adults can do orally with what they aim to do in reading and writing.
  3. Academic vocabulary: Individuals with limited or interrupted schooling tend to have lower levels of academic vocabulary in their first language, and thus, likely lower levels in English. A language learner may be quite fluent in using English in their everyday interactions, but that does not mean they have a strong academic vocabulary across different domains, such as health, math, science, civics, and finance. CILIA-T covers academic and discipline-specific vocabulary in a purposeful way. Academic vocabulary is closely related to the conceptual understanding of a phenomenon. Therefore, just learning word definitions is not enough. The vocabulary has to be contextualized with a conceptual understanding. For example, executive branch does not mean much by itself unless it is situated within an understanding of an overall governmental system. Similarly, the definition of the word mortgage may be forgotten quickly if the learner is not familiar with the loan and repayment system in the United States. Luckily, adults have a lot of background knowledge to facilitate such conceptual scaffolding, but that is for another blog.

What value do you hope CILIA-T might bring to the students, teachers, and communities?

We believe that when all individuals, but especially the newcomers, understand the systems, practices, historical contexts, and the language(s) of their society, they can become more active participants in their communities and can work towards accomplishing their life goals more effectively. We hope that CILIA-T provides the adult learners and the educators and programs that support them with a tool to facilitate this growth.


This blog was produced by Dr. Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer at NCER.

Research to Inform Stronger Adult Education ESL Policy and Practice

April is National Bilingual/Multilingual Learner Advocacy Month! As part of the IES 20th Anniversary celebration, we are highlighting NCER’s investments in field-initiated research. In this guest blog, Drs. Nikki Edgecombe (Community College Research Center at Teachers College, Columbia University) and George Bunch (University of California, Santa Cruz) discuss their IES-funded study focused on identifying the policies and practices that support multilingual learners (MLs) in community colleges, the important role of adult education English as a Second Language (adult ed ESL), and some of the lessons learned. MLs in community colleges, a significantly understudied population, include students who were classified as English learners in K-12 but also more recent arrivals to the US with a wide variety of education backgrounds and adult immigrants and refugees who have lived in the US for a number of years.  

Vital Role of Adult Education ESL

The demands on and opportunities for adult education programs, specifically adult ed ESL, are growing. The programs are affordable and accessible to immigrant communities and people with lower levels of academic preparation—many of whom were hit hard by the pandemic and disproportionately experienced negative education, health, and economic consequences. Simultaneously, the pandemic highlighted the increasing, critical need for multilingual workers across fields, including education, healthcare, and government. Adult ed ESL programs can help bring those workers into the labor market.

According to the National Reporting System for Adult Education, in program year 2021-22, nationally, free or low-cost adult education programs enrolled about 900,000 students, with nearly 50 percent in English language acquisition or integrated English literacy and civics education programs. Sixteen states house adult education programs in community and technical colleges, often in addition to other community-based and educational settings (see this site for information about state grants). These programs provide a unique opportunity for students to improve “everyday” English skills and serve as a ready-made pathway to a postsecondary credential. Yet, the programs and their students face a number of challenges. 

Asking the Questions Practitioners Want Answers To

For the last 4 years, we examined policies and practices that affect the experiences and outcomes of MLs in a large midwestern community college district. The research has focused on adult ed ESL, in part at the recommendation of our district partners. The district serves thousands of ESL students annually, and institutional leaders have actively pursued improvements to program access, instruction, and progression. As we learned more about the improvement efforts underway, we were able to gain a clearer picture of the stringent federal and state policies adult education operates under, which have at times challenged district leaders' ability to make the kinds of changes necessary to enhance student outcomes. Our institutional partners are not deterred, however, and continue to seek ways to strengthen their adult education program, make it more student-centered, and make policy more effective. They consider research an important resource in this improvement process. As such, we both documented their efforts and examined how the policy context affected what they were doing.

What We Are Finding

We wanted to better understand who the MLs are, their life circumstances, their college experiences, and their goals. We reported the following trends to our community college partners.  

  • Adult ed ESL students are older and less likely to be working than their peers in credit programs. As a group, they are more likely to either have not earned a high school diploma or GED or have previously earned a baccalaureate degree or higher. Adult ed ESL students in our survey sample report enrolling in adult ed ESL to improve their everyday English literacy skills, to strengthen their employment prospects and prepare themselves for the language and literacy demands of further postsecondary education.
  • The multiple ESL levels required by policy may generate obstacles to progression, particularly for students who initially place in the lower levels of the sequence. MLs in our partner district place into 1 of 6 adult ed ESL courses. As research has previously established, community college students rarely persist through long sequences of courses, and our preliminary administrative data analysis shows students in our sample generally persist for less than 2 semesters. In response, our district partner developed a full-time position to help students in the transitions into, through, and out of adult education; has offered short (4- and 8-week) courses; built out dedicated academic and nonacademic supports; and created an intentional on-ramp to credit programs. Nonetheless, the length of the sequence appears to undermine retention and progression.
  • The prescribed assessment and placement procedures make it difficult for community college-based adult ed ESL programs to meet the varied English language learning needs of enrollees. Language learning is a complex phenomenon that requires students to develop a range of productive (speaking and writing) and receptive (listening and reading) literacy skills for a wide range of academic, professional, and community participation goals. That learning can look quite different for different students in different education environments and proficiency measurement is equally complex. Our district partner used one of the federally mandated assessments to both place students and measure their proficiency gains. The test is relatively inexpensive and easy to administer, but it only measures reading, just one aspect of language proficiency, leaving no consistent record of proficiency levels in speaking, listening, and writing.

New Directions in Adult Ed ESL Policy Research

Early findings from the study have the potential to inform changes in policy and practice at our district partner. Our findings also raise issues that federal and state policymakers and practitioners working on the ground may need to work together to answer, including how policy systems can balance the perceived need for standards and accountability with community colleges’ need to structure and administer the programs in ways that best meet the needs of MLs.

To pursue this line of inquiry, we will explore the origin and rationale for adult ed ESL policy and how that policy translates from federal to state to institutional providers. We will also learn more about the goals and experiences of adult ed ESL students coming out of the pandemic, explore the perceptions and experiences of adult educators and program staff, and provide formative feedback on the reform efforts underway at our district partner with a particular focus on whether and how policy is helping or hindering their ability to meet their goals.


This blog was produced by Dr. Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer at NCER.