Inside IES Research

In recognition of National Bilingual/Multilingual Learner Advocacy Month, we want to highlight an IES-funded research project, Content-Integrated Language Instruction for Adults with Technology Support (CILIA-T). This work focuses on the needs of adult English learners (ELs) to help them build not only language proficiency but also knowledge of U.S. history and civics, akin to what may occur in part of adult education Integrated English Literacy and Civics Education programs while also enhancing digital literacy skills. In this interview blog, Theresa Sladek (Northstar, co-PI of CILIA-T), Aydin Durgunoglu (University of Minnesota Duluth, PI of CILIA-T), and Leah Hauge (Northstar), describe a creative approach to building adult digital literacy skills as a part of their curriculum for adult English learners. To help ensure the learners will benefit from technology-supported instruction, the team has created a series of videos and an instructional guide that are all publicly available and free.

What are the videos about, and why did you make them?

In addition to building  knowledge of English, civics, and U.S. history, we are also building digital proficiencies through CILIA-T. For example, we want to help adult learners become efficient in using digital tools such as Gmail and Zoom as part of their coursework and build digital proficiencies such as digital safety, finding reliable and relevant information online, knowing how to solve technical problems in different contexts and with different tools, and accessing, comprehending and integrating information across multiple modalities and resources.  

To help learners build these skills, we created six short, free videos in partnership with Northstar Digital Literacy, allowing any individual, anywhere in the world, who has a device and internet connection to access them. By partnering with Northstar, we are increasing the reach of this resource, as the Northstar website saw over 800,000 hits in the last year alone, and its YouTube channel has over 2.18k subscribers. Since being posted about three months ago, the videos have already had 900 views. These free tools are a step towards removing the digital divide through clear and concise entry-level video tutorials.

The tutorials cover six topics: Finding Information Online, Zoom, Smartphone Apps, WhatsApp, Quizlet, and Gmail. The videos help learners build entry-level skills in each topic and can also be used to onboard instructors (teachers or tutors) to each topic.

In addition, we also created an instructional guide for teachers or tutors, which is available in the Educator Resources section of the Literacy Minnesota website. The instructor guide that accompanies the video tutorials supports every step of the teaching process, including instructions for teachers or tutors to guide learners before, during, and after viewing the video content. Educator Resources has an extensive national and international reach, getting over 20,000 visits a year. Through our work in the adult foundational skills system (also called adult education), we know that many programs have limited budgets, and free curricula and teaching aids are a key to those programs being able to provide much-needed services. Although we made these materials with ELs and CILIA-T curriculum in mind, these materials can be used in a wide variety of contexts, with both ELs and native  speakers and in settings outside of education.

Why are the videos important for your project?

Because not all learners, teachers, and volunteers are familiar with these tools, we did not want this to be a barrier to accessing the CILIA-T curriculum. Addressing this barrier is an intentional act to bring equity and access to adult learners and those assisting them.

As part of the CILIA-T project, these videos are critical not only because they teach digital skills that can be used in a wide variety of settings and purposes but also because they assist students in learning the civics and U.S. history content in the curriculum and English language through a variety of modalities. Learners will be using these tools to read texts, answer questions, use multimodal resources, make presentations, create and share content, perform self-checks, comment on others’ work, engage in discussions and group projects, and complete internet searches and evaluations.  

What did you learn about making videos and disseminating them as part of this process?

We learned a few things along the way:

• Consider different operating systems. Because phones are the main device to perform digital tasks for most adult learners, we incorporated variations relevant to different operating systems (for example, iOS and Android) into digital skill teaching tools. For example, the process for downloading apps varies a bit between iOS and Android, so we have modeled both with our videos showing students when processes may vary between devices. 
• Always consider viewers’ English language level. For example, CILIA-T curriculum was written to support intermediate-level English language learners. So, we had to design the instructional script with digital skill vocabulary for such speakers in mind and use many examples and visuals to define vocabulary in the videos.
• Build in opportunities for learners and educators to pause, practice, and reflect. To do this, we intentionally divided videos into sections to make finding specific pieces of learning easier. We also added titled chapters to each portion of a video. 

We are excited about this new resource and invite teachers and learners to try them out and reach out to us with any feedback and questions: Theresa Sladek (tsladek@literacymn.org), Aydin Durgunoglu (adurguno@d.umn.edu), and Leah Hauge (lhauge@literacymn.org).

---

This blog was produced by Dr. Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education research at NCER.

 

Dr. Meredith Larson, program officer for adult education at NCER, interviewed Dr. Javier Suárez-Álvarez, associate professor and associate director at the Center for Educational Assessment, University of Massachusetts Amherst. Dr. Suárez-Álvarez has served as the project director for the Adult Skills Assessment Project: Actionable Assessments for Adult Learners (ASAP) grant and was previously an education policy analyst in France for the Organisation for Economic Co-operation and Development (OECD), where he was the lead author of the PISA report 21st-Century Readers: Developing Literacy Skills in a Digital World. He and the ASAP team are working on an assessment system to meet the needs of adult education learners, educators, and employers that leverages online validated and culturally responsive banks of literacy and numeracy tasks. In this interview, Dr. Suárez-Álvarez discusses the importance of attending to learners’ goals and cultural diversity in assessment.

How would you describe the current context of assessment for adult education, and how does ASAP fit in it?

In general, the adult education field lacks assessments that meet the—sometimes competing—needs and goals of educators and employers and that attend to and embrace learner characteristics, goals, and cultural diversity. There is often a disconnect where different stakeholders want different things from the same assessments. Educators ask for curriculum-aligned assessments, learners want assessments to help them determine whether they have job-related skills for employment or promotion, and employers want to determine whether job candidates are trained in high-demand skills within their industries.

Despite these differing needs and interests, everyone involved needs assessment resources for lower skilled and culturally diverse learners that are easy to use, affordable or free, and provide actionable information for progress toward personal or occupational goals. ASAP is one of the first attempts to respond to these needs by developing an assessment system that delivers real-time customizable assessments to measure and improve literacy and numeracy skills. ASAP incorporates socioculturally responsive assessment principles to serve the needs of all learners by embracing the uniqueness of their characteristics. These principles involve ensuring that stakeholders from diverse socioeconomic, cultural, linguistic, racial, and ethnic groups are represented in our test design and development activities.

Why is attending to cultural diversity important to ASAP and assessment, and how are you incorporating this into your work?

U.S. Census projections for 2045 predict a shift in the demographic composition of the population from a White majority to a racially mixed majority. This suggests that we should prepare for cultural shifts and ensure our assessments fully embrace socioculturally responsive assessment practices. Without these practices, assessments limit the ability of adults from varied demographic backgrounds to demonstrate their capabilities adequately. Socioculturally responsive assessments are pivotal for representing the growing diversity in the learner population and for uncovering undetected workforce potential.

In ASAP, we are conducting focus groups, interviews, and listening sessions with learners, educators, and employers to understand their needs. We are also co-designing items in collaboration with key stakeholders and building consensus across adult education, workforce, and policy experts. We are developing use cases to understand hypothetical product users and conducting case studies to establish linkages between instruction and assessment as well as across classroom and workplace settings.

How has your background informed your interest in and contributions to ASAP?

As a teenager growing up in Spain, I saw first-hand the possible negative impact assessments could have when they don’t attend to learner goals and circumstances. When I was 15, my English teacher, based on narrow assessments, told my parents I was incapable of learning English, doubted my academic potential, and suggested I forego higher education for immediate employment. Defying this with the support of other teachers and my family, I pursued my passion. I became proficient in English at the age of 25 when I needed it to be a researcher, and I completed my PhD in psychology (psychometrics) at the age of 28.

Many adult students may have heard similar messages from prior teachers based on assessment results. And even now, many of the assessments the adult education field currently uses for these learners are designed by and for a population that no longer represents most learners. These adult learners may be getting advice or feedback that does not actually reflect their abilities or doesn’t provide useful guidance. Unfortunately, not all students are as lucky as I was. They may not have the support of others to counterbalance narrow assessments, and that shouldn’t be the expectation.

What are your hopes for the future of assessments for this adult population and the programs and employers that support them?

I hope we switch from measuring what we know generally how to measure (such as math and reading knowledge on a multiple-choice test) to measuring what matters to test takers and those using assessment results so that they can all accomplish goals in ways that honor individuals’ circumstances. Knowledge and skills—like the real world—are much more than right and wrong responses on a multiple-choice item. I also hope that as we embrace the latest developments in technology, such as AI, we can use them to deliver more flexible and personalized assessments.

In addition, I hope we stop assuming every learner has the same opportunities to learn or the same goals for their learning and that we start using assessments to empower learners rather than just as a measure of learning. In ASAP, for example, the adult learner will decide the type of test they want to take when to take it, the context within which the assessment will be framed, and when, where, and to whom the assessment result will be delivered.

This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), program officer for adult education at NCER.

On August 23, 2023, WestEd research and policy experts presented at a briefing focused on strengthening adult education and career pathways for Senate staff in Washington, DC. hosted by Senator Jack Reed (D-RI) and Senator Susan Collins (R-ME). The briefing drew from research and development WestEd is conducting for both IES (Adult Numeracy in the Digital Era: Adaptive Technology for Quantitative and Digital Literacy, ANDE) and the Office of Career, Technical, and Adult Education (Adult Numeracy Instruction 2.0), as well as WestEd’s experience building and managing statewide adult education and postsecondary longitudinal dashboards and leading technical assistance on building data-informed accelerated pathways to living wage careers for adult learners.

NCER program officer Meredith Larson asked WestEd’s Dr. Ann Edwards about this briefing and how she understands the role of researchers and research in communicating with policymakers. As the principal investigator on the ANDE grant, Dr. Edwards has been involved in recent work in adult education and provided some context for the briefing and the work that was discussed.

How did this briefing come about?

Senators Jack Reed (D-RI) and Todd Young (R-IN) recently signaled their commitment to strengthening adult education by introducing the Strengthening Research in Adult Education Act. To deepen their understanding about the current state of research in the field, the co-chairs of the Senate Adult Literacy Caucus, Senators Reed and Collins (R-ME), co-hosted this briefing and asked WestEd to share what we are learning with a bipartisan group of Senate staff.

What were some of the main points made at the briefing?

We covered many topics, but we presented our interpretation of existing evidence. Some of our comments included the following:

• Much of the existing research on adult education is outdated or sparse. Existing research tends to focus on adult learning broadly or on adult learners in other contexts, with less attention paid to the unique educational needs and goals of learners in adult education programs. For example, some adult learners enroll in programs to pass a test for a job, continue their education, or improve their English language skills. These learner-centric goals aren’t always the focus of the research.
• Moving forward, we can build more knowledge of adult education by investing in studies that are situated in the actual adult education settings that adult learners can, and do, access. For example, much of our knowledge about how adults might learn and appropriate instructional practices for them draws from K-12 and higher education systems rather than the adult education system.
• When thinking about how to improve the adult education system, it’s important to understanding which strategies work for learners in adult education programs who face a wide range of life experiences as they balance jobs and family responsibilities. We need to consider the range of reasons they access programs and how these needs are changing. For example, in addition to foundational skills in numeracy and literacy (reading, writing, language), digital literacy is increasingly important for adult learners as they seek to achieve high school equivalency and look to engage in the workforce.
• Investing in research could help us modify and strengthen programs. We have seen how effective something like a research network (for example, the IES-funded CREATE Adult Skills Network) can be for rapidly generating new insights that can strengthen adult learning.
• We also identified a few specific areas where more research is necessary and could be key to adult education:
  • strengthening the adult educator workforce
  • identifying and applying literacy, numeracy, and English as a Second Language instructional practices developed specifically for adult learners
  • understanding promising models and instructional strategies that can be scaled across the adult education system
  • measuring short- and long-term outcomes of adult education programs
  • exploring the use of artificial intelligence to strengthen teaching and learning for adult learners

• To this end, there is room for strengthening adult education data collections that are connected to federal reporting and accountability systems. The adult education system needs rigorous and reliable data and research to know accurately what works for adult learners and what improvements are needed.

Why do you feel this sort of communication is important?

We hope that briefings can help congressional staff better understand the complexities of the adult education system and the uniqueness of this population. Additionally, we wanted to emphasize for policymakers the limits of existing research on adult learners. Because federal reporting requirements often dictate which data are collected, we wanted to suggest ways the federal government could improve data collection, integration, and reporting. We also wanted to underscore and illustrate how research, policy, and practice are connected in efforts to enhance adult education outcomes. Sharing our research and insights with Senate staffers can help inform their decisionmaking by grounding policy in research, and we believe that can help to improve outcomes and increase opportunities for the economic mobility of this often overlooked population.

Do you have any plans for future conversations with policymakers?

Yes, we plan to stay connected to policymakers and look forward to future opportunities to contribute our insights on research and policy. As Congress takes on new opportunities such as the Strengthening Adult Education Research Act or the reauthorization of the Workforce Innovation and Opportunity Act (WIOA), we hope they will turn to researchers and practitioners to learn about what is working in the field and how research can inform progress.

This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education, NCER.

In this blog, NCER program officer, Dr. Meredith Larson, interviews Dr. Holly Xie from NCES about the Program for the International Assessment of Adult Competencies (PIAAC), an OECD-developed international survey  of adult skills in literacy, numeracy, and digital problem solving administered at least once a decade. PIAAC also collects information on adult activities (such as skill use at home or work, civic participation, etc.), demographics (such as level of education, race), and other factors (such as health outcomes). To date, NCER has funded three research grants (here, here, and here) and one training grant that relied on PIAAC data.

NCES has led the U.S. efforts in administering PIAAC and has been sharing results for over a decade. PIAAC in Cycle I (PIAAC I) included three waves of data collection in the United States with the first data released in 2013. From PIAAC I, we learned a wealth of information about the skills of U.S. adults. For example, the 2017 wave of data collection found that the percentages of U.S. adults performing at the lowest levels were 19 percent in literacy, 29 percent in numeracy, and 24 percent in digital problem solving. As we look forward to learning from PIAAC II, Dr. Xie reflects on the products from PIAAC I and possibilities for PIAAC II (release in 2024).

What is your role at NCES and with PIAAC specifically?

I am the PIAAC national program manager and oversee all aspects of the PIAAC study in the United States, including development and design, data collection, analysis and reporting, and dissemination/outreach. I also represent the United States at PIAAC international meetings.

What is something you’re particularly excited about having produced during PIAAC I?

I am most excited about the U.S. PIAAC Skills Map. The Skills Map provides information on adult skills at the state and county levels. Users can explore adult skills in literacy and numeracy in their state or county and get estimates of literacy or numeracy proficiency overall and by age and education levels. Or they can compare a county to a state, a state to the nation, or compare counties (or states) to each other. The map also has demographic and socioeconomic data from the American Community Survey (ACS) to provide context for the state or county estimates. This YouTube video demonstrates what the map can do.

We also have other PIAAC web products and publications such as national and international reports, Data Points, and PIAAC publications that provide invaluable information on U.S. adult skills and interrelationships of those skills to other social, economic, and demographic factors.

Do you have examples of how information from PIAAC I has been used?

PIAAC data cover results at the national, state, and county levels, and as such, they can be useful for policymakers or decision makers who would like to know where things stand in terms of the skills of their adult population and where they need to allocate resources at these different levels of the system. In other words, PIAAC data can be useful for drafting targeted policies and programs that will benefit their population and constituencies.

For instance, at the national level, former Vice President Biden used information from PIAAC I in his report Ready to Work for the June 2014 reauthorization of the Workforce Innovation and Opportunity Act, known as WIOA. PIAAC was also cited in the discussion of extending the Second Chance Pell experiment as identified in the 2019 report titled Prisoners’ Eligibility for Pell Grants: Issues for Congress.

The Digital Equity Act of 2021 also leveraged the PIAAC. This legislation identifies particular populations that determine the funding formula. The quick guide to these populations uses PIAAC to estimate one of these populations: Individuals with a language barrier, including individuals who are English learners and have low levels of literacy.

Local governments have also used PIAAC products. For example, the Houston Mayor’s Office for Adult Literacy in collaboration with the Barbara Bush Foundation used the PIAAC Skills Map data to inform the Adult Literacy Blueprint.

And the adult education advocacy group, ProLiteracy, also used the PIAAC and the Skills Map to develop a toolkit for local program adult education and adult literacy program advocacy.

When will the results of PIAAC II be available, and how does this cycle differ from PIAAC I?

PIAAC II data collection began in 2022 and results will be released in December 2024 and will include information on the literacy, numeracy, and adaptive problem-solving skills of adults in the United States. The numeracy assessment now includes a measure of “numeracy components,” which focus on number sense, smaller/bigger number values, measurement, etc. This information will help us learn more about the skills of adults who have very low numeracy skills. The adaptive problem-solving component is a new PIAAC module and will measure the ability to achieve one’s goals in a dynamic situation in which a method for reaching a solution is not directly available.

PIAAC II will also include, for the first time, questions about financial literacy in the background questionnaire, using items on managing money and tracking spending and income, savings methods, and budgeting. These additional questions will allow people to explore relationships between foundational skills, financial literacy, and other constructs in PIAAC.

What types of research could you imagine stemming from the PIAAC II?

One of the most unique features of PIAAC (both PIAAC I and II) is the direct assessment of literacy, numeracy, and problem-solving skills (information that no other large-scale assessment of adults provides). Thirty-one countries, including the United States, participated in PIAAC II (2022/23), so researchers will be able to compare the adult skills at the international level and also study trends between PIAAC I and PIAAC II.

It’s worth noting that the data collection took place while we were still experiencing the effects of the COVID-19 pandemic. This may provide researchers opportunities to explore how the pandemic is related to adults’ skills, health, employment, training, and education status.

Where can the public access data from PIAAC?

Researchers can find information about the available data from the national U.S. PIAAC 2017 Household, PIAAC 2012/14 Household, and PIAAC 2014 Prison datasets, and international and trend datasets on the NCES Data Files page. PIAAC restricted-use data files contain more detailed information, such as continuous age and earnings variables, that can be used for more in-depth analysis. Accessing the restricted-use data requires a restricted-use license from NCES.

NCES also has an easy-to-use online analysis tool: the International Data Explorer (IDE). The IDE allows users to work directly with the PIAAC data and produce their own analyses, tables, regressions, and charts. An IDE tutorial video provides comprehensive, step-by-step instructions on how to use this tool. It contains detailed information about the content and capabilities of the PIAAC IDE, as well as how the PIAAC data are organized in the tool.

This blog was produced by Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer for postsecondary and adult education, NCER.

As part of the IES 20^th Anniversary, NCER is reflecting on the past, present, and future of adult education research. In this blog, Dr. Aydin Durgunoglu, Distinguished Global Professor Emeritus at the University of  Minnesota-Duluth, reflects on how her life and training have influenced her work. Dr. Durgunoglu is the principal investigator on Content-Integrated Language Instruction for Adults with Technology Support, one of the six research projects that comprise the CREATE Adult Skills Research Network.  As part of this network, Dr. Durgunoglu and her team are focusing on the needs of adult English learners and on U.S. history and civics education, such as what might be taught as part of Integrated English Literacy and Civics Education programs. Hers is the first grant NCER has funded that is focusing on this area for this population.

Please describe your IES project.

My colleagues and I are developing a curriculum called CILIA-T (Content-Integrated Language Instruction for Adults with Technology Support). We are embedding English instruction into U.S. History and Civics content and providing technology supports for both students and teachers as part of the curriculum. Our goal is to provide a complete and integrated resource that can be used by teachers with varying levels of experience in English as a Second Language (ESL), civics/U.S. history and citizenship classes.

What motivates you to do this work?

Two of my motivations are my background as an English learner and immigrant and my training as a cognitive psychologist.

I started learning English when I was 12 in an immersion-based approach. I recall some of the struggles I had such as misunderstanding that “you may sit down” was a full sentence in English because, in Turkish, the single word oturun has a similar meaning to the English sentence. This realization along with both Turkish and American experiences helped me to see the importance of culture, language, and instruction.

As a cognitive psychologist by training, I am interested in learning, memory, knowledge acquisition, and—most of all—language. One of my research areas has been how literacy develops across different languages—how it may progress differently in Spanish, Turkish, English, Hmong, etc. and whether it involves general cognitive processes that are language independent.

These experiences and interests have long influenced my work. For example, my colleagues and I collaborated on literacy projects for Mother Child Education Foundation (MOCEF) based in Turkey that have evolved and currently include a focus on women’s citizenship and empowerment. This work was based on my theoretical work on literacy development in Turkish. In the United States, I have conducted studies with adults and children on how what they know in their home languages can help them learn English (cross-language transfer). All of these experiences led me to our work as the CILIA-T team.

How are you leveraging your experiences to build CILIA-T?

In addition to my theoretical and applied experiences in adult education, this project is benefiting from the contributions of a group of dedicated adult educators. These colleagues are teaching ESL, citizenship, history, and civics classes. We are collaborating on writing a curriculum that teachers like themselves would like to use. Based on our experiences and findings from the field, we have identified the components that we feel are key for CILIA-T. Three of these main components include

1. Multimodal input:  Contrary to how I started learning English, providing linguistic input in several different modalities is helpful. Technology provides many opportunities to realize this goal. Learners can interact with and produce content in many forms. For example, they can create and share academic vocabulary sets and review them like a game. Technology can also facilitate deep conceptual understanding of academic topics. For example, learners can share and discuss not only texts but also audios and videos for a deeper analysis and application of civics and history topics.
2. Build on first languages (L1): Adults already have a well-developed language system or systems, if they know multiple languages. They use the clues from their L1 to understand how English operates. Therefore, we can provide opportunities to bring that existing linguistic knowledge to the forefront and to compare and contrast explicitly. One clear way to leverage L1 is to integrate oral language and help bridge what the adults can do orally with what they aim to do in reading and writing.
3. Academic vocabulary: Individuals with limited or interrupted schooling tend to have lower levels of academic vocabulary in their first language, and thus, likely lower levels in English. A language learner may be quite fluent in using English in their everyday interactions, but that does not mean they have a strong academic vocabulary across different domains, such as health, math, science, civics, and finance. CILIA-T covers academic and discipline-specific vocabulary in a purposeful way. Academic vocabulary is closely related to the conceptual understanding of a phenomenon. Therefore, just learning word definitions is not enough. The vocabulary has to be contextualized with a conceptual understanding. For example, executive branch does not mean much by itself unless it is situated within an understanding of an overall governmental system. Similarly, the definition of the word mortgage may be forgotten quickly if the learner is not familiar with the loan and repayment system in the United States. Luckily, adults have a lot of background knowledge to facilitate such conceptual scaffolding, but that is for another blog.

What value do you hope CILIA-T might bring to the students, teachers, and communities?

We believe that when all individuals, but especially the newcomers, understand the systems, practices, historical contexts, and the language(s) of their society, they can become more active participants in their communities and can work towards accomplishing their life goals more effectively. We hope that CILIA-T provides the adult learners and the educators and programs that support them with a tool to facilitate this growth.

This blog was produced by Dr. Meredith Larson (Meredith.Larson@ed.gov), research analyst and program officer at NCER.