IES Blog

Institute of Education Sciences

New International Comparisons of Reading, Mathematics, and Science Literacy Assessments

The Program for International Student Assessment (PISA) is a study of 15-year-old students’ performance in reading, mathematics, and science literacy that is conducted every 3 years. The PISA 2018 results provide us with a global view of U.S. students’ performance compared with their peers in nearly 80 countries and education systems. In PISA 2018, the major domain was reading literacy, although mathematics and science literacy were also assessed.

In 2018, the U.S. average score of 15-year-olds in reading literacy (505) was higher than the average score of the Organization for Economic Cooperation and Development (OECD) countries (487). Compared with the 76 other education systems with PISA 2018 reading literacy data, including both OECD and non-OECD countries, the U.S. average reading literacy score was lower than in 8 education systems, higher than in 57 education systems, and not measurably different in 11 education systems. The U.S. percentage of top performers in reading was larger than in 63 education systems, smaller than in 2 education systems, and not measurably different in 11 education systems. The average reading literacy score in 2018 (505) was not measurably different from the average score in 2000 (504), the first year PISA was administered. Among the 36 education systems that participated in both years, 10 education systems reported higher average reading literacy scores in 2018 compared with 2000, and 11 education systems reported lower scores.

The U.S. average score of 15-year-olds in mathematics literacy in 2018 (478) was lower than the OECD average score (489). Compared with the 77 other education systems with PISA 2018 mathematics literacy data, the U.S. average mathematics literacy score was lower than in 30 education systems, higher than in 39 education systems, and not measurably different in 8 education systems. The average mathematics literacy score in 2018 (478) was not measurably different from the average score in 2003 (483), the earliest year with comparable data. Among the 36 education systems that participated in both years, 10 systems reported higher mathematics literacy scores in 2018 compared with 2003, 13 education systems reported lower scores, and 13 education systems reported no measurable changes in scores.  

The U.S. average score of 15-year-olds in science literacy (502) was higher than the OECD average score (489). Compared with the 77 other education systems with PISA 2018 science literacy data, the U.S. average science literacy score was lower than in 11 education systems, higher than in 55 education systems, and not measurably different in 11 education systems. The average science literacy score in 2018 (502) was higher than the average score in 2006 (489), the earliest year with comparable data. Among the 52 education systems that participated in both years, 7 education systems reported higher average science literacy scores in 2018 compared with 2006, 22 education systems reported lower scores, and 23 education systems reported no measurable changes in scores.

PISA is conducted in the United States by NCES and is coordinated by OECD, an intergovernmental organization of industrialized countries. Further information about PISA can be found in the technical notes, questionnaires, list of participating OECD and non-OECD countries, released assessment items, and FAQs.

 

By Thomas Snyder

IES at the Conference on Computing and Sustainable Societies

Over the summer, researchers, technologists, and policymakers gathered in Accra, Ghana for the Association for Computing Machinery’s Conference on Computing and Sustainable Societies (ACM COMPASS) to discuss the role of information technologies in international development.

Two IES-funded researchers from Carnegie Mellon University’s Program in Interdisciplinary Education Research, Michael Madaio and Dr. Amy Ogan, shared their research on developing voice-based early literacy technologies and evaluating their efficacy with low-literate, bilingual families in the Ivory Coast. 

Their research draws on methods from human-computer interaction, the learning sciences, and information-communication technology for development, to design educational technologies that are culturally and contextually appropriate.

Although the COMPASS conference focused on cross-cultural applications and technology for development, the research presented has implications for U.S. based education researchers, practitioners, and policymakers.

For instance, while research provides evidence for the importance of parental involvement in early literacy, parents with low literacy in the target language – as in many bilingual immigrant communities in the U.S. – may not be able to support their children with the explicit, instrumental help suggested by prior research (for example, letter naming or bookreading). This suggests that there may be opportunities for technology to scaffold low-literate or English Learners (EL) parental support in other ways.

At the conference, researchers described interactive voice-based systems (known as “IVR”) that help low-literate users find out about crop yields, understand local government policies, and engage on social media.  

This body of work has implications for designers of learning technologies in the U.S. Many families may not have a smartphone, but basic feature phones are ubiquitous worldwide, including in low-income, immigrant communities in the U.S. Thus, designers of learning technologies may consider designing SMS- or voice-based (such as IVR) systems, while schools or school districts may consider how to use voice-based systems to engage low-literate or EL families who may not have a smartphone or who may not be able to read SMS information messages.

In a rapidly changing, increasingly globalized world, research at IES may benefit from increased international engagement with international research, both focusing specifically on education, as well as information technology research that has implications for educational research, practice, and policy.

This guest blog was written by Michael Madaio. He is an IES Predoctoral Fellow in the Program in Interdisciplinary Education Research at Carnegie Mellon University. He is placed in the Human-Computer Interaction Institute.

In Nebraska, a focus on evidence-based reading instruction

Researchers have learned a lot about reading over the last few decades, but these insights don’t always make their way into elementary school classrooms. In Nebraska, a fresh effort to provide teachers with practical resources on reading instruction could help bridge the research-practice divide.

At the start of the 2019-2020 school year, the Nebraska Department of Education (NDE) implemented NebraskaREADS to help “serve the needs of students, educators, and parents along the journey to successful reading.” As part of a broader effort guided by the Nebraska Reading Improvement Act, the initiative emphasizes the importance of high-quality reading instruction and targeted, individualized support for struggling readers.

As part of NebraskaREADS, NDE is developing an online resource inventory of tools and information to support high-quality literacy instruction for all Nebraska students. After reaching out to REL Central for support, experts from both agencies identified the What Works Clearinghouse (WWC) as a prime source for evidenced-based information on instructional practices and policies and partnered with REL Central to develop a set of instructional strategies summary documents based on recommendations in eight WWC practice guides.

Each WWC practice guide presents recommendations for educators based on reviews of research by a panel of nationally recognized experts on a particular topic or challenge. NDE’s instructional strategies summary documents align the evidence-based strategies in the practice guides more directly with NebraskaREADS.

REL Central and NDE partners used the WWC practice guides to develop the 35 instructional strategies summary documents, each of which condenses one recommendation from a practice guide. For each recommendation, the instructional strategies summary document provides the associated NebraskaREADS literacy focus, implementation instructions, appropriate grade levels, potential roadblocks and ways to address them, and the strength of supporting evidence.

NDE launched the instructional strategies summary documents in spring 2019, and already educators have found them to be helpful resources. Dee Hoge, executive director of Bennington Public Schools, described the guides as “checkpoints for strong instruction” and noted her district’s plan to use them to adapt materials and provide professional development. Marissa Payzant, English language arts education specialist at NDE and a lead partner in this collaboration, shared that the “districts are excited to incorporate the strategy summaries in a variety of professional learning and other initiatives. They make the information in the practice guides much more accessible, and all the better it’s from a trustworthy source.”

Building on the positive reception and benefits of the instructional strategies summary documents, NDE and REL Central plan to expand the resource inventory into other content areas. Currently, they are working together to develop instructional strategy summaries using the WWC practice guides for math instruction.

by Douglas Van Dine​, Regional Educational Laboratory Central

Fieldnotes: Reflections from an Adult Education Instructor on Research and Practice

Approximately 18 percent of US adults are at the lowest levels of literacy and nearly 30 percent are at the lowest levels of numeracy. The adult education system serves adults with low skills, but many education researchers know little about the students or the setting.  Recently, NCER convened a working group of adult education instructors, administrators, and researchers to discuss adult education’s research and dissemination needs.

Mr. Marcus Hall, an adult education instructor at the Community Learning Center and JEVS Human Services in Philadelphia, participated in this working group. He spoke with Meredith Larson, NCER program officer for adult education, about his experiences and interests in research. A copy of the working group meeting summary is available here.

Please describe your adult education classroom.

I once taught a 7-week course with students ranging from 18 to over 60 years old who had low literacy or math scores. I tried to contextualize instruction around their career interests and differentiate it to their learning needs. For example, some students were proficient readers but needed comprehension and math practice while others struggled with one or more of the basic components of reading. Somehow, I needed to help those learning phonics and those struggling with fluency while also challenging those ready for comprehension work. It’s hard to meet student needs in such a short time without teacher aides or adaptive technologies.

Why is research particularly important for adult education?

The challenges we face are monumental. Despite the large number of adults in need, adult education feels under-funded, under-staffed, and under-appreciated. Our students need complex, comprehensive, and well-rounded intervention, but we often have to make the most out of slightly targeted, inexpensive, and difficult-to-implement solutions. We need researchers to provide practical information and recommendations that we can use today to help adults learn and retain information.

Have you used research into your teaching?

Specifically, for reading instruction, I use techniques and activities built on evidence-based reading interventions. I start with tested diagnostic assessments to determine the needs of my students followed by strategies such as Collaborative Oral Reading or Repeated Reading exercises to support my students.

What topic during the meeting stood out to you?

The discussion about the workforce and professional development resonated with me. Many of our educators are part-time, come out of K-12, close to retirement, and may not have specific training for working with adults. They are asked to teach subjects they may not have any certification in, and their programs may not be able to provide the professional development they need. Just as we need supports for our learners, we need research to develop supports for us educators.

What additional research would you like to see?

Many of my students have had traumatic experiences that, when relived in the classroom, can cause them to disengage or struggle. I feel that understanding triggers and signs of discomfort has greatly enhanced my ability to help my students. Many educators want to leverage mental health approaches, like trauma-informed care, but we could use help learning how to integrate these strategies into instruction.

What do you hope researchers and educators keep in mind regarding one another?

It seems that researchers publish and promote their work to other researchers and then move to the next topic. This may be due to time constraints, publishing demands, or institutional requirements. I hope researchers take the time to come into our settings and observe us in action. I want researchers to work with us to help us understand and accept what is and isn’t working.

As for educators, we need to not try things and then stop using them when something unexpected occurs. At times, we revert back to what we know and are most comfortable with in the classroom. We educators can and must think critically about our norms and be ready and willing to enhance our practice with new information. 

Equity Through Innovation: New Models, Methods, and Instruments to Measure What Matters for Diverse Learners

In today’s diverse classrooms, it is both challenging and critical to gather accurate and meaningful information about student knowledge and skills. Certain populations present unique challenges in this regard – for example, English learners (ELs) often struggle on assessments delivered in English. On “typical” classroom and state assessments, it can be difficult to parse how much of an EL student’s performance stems from content knowledge, and how much from language learner status. This lack of clarity makes it harder to make informed decisions about what students need instructionally, and often results in ELs being excluded from challenging (or even typical) coursework.

Over the past several years, NCER has invested in several grants to design innovative assessments that will collect and deliver better information about what ELs know and can do across the PK-12 spectrum. This work is producing some exciting results and products.

  • Jason Anthony and his colleagues at the University of South Florida have developed the School Readiness Curriculum Based Measurement System (SR-CBMS), a collection of measures for English- and Spanish-speaking 3- to 5-year-old children. Over the course of two back-to-back Measurement projects, Dr. Anthony’s team co-developed and co-normed item banks in English and Spanish in 13 different domains covering language, math, and science. The assessments are intended for a variety of uses, including screening, benchmarking, progress monitoring, and evaluation. The team used item development and evaluation procedures designed to assure that both the English and Spanish tests are sociolinguistically appropriate for both monolingual and bilingual speakers.

 

  • Daryl Greenfield and his team at the University of Miami created Enfoque en Ciencia, a computerized-adaptive test (CAT) designed to assess Latino preschoolers’ science knowledge and skills. Enfoque en Ciencia is built on 400 Spanish-language items that cover three science content domains and eight science practices. The items were independently translated into four major Spanish dialects and reviewed by a team of bilingual experts and early childhood researchers to create a consensus translation that would be appropriate for 3 to 5 year olds. The assessment is delivered via touch screen and is equated with an English-language version of the same test, Lens on Science.

  • A University of Houston team led by David Francis is engaged in a project to study the factors that affect assessment of vocabulary knowledge among ELs in unintended ways. Using a variety of psychometric methods, this team explores data from the Word Generation Academic Vocabulary Test to identify features that affect item difficulty and explore whether these features operate similarly for current, former, as well as students who have never been classified as ELs. The team will also preview a set of test recommendations for improving the accuracy and reliability of extant vocabulary assessments.

 

  • Researchers led by Rebecca Kopriva at the University of Wisconsin recently completed work on a set of technology-based, classroom-embedded formative assessments intended to support and encourage teachers to teach more complex math and science to ELs. The assessments use multiple methods to reduce the overall language load typically associated with challenging content in middle school math and science. The tools use auto-scoring techniques and are capable of providing immediate feedback to students and teachers in the form of specific, individualized, data-driven guidance to improve instruction for ELs.

 

By leveraging technology, developing new item formats and scoring models, and expanding the linguistic repertoire students may access, these teams have found ways to allow ELs – and all students – to show what really matters: their academic content knowledge and skills.

 

Written by Molly Faulkner-Bond (former NCER program officer).