IES Blog

Institute of Education Sciences

Better Reading Comprehension When You Know That You Don’t Know

The more you already know about a topic, the easier it may be to comprehend and learn from texts about that topic. But knowledge has to start somewhere. So how can we help students learn from texts when they may have low background knowledge?

In their exploratory study, researchers from ETS found that lack of knowledge is not necessarily a barrier to comprehension. Rather, they suggest that students who can identify their lack of background knowledge are more likely to comprehend and learn new information than students who do not acknowledge they lack background knowledge. In other words, knowing that you might not know may lead to better outcomes.

To determine the role of background knowledge, the researchers pretested middle and high school students’ background knowledge through questions related to topics the students may have some but not complete knowledge of, such as ecology, immigration, and wind power. The pretest included an “I don’t know” option, along with correct and incorrect responses.

Students then took a scenario-based assessment in which they read multiple sources about each of the topics. This type of assessment mirrors real-world learning by encouraging readers to build their own interpretations of a topic, which helps researchers determine whether students comprehend what they read.

They found that students who selected “I don’t know” when answering background knowledge questions had better understanding of the content than those who provided wrong answers on these questions. In fact, students who selected “I don’t know” rather than answering incorrectly were nearly three times as likely to learn from sources that provided the correct information than students who had answered the pretest incorrectly. Students who selected “I don’t know” may also learn more than students who had a comparable level of weak background knowledge. The researchers suggest that the “I don’t know” readers may have set different reading goals prior to engaging with the sources than those who guessed incorrectly.

 

Possible Implications for Teaching and Learning

The results from this work support the idea that having and building background knowledge is key. Thus, teachers may want to assess existing knowledge and address knowledge gaps prior to instruction.

Teachers may also want to provide an “I don’t know” option or options that allow students to rate their level of certainty. Doing so may help teachers distinguish between students who recognize their own gaps in knowledge from those who may not be aware that they are wrong or that they simply do not know. This latter group of students may need more help in determining the accuracy of their judgments or may have incorrect knowledge that could interfere with learning.

The researchers further suggest that teachers may want to go beyond the role of background knowledge by teaching students how to set appropriate reading goals and use strategic reading approaches to learn new facts or correct existing misunderstandings.

 


The research reported here was conducted under NCER grant R305A150176: What Types of Knowledge Matters for What Types of Comprehension? Exploring the Role of Background Knowledge on Students' Ability to Learn from Multiple Texts.

This blog was written by Dr. Meredith Larson. Contact her for more information about this project.

New International Data Show Large and Widening Gaps Between High- and Low-Performing U.S. 4th- and 8th-Graders in Mathematics and Science

NCES recently released results from the 2019 Trends in International Mathematics and Science Study (TIMSS). TIMSS tests students in grades 4 and 8 in mathematics and science every 4 years. The results show that

  • Across both subjects and grades, the United States scored, on average, in the top quarter of the education systems that took part in TIMSS 2019.
    • Among the 64 education systems that participated at grade 4, the United States ranked 15th and 8th in average mathematics and science scores, respectively.
    • Among the 46 education systems that participated at grade 8, the United States ranked 11th in average scores for both subjects.
  • On average, U.S. scores did not change significantly between the 2011 and 2019 rounds of TIMSS.

Average scores are one measure of achievement in national and international studies. However, they provide a very narrow perspective on student performance. One way to look more broadly is to examine differences in scores (or “score gaps”) between high-performing students and low-performing students. Score gaps between high performers and low performers can be one indication of equity within an education system. Here, high performers are those who scored in the 90th percentile (or top 10 percent) within their education system, and low performers are those who scored in the 10th percentile (or bottom 10 percent) within their education system.

In 2019, while some education systems had a higher average TIMSS score than the United States, none of these education systems had a wider score gap between their high and low performers than the United States. This was true across both subjects and grades.

Figure 1 shows an example of these findings using the grade 8 mathematics data. The figure shows that 17 education systems had average scores that were higher or not statistically different from the U.S. average score.

  • Of these 17 education systems, 13 had smaller score gaps between their high and low performers than the United States. The score gaps in 4 education systems (Singapore, Chinese Taipei, the Republic of Korea, and Israel) were not statistically different from the score gap in the United States.
  • The score gaps between the high and low performers in these 17 education systems ranged from 170 points in Quebec, Canada, to 259 points in Israel. The U.S. score gap was 256 points.
  • If you are interested in the range in the score gaps for all 46 education systems in the TIMSS 2019 grade 8 mathematics assessment, see Figure M2b of the TIMSS 2019 U.S. Highlights Web Report, released in December 2020. This report also includes these results for grade 8 science and both subjects at the grade 4 level.

Figure 1. Average scores and 90th to 10th percentile score gaps of grade 8 students on the TIMSS mathematics scale, by education system: 2019

NOTE: This figure presents only those education systems whose average scores were similar to or higher than the U.S. average score. Scores are reported on a scale of 0 to 1,000 with a TIMSS centerpoint of 500 and standard deviation of 100.

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Trends in International Mathematics and Science Study (TIMSS), 2019.


From 2011 to 2019, U.S. average scores did not change significantly. However, the scores of low performers decreased, and score gaps between low and high performers grew wider in both subjects and grades. In addition, at grade 8, there was an increase in the scores of high performers in mathematics and science over the same period. These two changes contributed to the widening gaps at grade 8.

Figure 2 shows these results for the U.S. grade 8 mathematics data. Average scores in 2011 and 2019 were not significantly different. However, the score of high performers increased from 607 to 642 points between 2011 and 2019, while the score of low performers decreased from 409 to 385 points. As a result, the score gap widened from 198 to 256 points between 2011 and 2019. In addition, the 2019 score gap for grade 8 mathematics is significantly wider than the gaps for all previous administrations of TIMSS.


Figure 2. Trends in average scores and selected percentile scores of U.S. grade 8 students on the TIMSS mathematics scale: Selected years, 1995 to 2019

* p < .05. Significantly different from the 2019 estimate at the .05 level of statistical significance.

NOTE: Scores are reported on a scale of 0 to 1,000 with a TIMSS centerpoint of 500 and standard deviation of 100.

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Trends in International Mathematics and Science Study (TIMSS), 1995, 1999, 2003, 2007, 2011, 2015, 2019.


These TIMSS findings provide insights regarding equity within the U.S. and other education systems. Similar results from the National Assessment of Educational Progress (NAEP) show that mathematics scores at both grades 4 and 8 decreased or did not change significantly between 2009 and 2019 for lower performing students, while scores increased for higher performing students. More national and international research on the gap between high- and low-performing students could help inform important education policy decisions that aim to address these growing performance gaps.

To learn more about TIMSS and the 2019 U.S. and international results, check out the TIMSS 2019 U.S. Highlights Web Report and the TIMSS 2019 International Results in Mathematics and Science. Registration is also open for an upcoming RISE Webinar on February 24, 2021 (What Do TIMSS and NAEP Tell Us About Gaps Between High- and Low-Performing 4th and 8th Graders?) that explores these topics further.  

 

By Katie Herz, AIR; Marissa Hall, AIR; and Lydia Malley, NCES

The Importance of Partnering with Practitioners in English Learner Research

IES values and encourage collaborations between researchers and practitioners to ensure that research findings are relevant, accessible, feasible, and useful. In FY 2014, Dr. Karen Thompson was awarded a grant for The Oregon English Learner Alliance: A Partnership to Explore Factors Associated with Variation in Outcomes for Current and Former English Learners in Oregon to determine best practices to support academic achievement among current and former English learners. Dr. Thompson and her colleagues wrote a guest blog post describing the work that the partnership undertook to better understand and improve the performance of English learners in Oregon. In this blog, we interviewed Dr. Thompson—three years after the end of the grant—to get her perspectives on the partnership, outcomes of their work, and where things currently stand.

 

What was the purpose of your research and what led you to do this work?

When I came to Oregon from California in 2012, there was growing momentum in the state to better understand and meet the needs of the state’s multilingual student population, particularly students classified as English learners (ELs). The state had developed an ambitious EL strategic plan, which included a variety of goals and action steps, such as identifying model programs and sharing best practices. I noticed that Oregon did not have publicly available information about the state’s former EL students. In prior work, other researchers and I had demonstrated that analyzing data only about students currently classified as English learners without also analyzing data about former EL students can provide incomplete and misleading information. Therefore, for Oregon to realize its goals and truly understand which programs and practices were most effectively educating its multilingual students, the state needed to make changes to its data systems. This was the seed that led to the Oregon Department of Education/Oregon State University English Language Learner Partnership. Our first goal was to simply determine how many former EL students there were in the state. Then, once the state had created a flag to identify former EL students, we were able to conduct a wide range of analyses to better understand opportunities and outcomes for both current and former EL students in ways that have informed state reporting practices and policy decisions.

 

How does this research differ from other work in the field? Why do you think partnerships with practitioners were necessary to carry out the work?

When we began our partnership, collecting and analyzing information about both current and former EL students was not common. Happily, more and more researchers and education agencies have now adopted these approaches, and we think our partnership has helped play a role in this important and illuminating shift.  

It was crucial to conduct this work via partnerships between researchers and practitioners. Practitioner partners had deep knowledge of the state’s current data systems, along with knowledge about which reporting and analysis practices could shift to incorporate new information about current and former EL students. Research partners had the bandwidth to conduct additional analyses and to lead external dissemination efforts. Our regular partnership meetings enabled our work to evolve in response to new needs. 

 

What do you think was the most important outcome of your work and why?

I think the most important outcome of our work is that educators across Oregon now have information about both their current and former English learner students and can use this data to inform policy and practice decisions. Other analyses we conducted have also informed state actions. For example, our analysis of how long it takes Oregon EL students to develop English proficiency and exit EL services informed the state’s EL progress indicator under the Every Student Succeeds Act.

 

What are the future directions for this work?

Our IES-funded partnership led to funding from the Spencer Foundation to do further research about EL students with disabilities in Oregon, which has impacted practices in the state. In addition, I am excited to be one of the collaborators in the new IES-funded National Research and Development Center to Improve Education for Secondary English Learners (PI: Aída Walquí, WestEd). As part of the Center’s research, I am working with colleagues at the University of Oregon and the University of California, Los Angeles to analyze malleable factors impacting content-course access and achievement for secondary EL students. We are collaborating with four states in this work, and as in our ODE/OSU partnership, we will be analyzing data for both current and former EL students. At a policy level, colleagues and I are involved in conversations about how data collection and reporting at the federal level could also incorporate analysis of data for both current and former EL students, including ways this might inform future reauthorizations of the Elementary and Secondary Education Act.

 

---

Dr. Karen Thompson is an Associate Professor at the College of Education at Oregon State University. Her research focuses on how curriculum and instruction, teacher education and policy interact to share the classroom experiences of K-12 multilingual students.

 

Written by Helyn Kim (Helyn.Kim@ed.gov), Program Officer for English Learner Program, National Center for Education Research.

Online Training for the 2019 NHES Early Childhood Program Participation Survey Data and Parent and Family Involvement in Education Survey Data

The NCES National Household Education Survey (NHES) program administered two national surveys in 2019—the Early Childhood Program Participation (ECPP) survey and the Parent and Family Involvement in Education (PFI) survey. The ECPP survey collects information on young children’s care and education, including the use of home-based care with both relatives and nonrelatives and center-based care and education. The survey examines how well these care arrangements cover work hours, costs of care, location of care, the process of selecting care, and factors making it difficult to find care. The PFI survey collects information on a range of issues related to how families connect to schools, including information on family involvement with schools, school choice, homeschooling, virtual education, and homework practices.

NCES released data from the 2019 NHES administration on January 28, 2021. For each of the two surveys, this release includes the following:

  • Public-use data files, in ASCII, CSV, SAS, SPSS, Stata, and R
  • Restricted-use data files (in formats listed above and with codebook)
  • Public-Use Data File Codebook
  • Data File User’s Manual (for both public-use and restricted-use files)

That’s a lot of information! How should you use it? We suggest you start by viewing the NHES online data Distance Learning Dataset Training modules. The modules provide a high-level overview of the NHES program and the data it collects. They also include important considerations to ensure that your analysis takes into account the NHES’s complex sample design (such as applying weights and estimating standard errors).   

You should first view the five general NHES modules, which were developed for the 2012 NHES data. These modules are:

  • Introduction to the NHES
  • Getting Started with the NHES Data
  • Data Collected Through the NHES
  • NHES Sample Design, Weights, Variance, and Missing Data
  • Considerations for Analysis of NHES Data

A sixth module explains key changes in the 2019 ECPP and PFI surveys compared to their respective 2012 surveys:

  • Introduction to the 2019 NHES Data Collection

The sixth module also provides links to the 2019 ECPP and PFI data, restricted-use licensing information, and other helpful resources.

Now you are ready to go! If you have any questions, please contact us at NHES@ed.gov.

By Lisa Hudson, NCES

National Mentoring Month: Celebrating Mentors in Special Education Research

January marks the 20th annual National Mentoring Month, a campaign that was formally established by former President George W. Bush in 2002. National Mentoring Month recognizes mentorship opportunities for young individuals across the United States, with the goal of improving academic, social, and economic opportunities to strengthen communities. In honor of National Mentoring Month, we are showcasing two programs from the National Center for Special Education Research (NCSER) that promote mentorship in special education research – the Early Career Development and Mentoring program and the Multi-Tiered Systems of Support (MTSS) Network from the Research Networks Focused on Critical Problems of Policy and Practice in Special Education program.

Early Career Development and Mentoring

The Early Career Development and Mentoring (Early Career) program, part of NCSER’s Research Training in Special Education, supports projects that prepare early career researchers to conduct independent, rigorous, and relevant early intervention and special education research. NCSER established this training program to support investigators in the early stages of their faculty or research scientist positions at academic institutions. This program prepares early career researchers to develop and evaluate instructional approaches, design and validate assessments, and address applied research problems using advanced methods and statistical analyses. As part of an integrated research and career development plan, investigators with Early Career grants identify one or more mentors with relevant expertise with whom they meet regularly in order to accomplish their grant goals. They receive feedback and guidance on research methods, data analysis and interpretation, dissemination, and grant writing. The ultimate goal of this program is to help launch the independent research careers for scientists interested in focusing on children with or at risk for disabilities, leading to an increased capacity of the field to conduct rigorous research.

Research Networks Focused on Critical Problems of Policy and Practice in Special Education: MTSS Network

The Research Networks Focused on Critical Problems of Policy and Practice in Special Education program establishes a structure for researchers working on high-priority issues in special education to share ideas, build new knowledge, and strengthen research and dissemination capacity. An important part of this network structure is the cross-team training of early career researchers. The MTSS Network was established as the first network under this program, conducting research examining integrated academic and behavioral MTSS in elementary schools. The MTSS Network, which consists of four research teams and one network lead, has established an Early Career Scholars program. Brandi Simonsen (Co-Principal Investigator on both the network lead and a research team) recently shared some information about this program. “The IES Research Network on Integrated Multi-Tiered Systems of Support engages two cohorts of Early Career Scholars in a range of mentoring activities to develop competency in conducting rigorous and relevant research on MTSS.” For example, mentorship activities for Early Career Scholars have included large group meetings to discuss Integrated Multi-Tiered Systems of Support: Blending RTI and PBIS by McIntosh and Goodman (Goodman is a member of the MTSS Network), which have provided opportunities for scholars to review and learn about integrated MTSS while engaging in discussion of ideas with network members. Scholars also meet in small groups with MTSS Network investigators to discuss specific research projects. For example, the early career scholars on the University of Connecticut research team meet with the investigators weekly to discuss on-going supports for participating schools, refine plans for research studies, and continue other grant-related activities.

For more information about NCSER’s programs of research, please see here.

This blog was authored by Alice Bravo (University of Washington), IES intern through the Virtual Student Federal Service. For more information about the Early Career Development and Mentoring program, contact Dr. Katie Taylor. For more information on the Research Networks program, contact Dr. Amy Sussman.