IES Blog

NCES recently released the results of the Program for International Student Assessment (PISA) 2018 assessment of financial literacy. This assessment measured 15-year-old students’ knowledge and understanding of financial concepts, products, and risks and their ability to apply that knowledge to real-life situations. It found that, on average, U.S. students performed similarly to their peers across the 12 other participating Organization for Economic Cooperation and Development (OECD) countries. 

The assessment also found that 12 percent of U.S. students performed at the highest level of proficiency (level 5). Performance at this level indicates that students can apply their understanding of financial terms and concepts to analyze complex financial products, solve nonroutine financial problems, and describe potential outcomes of financial decisions in the big picture.^[1] The U.S. percentage was again similar to the OECD average.

However, this analysis also identified a group of students who might be considered “resilient.” In education research, resilience is defined as the ability to perform well academically despite coming from the disadvantaged backgrounds that have more commonly been associated with lower performance.

High-performing students came from across the spectrum of school poverty levels, as measured by the percentage of students eligible for free or reduced-price lunch (FRPL).^[2] In particular, 7 percent of high-performing students in financial literacy came from the highest poverty schools (figure 1).

Figure 1. Percentage distribution of U.S. 15-year-olds in public schools scoring below level 2 and at level 5 of proficiency on the PISA financial literacy scale, by percentage of students eligible for free or reduced-price lunch (FRPL) at their school: 2018

NOTE: Data for percentage of students eligible for FRPL were available for public schools only. An individual student’s level of poverty may vary within schools. Detail may not sum to totals due to rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2018.

---

It is these 7 percent of students who could be considered “resilient” and may be of interest for further study. For example, research could identify if there are factors that are associated with their high performance when compared to their lower performing peers in similar schools. Research on academically resilient students that used eighth-grade data from TIMSS found, for example, that having high educational aspirations increased the likelihood that students with few home education resources performed at or above the TIMSS Intermediate international benchmark in mathematics.^[3] Experiencing less bullying also increased this likelihood.

Examining the “resilient” PISA financial literacy students more closely could also determine the extent to which their individual backgrounds are related to performance. This would be of interest because, even within high-poverty schools, students’ individual circumstances may vary. 

Patterns in Other PISA Subjects

There are similar subsets of “resilient” students in the other PISA 2018 subjects (table 1). Eight percent of high performers in reading were from the highest poverty schools, as were 5 percent of high performers in mathematics and 7 percent of high performers in science.

---

Table 1. Percentage of U.S. 15-year-olds in public schools scoring at or above level 5 of proficiency, by PISA subject and their schools’ free or reduced-price lunch (FRPL) status: 2018

[Standard errors appear in parentheses]

NOTE: Results are scaled separately; thus, percentages cannot be compared across subjects. Level 5 is the highest level of proficiency in financial literacy; levels 5 and 6 are the highest levels of proficiency in the other PISA subjects. Data for students eligible for FRPL were available for public schools only.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2018.

---

For more information on the PISA 2018 results in financial literacy and other subjects, visit the NCES International Activities website. To create customized data and charts using PISA and other international assessment data, use the International Data Explorer.

 

By Maria Stephens, AIR

---

[1] See https://nces.ed.gov/surveys/pisa/pisa2018/index.asp#/finance/intlcompare.

[2] Data for students eligible for FRPL are available for public schools only.

[3] Students at the Intermediate international benchmark can apply basic mathematical knowledge in a variety of situations, and those above this benchmark can do so in increasingly complex situations and, at the highest end, reason with information, draw conclusions, make generalizations, and solve linear equations.

This blog post is part of an interview series with education researchers who have successfully scaled their interventions.

Assessment-to-Instruction (A2i) is an online Teacher Professional Support System that guides teachers in providing Kindergarten to Grade 3 students individualized literacy instruction and assessments. Students complete the assessments independently online without the teacher taking time away from instruction. A2i generates instantaneous teacher reports with precise recommendations for each student and group recommendations. See a video demo here. Between 2003 and 2017, researchers at Florida State University (FSU) and Arizona State University (ASU), led by Carol Connor, developed and evaluated A2i with the support of a series of awards from IES and the National Institutes of Health. Findings from all publications on the A2i are posted here.

While results across seven controlled studies demonstrated the effectiveness of A2i, feedback from practitioners in the field demonstrated that implementation often required substantial amounts of researcher support and local district adaptation, and that the cost was not sustainable for most school district budgets. In 2014, the development firm Learning Ovations, led by Jay Connor, received an award from the Department of Education (ED) and IES’s Small Business Innovation Research program (ED/IES SBIR) to develop an technologically upgraded and commercially viable version of A2i to be ready to be used at scale in classrooms around the country. In 2018, with the support of a five-year Education Innovation and Research (EIR) expansion grant from ED totaling $14.65 million, A2i is now used in more than 110 schools across the country, with plans for further expansion. 

Interview with Carol Connor (CC) and Jay Connor (JC)

From the start of the research in the early 2000s, was it always the goal to develop a reading intervention that would one day be used on a wide scale?
CC: Yes and no. First, we had to answer the question as to whether individualization was effective in achieving student literacy outcomes. Once the research established that, we knew that this work would have wide-scale application.

When did you start thinking about a plan for distribution? 
CC: Before embarking on the cumulative results studies, in 2008, Jay said that we needed to know who the “customer” was… i.e., how purchasing decisions were made at scale.  His 2008 Phase I ED/IES SBIR was critical in shifting our research focus from individual classrooms to school districts as the key scaling point. 

Did you work with a technology transfer office at the university?
CC: Only to the extent of contractually clarifying intellectual property (IP) ownership and licensing. 

Who provided the support on the business side?
CC: Jay, who has an MBA/JD and has been a senior officer in two Fortune 100 companies was very instrumental in guiding our thinking of this evolution from important research to practical application. 

 Do you have any agreement about the IP with the university? What were the biggest challenges in this area?

JC: Yes, Learning Ovations has a 60-year renewable exclusive licensing agreement with FSU Foundation. FSU couldn’t have been better to work with.  Though there were expected back-and-forth elements of the original negotiations, it was clear that we shared the central vision of transforming literacy outcomes.  They continue to be a meaningful partner.

When and why was Learning Ovations first launched?
JC: In order to pursue SBIR funding we needed to be a for-profit company.  At first, I used my consulting business – Rubicon Partners LLP – as the legal entity for a 2008 Phase I award from ED/IES SBIR.  When we considered applying (and eventually won) a Fast Track Phase I & II award from SBIR in 2014, it was clear that we needed to create a full C – Corp that could expand with the scaling of the business, thus Learning Ovations was formed.

Who has provided you great guidance on the business side over the year? What did they say and do? 
JC: Having run large corporate entities and worked with small business start-ups in conjunction with Arizona State University (Skysong) and the University of California, Irvine (Applied Innovation at The Cove) and having taught entrepreneurship at The Paul Merage School of Business at UC Irvine, I had the experience or network to connect to whatever business guidance we needed.  Further, having attended a number of reading research conferences with Carol, I was quite conversant in the literacy language both from the research side and from the district decision maker’s side.

How do you describe the experience of commercializing the A2i? What were the biggest achievements and challenges in terms of preparing for commercialization?

JC: Having coached scores of entrepreneurs at various stages, I can safely say that there is no harder commercialization than one that must stay faithful to the underlying research.  A key strategy for most new businesses: being able to pivot as you find a better (easier) solution.  It is often circumscribed by the “active ingredients” of the underlying research.  Knowing this, we imbued Learning Ovations with a very strong outcomes mission – all children reading at, or above, grade level by 3^rd grade.  This commitment to outcomes certainty is only assured by staying faithful to the research.  Thus, a possible constraint, became our uncontroverted strength.

Do you have advice for university researchers seeking to move their laboratory research in education into wide-spread practice? 
JC:  Start with the end in mind.  As soon as you envision wide-scale usage, learn as much as you can about the present pain and needs of your future users and frame your research questions to speak to this.  Implementation should not be an after-the-fact consideration; build it into how you frame your research questions. On one level you are asking simultaneously “will this work with my treatment group” AND “will this help me understand/deliver to my end-user group.”  I can’t imagine effective research being graphed onto a business after the fact.  One key risk that we see a number of researchers make is thinking in very small fragments whereas application (i.e., the ability to go to scale) is usually much more systemic and holistic.

In one sentence, what would say is most needed for gaining traction in the marketplace?
JC: If not you, as a researcher, someone on your team of advisors needs to know the target marketplace as well as you know the treatment protocols in your RCT.

____________

Carol Connor is a Chancellor’s Professor in the UC Irvine School of Education. Prior she was a professor of Psychology and a Senior Learning Scientist at the Learning Sciences Institute at ASU. Carol’s research focuses on teaching and learning in preschool through fifth grade classrooms – with a particular emphasis on reading comprehension, executive functioning, and behavioral regulation development, especially for low-income children.

Joseph “Jay” Connor, JD/MBA, is the Founder/CEO of Learning Ovations, Inc, the developer of the platform that has enabled the A2i intervention to scale.  Jay has 20+ years of experience in senior business management at the multi-billion dollar corporate level, and has experience in the nonprofit and public policy arenas.

This interview was produced by Edward Metz of the Institute of Education Sciences.

The Program for International Student Assessment (PISA) is a study of 15-year-old students’ performance in reading, mathematics, and science literacy that is conducted every 3 years. The PISA 2018 results provide us with a global view of U.S. students’ performance compared with their peers in nearly 80 countries and education systems. In PISA 2018, the major domain was reading literacy, although mathematics and science literacy were also assessed.

In 2018, the U.S. average score of 15-year-olds in reading literacy (505) was higher than the average score of the Organization for Economic Cooperation and Development (OECD) countries (487). Compared with the 76 other education systems with PISA 2018 reading literacy data, including both OECD and non-OECD countries, the U.S. average reading literacy score was lower than in 8 education systems, higher than in 57 education systems, and not measurably different in 11 education systems. The U.S. percentage of top performers in reading was larger than in 63 education systems, smaller than in 2 education systems, and not measurably different in 11 education systems. The average reading literacy score in 2018 (505) was not measurably different from the average score in 2000 (504), the first year PISA was administered. Among the 36 education systems that participated in both years, 10 education systems reported higher average reading literacy scores in 2018 compared with 2000, and 11 education systems reported lower scores.

The U.S. average score of 15-year-olds in mathematics literacy in 2018 (478) was lower than the OECD average score (489). Compared with the 77 other education systems with PISA 2018 mathematics literacy data, the U.S. average mathematics literacy score was lower than in 30 education systems, higher than in 39 education systems, and not measurably different in 8 education systems. The average mathematics literacy score in 2018 (478) was not measurably different from the average score in 2003 (483), the earliest year with comparable data. Among the 36 education systems that participated in both years, 10 systems reported higher mathematics literacy scores in 2018 compared with 2003, 13 education systems reported lower scores, and 13 education systems reported no measurable changes in scores.  

The U.S. average score of 15-year-olds in science literacy (502) was higher than the OECD average score (489). Compared with the 77 other education systems with PISA 2018 science literacy data, the U.S. average science literacy score was lower than in 11 education systems, higher than in 55 education systems, and not measurably different in 11 education systems. The average science literacy score in 2018 (502) was higher than the average score in 2006 (489), the earliest year with comparable data. Among the 52 education systems that participated in both years, 7 education systems reported higher average science literacy scores in 2018 compared with 2006, 22 education systems reported lower scores, and 23 education systems reported no measurable changes in scores.

PISA is conducted in the United States by NCES and is coordinated by OECD, an intergovernmental organization of industrialized countries. Further information about PISA can be found in the technical notes, questionnaires, list of participating OECD and non-OECD countries, released assessment items, and FAQs.

By Thomas Snyder

The average reading score for U.S. 4th- and 8th-grade students decreased between 2017 and 2019. Changes in mathematics scores were mixed during this period, with an increase at grade 4 and a decrease at grade 8. These data are from the National Assessment of Educational Progress (NAEP)—also known as The Nation’s Report Card. NAEP is the largest nationally representative and continuing assessment of what students in the United States know and can do in various subject areas and is frequently referred to as the “gold standard” of student assessments.

In 4th-grade reading, the average scale score in 2019 was 220, one point lower than in 2017 (figure 1). In 8th-grade reading, the average scale score was 263, three points lower than in 2017 (figure 2). Compared with a decade ago in 2009, the 2019 average reading scale scores at each grade were not significantly different, but they were higher than the scale scores in 1992, the first time the reading assessment was administered.

Figure 1. Average National Assessment of Educational Progress (NAEP) reading scale scores of 4th-grade students: Selected years, 1992–2019

* Significantly different (p < .05) from 2019

--- Accommodations not permitted

— Accommodations permitted

Figure 2. Average National Assessment of Educational Progress (NAEP) reading scale scores of 8th-grade students: Selected years, 1992–2019

* Significantly different (p < .05) from 2019

--- Accommodations not permitted

— Accommodations permitted

In 4th-grade mathematics, the average scale score in 2019 was 241, one point higher than in 2017 (figure 3). In 8th-grade mathematics, the average scale score in 2019 was 282, one point lower than in 2017 (figure 4). Like reading, average scale scores for mathematics at both grades in 2019 were not significantly different than in 2009. Mathematics scale scores for both grade were higher in 2019 than in 1990, the first time the mathematics assessments were administered.

Figure 3. Average National Assessment of Educational Progress (NAEP) mathematics scale scores of 4th-grade students: Selected years, 1990–2019

* Significantly different (p < .05) from 2019

--- Accommodations not permitted

— Accommodations permitted

Figure 4. Average National Assessment of Educational Progress (NAEP) mathematics scale scores of 8th-grade students: Selected years, 1990–2019

* Significantly different (p < .05) from 2019

--- Accommodations not permitted

— Accommodations permitted

The Nation’s Report Card also presents data by different demographic groups—such as race/ethnicity—gender, school type, and region. White and Black 4th- and 8th-grade students scored lower in reading in 2019 than in 2017. Hispanic and American Indian/Alaska Native 8th-grade students also scored lower in reading in 2019 than in 2017. In mathematics, 4th-grade Hispanic students scored higher in 2019 than in 2017, and 8th-grade American Indian/Alaska Native students scored lower in 2019 than in 2017. From 2017 to 2019, males’ scores increased in mathematics at grade 4 but decreased in reading at both grades.

NCES administered the 2019 NAEP mathematics and reading assessments to almost 600,000 4th- and 8th-graders in public and private schools in all 50 states, the District of Columbia, the U.S. Department of Defense schools, and 27 urban districts. Samples of schools and students are drawn from each state and from the District of Columbia and Department of Defense schools.

Visit https://nces.ed.gov/nationsreportcard/ to view the report.

Which schools would you guess, on average, spend more instructional time on English, reading, and language arts—public schools or private schools? How about on mathematics?

These questions and many others are answered in recently released reports on U.S. public and private schools and principals. The data in these reports are from the 2017–18 National Teacher and Principal Survey (NTPS), which is administered by the National Center for Education Statistics (NCES). NTPS previously collected data from public schools, principals, and teachers during the 2015–16 school year, but this is the first private school collection since the 2011–12 school year. (The latest NTPS data on public and private school teachers will be released later this year.)

The NTPS collects data about principals’ educational backgrounds and goals, as well as the climate of their schools and other general information about their schools and special programs and services provided. These data serve as a resource for researchers, policymakers, and the general public who are interested in understanding the current experiences and conditions of U.S. public and private schools.

The 2017–18 NTPS featured several new topic areas, such as the following:

• School instruction time. Overall, schools reported that third-graders spent a weekly average of 500 minutes on instruction in English, reading, and language arts; 350 minutes on instruction in arithmetic or mathematics; and 170 minutes each on instruction in science and social studies or history. Here are some data to answer the questions from the beginning of this post:
  • Public schools reported that third-graders spent a weekly average of 540 minutes on instruction in English, reading, and language arts; 370 minutes on instruction in arithmetic or mathematics; 170 minutes on instruction in science; and 160 minutes on instruction in social studies or history.
  • Private schools reported that third-graders spent a weekly average of 400 minutes on instruction in English, reading, and language arts; 280 minutes on instruction in arithmetic or mathematics; and 170 minutes each on instruction in science and social studies or history.
     

Figure 1. Average minutes reported by public and private schools that third-grade students spend on selected subjects per week: 2017–18

NOTE: Schools that reported 0 minutes per week for a subject were excluded from the calculations of average minutes per week.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), “Public School and Private School Documentation Data Files,” 2017–18. Please see Characteristics of Public and Private Elementary and Secondary Schools in the United States: Results From the 2017–18 National Teacher and Principal Survey First Look, table 7.

• ​Principals’ professional development. Overall, 83 percent of all principals reported participating in any professional development activities in the 2016–17 school year. Specifically, 85 percent of public school principals and 77 percent of private school principals reported doing so.

• Evaluation of principals. Among public school principals, relatively more principals in traditional public schools were evaluated during the last school year than were principals in public charter schools (79 and 69 percent, respectively). Relatively more private school principals in Catholic and nonsectarian schools (63 and 58 percent, respectively) were evaluated during the last school year than were principals in other religious schools (41 percent).

Data files for the 2017–18 school and principal questionnaires will be released later this year. In order to protect the identities of responding schools and principals, researchers must apply for a restricted-use license to access the full restricted-use data files. Data will also be available through NCES’ online data tool, DataLab, where users can create custom tables and regressions without a restricted-use license.

By Maura Spiegelman