IES Blog

Institute of Education Sciences

What National and International Assessments Can Tell Us About Technology in Students’ Learning: Technology Instruction, Use, and Resources in U.S. Schools

As schools and school districts plan instruction amid the current coronavirus pandemic, the use of technology and digital resources for student instruction is a key consideration.

In this post, the final in a three-part series, we present results from the NAEP TEL and ICILS educator questionnaires (see the first post for information about the results of the two assessments and the second post for the results of the student questionnaires). The questionnaires ask about the focus of technology instruction in schools, school resources to support technology instruction, and the use of technology in teaching practices.

It is important to note that NAEP TEL surveys the principals of U.S. eighth-grade students, while ICILS surveys a nationally representative sample of U.S. eighth-grade teachers.

Emphasis in technology instruction

According to the 2018 NAEP TEL principal questionnaire results, principals1 of 61 percent of U.S. eighth-grade students reported that prior to or in eighth grade, much of the emphasis in information and communication technologies (ICT) instruction was placed on teaching students how to collaborate with others. In addition, principals of 51 percent of eighth-grade students reported that a lot of emphasis was placed on teaching students how to find information or data to solve a problem. In comparison, principals of only 10 percent of eighth-grade students reported that a lot of emphasis was placed on teaching students how to run simulations (figure 1).



According to the 2018 ICILS teacher questionnaire results, 40 percent of U.S. eighth-grade teachers reported a strong emphasis on the use of ICT instruction to develop students’ capacities to use computer software to construct digital work products (e.g., presentations). In addition, 35 percent of eighth-grade teachers reported a strong emphasis on building students’ capacities to access online information efficiently. In comparison, 17 percent reported a strong emphasis on developing students’ capacities to provide digital feedback on the work of others (figure 2).  



Resources at school

NAEP TEL and ICILS used different approaches to collect information about technology-related school resources. NAEP TEL asked about hindrances that limited schools’ capabilities to provide instruction in technology or engineering concepts. According to NAEP TEL, principals of 5 percent of U.S. eighth-grade students indicated that a lack or inadequacy of internet connectivity was a “moderate” or “large” hindrance in their schools. However, principals of 61 percent of eighth-grade students indicated that a lack of time due to curriculum content demands was a “moderate” or “large” hindrance. Principals of 44 percent of eighth-grade students indicated that a lack of qualified teachers was a “moderate” or “large” hindrance (figure 3).



ICILS asked about the adequacy of school resources to support ICT use in teaching. Eighty-six percent of U.S. teachers “agreed” or “strongly agreed” that technology was considered a priority for use in teaching. Nearly three-quarters of teachers “agreed” or “strongly agreed” that their schools had access to sufficient digital learning resources and had good internet connectivity (74 and 73 percent, respectively) (figure 4).



Use of technology in teaching

Teachers of U.S. eighth-grade students reported that they often used technology in their teaching practices. ICILS found that 64 percent of U.S. teachers regularly (i.e., “often” or “always”) used technology to present class instruction. Fifty-four percent of teachers regularly used technology to communicate with parents or guardians about students’ learning. In addition, 45 percent of teachers regularly used technology to provide remedial or enrichment support to individual or small groups of students, and a similar percentage (44 percent) regularly used technology to reinforce skills through repetition of examples (figure 5).



ICILS also reported results from U.S. eighth-grade teachers about how they collaborated on technology use. About three-quarters “agreed” or “strongly agreed” that they talked to other teachers about how to use technology in their teaching. Similarly, about three-quarters “agreed” or “strongly agreed” that they shared technology resources with other teachers in the school. More than half of the teachers “agreed” or “strongly agreed” that they collaborated with colleagues on the development of technology-based lessons.

Overall, the responses of teachers and principals suggested that emphasis had been put on different aspects of instruction for eighth-grade students. The majority of schools had enough digital resources and adequate internet access. However, technologies were also used differently in different teaching practices.

It should be noted that the data presented here were collected in 2018; any changes since then due to the coronavirus pandemic are not reflected in the results reported here. The NAEP TEL and ICILS samples both include public and private schools. The 2018 ICILS also included a principal questionnaire, but the questions are not directly related to the topics included in this blog. Data reported in the text and figures are rounded to the nearest integer.

 

Resources for more information:

 

By Yan Wang, AIR, and Taslima Rahman, NCES


[1] The unit of analysis for TEL principal responses is student.

What National and International Assessments Can Tell Us About Technology in Students’ Learning: Eighth-Graders’ Experience with Technology

The use of technology has become an integral part of life at work, at school, and at home throughout the 21st century and, in particular, during the coronavirus pandemic.

In this post, the second in a three-part series, we present results from the NAEP TEL and ICILS student questionnaires about students’ experience and confidence using technology (see the first post for more information about these assessments and their results). These results can help to inform education systems that are implementing remote learning activities this school year.

Uses of information and communication technologies (ICT) for school

Both NAEP TEL and ICILS collected data in 2018 on U.S. eighth-grade students’ uses of ICT in school or for school-related purposes.

According to the NAEP TEL questionnaire results, about one-third of U.S. eighth-grade students reported that they used ICT regularly (i.e., at least once a week) to create, edit, or organize digital media (figure 1). About a quarter used ICT regularly to create presentations, and 18 percent used ICT regularly to create spreadsheets.



According to the ICILS questionnaire results, 72 percent of U.S. eighth-grade students reported that they regularly used the Internet to do research, and 56 percent regularly used ICT to complete worksheets or exercises (figure 2). Forty percent of eighth-grade students regularly used ICT to organize their time and work. One-third regularly used software or applications to learn skills or a subject, and 30 percent regularly used ICT to work online with other students.



Confidence in using ICT

Both the 2018 NAEP TEL and ICILS questionnaires asked U.S. eighth-grade students about their confidence in their ICT skills. NAEP TEL found that about three-quarters of eighth-grade students reported that they were confident that they could—that is, they reported that they “probably can” or “definitely can”—compare products using the Internet or create presentations with sound, pictures, or video (figure 3). Seventy percent were confident that they could organize information into a chart, graph, or spreadsheet.



ICILS found that 86 percent of U.S. eighth-grade students reported that they knew how to search for and find relevant information for a school project on the Internet (figure 4). Eighty-three percent knew how to both upload text, images, or video to an online profile and install a program or app. About three-quarters of eighth-grade students knew how to change the settings on their devices, and 65 percent knew how to edit digital photographs or other graphic images.



Years of experience using computers

In the 2018 ICILS questionnaire, U.S. eighth-grade students were also asked how many years they had been using desktop or laptop computers. One-third of eighth-grade students reported using computers for 7 years or more—that is, they had been using computers since first grade (figure 5). This finding was similar to results from the Computer Access and Familiarity Study (CAFS), which was conducted as part of the 2015 NAEP. The CAFS found that in 2015, about 35 percent of eighth-grade public school students reported first using a laptop or desktop computer in kindergarten or before kindergarten.

Nineteen percent of eighth-grade students reported that they had used computers for at least 5 but less than 7 years. However, 9 percent of eighth-grade students had never used computers or had used them for less than one year, meaning they had only started using computers when they reached eighth grade.



Overall, responses of eighth-grade students showed that some had more years of experience using computers than others. Although there were differences in students’ use of ICT for school-related purposes, most students felt confident using ICT.

It should be noted that the data presented here were collected in 2018; any changes since then due to the coronavirus pandemic or other factors are not reflected in the results reported here. The NAEP TEL and ICILS samples both include public and private schools. Data reported in the text and figures are rounded to the nearest integer.

 

Resources for more information:

 

By Yan Wang, AIR, and Taslima Rahman, NCES

What National and International Assessments Can Tell Us About Technology in Students’ Learning: Eighth-Graders’ Readiness to Use Technology

Across the country in 2020, students, teachers, and parents have had to adapt to changes in the delivery of education instruction due to the coronavirus pandemic and turn to information and communication technologies (ICT) to learn, interact, and assess progress. It is important that we are able to assess students’ abilities to understand and use technology now more than ever. This post, the first in a three-part series, discusses the results of two technology-focused assessments, NAEP TEL and ICILS, which NCES administered in 2018 (see textbox for more information about these assessments).

Students’ performance on NAEP TEL and ICILS

According to the 2018 NAEP TEL, 46 percent of U.S. eighth-grade students scored at or above the NAEP Proficient level, meaning that they were able to demonstrate the selection and use of an appropriate range of tools and media (figure 1). According to the 2018 ICILS, 25 percent of eighth-grade students scored at or above proficiency level 3 for computer and information literacy—that is, they demonstrated the capacity to work independently when using computers as information-gathering and management tools. In addition, 20 percent of eighth-grade students scored in the upper region for computational thinking, meaning that they demonstrated an understanding of computation as a problem-solving framework.


Figure 1. Percentage of eighth-grade students identified as at or above proficient, by assessment: 2018


There are many factors that may affect performance on these assessments, such as access to technology, devices, hardware, and software; access to learning opportunities using technology; amount of experience using technology; and attitudes toward technology.

Student’s participation in technology-related classes

For NAEP TEL, students were asked if they were currently taking a technology-related class or if they had taken one in the past. In 2018, about 57 percent of U.S. eighth-grade students were either currently enrolled in or had taken at least one technology-related class, such as industrial technology, engineering, or a class that involved learning to use, program, or build computers (table 1). In addition, a higher percentage of students who had completed a technology-related class before or during eighth-grade scored at or above the NAEP Proficient level, compared with students who had not completed such a class (table 2).




Overall, both assessments show that a portion of students demonstrated the knowledge and skills necessary to meet the assessment’s defined proficiency measures, although more than half reported taking technology-related classes before or during eighth grade.

It should be noted that the data presented here were collected in 2018; any changes since then due to the coronavirus pandemic or other factors are not reflected in the results reported here. The NAEP TEL and ICILS samples both include public and private schools, but the two assessments use different methods for reporting student performance. Data from these assessments do not support any causal inferences as they are not experimental studies. Data reported in the text and figures are rounded to the nearest integer.

 

Resources for more information:

By Mary Ann Fox, AIR, and Taslima Rahman, NCES

Teaching with Technology: U.S. Teachers’ Perceptions and Use of Digital Technology in an International Context

The coronavirus pandemic forced teachers across the world to immediately transition instruction to a virtual setting in early 2020. To understand U.S. teachers’ level of preparedness for this shift in an international context, this blog examines recent international data from U.S. teachers’ responses to questions on the following topics:

  • Their perceptions of information and communications technologies (ICT) resources
  • Their use of ICT for instruction prior to the pandemic

In general, the results suggest that U.S. teachers are more resourced in ICT than their international peers, and they use ICT at a similar frequency at school when teaching.

 

Teachers’ perceptions of ICT resources at their school

The quantity and quality of ICT resources available in school systems prior to the coronavirus pandemic may impact teachers’ access to such resources for instructional purposes while classrooms are functioning in a virtual format. The United States participated in the 2018 International Computer and Information Literacy Study (ICILS), which asked questions about ICT resources to a nationally representative sample of eighth-grade teachers from 14 education systems.

The results from this study show that 86 percent of eighth-grade teachers both in the United States and across ICILS 2018 education systems “strongly agreed” or “agreed” that ICT is considered a priority for use in teaching (figure 1). Compared with the ICILS 2018
averages,[1] higher percentages of U.S. eighth-grade teachers “strongly agreed” or “agreed” with various statements about the use of ICT.

While 86 percent of U.S. eighth-grade teachers “strongly agreed” or “agreed” that “ICT is considered a priority for use in teaching,” only 61 percent “strongly agreed” or “agreed” that “there is sufficient opportunity for me to develop expertise in ICT” (figure 1). Additionally, 62 percent of U.S. eighth-grade teachers “strongly agreed” or “agreed” that “there is enough time to prepare lessons that incorporate ICT.” These disparities may have had an impact on teacher capacity during the sudden shift to 100 percent online learning as a result of the coronavirus pandemic, which would be a good topic for future research and analyses.  


Figure 1. Percentage of eighth-grade teachers who reported that they “strongly agree” or “agree” with statements about using ICT in teaching at school, by statement: 2018

p < .05. Significantly different from the U.S. estimate at the .05 level of statistical significance.

¹ Did not meet the guidelines for a sample participation rate of 85 percent and not included in the international average.
² National Defined Population covers 90 to 95 percent of National Target Population.
NOTE: ICT = information and communications technologies. The ICILS 2018 average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. Statements are ordered by the percentages of U.S. teachers reporting “strongly agree” or “agree” from largest to smallest.
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), The International Computer and Information Literacy Study (ICILS), 2018. Modified reproduction of figure 17 from U.S. Results from the 2018 ICILS Web Report.


Teachers’ perceptions of the use of ICT for instruction

Teachers’ views on the role of ICT in virtual instruction during the coronavirus pandemic are not yet clear. However, in 2018, when instruction was conducted in physical classrooms, most U.S. eighth-grade teachers participating in ICILS expressed positive perceptions about “using ICT in teaching and learning at school,” as did many teachers internationally.

Among eighth-grade teachers in the United States, 95 percent agreed that ICT “enables students to access better sources of information,” 92 percent agreed that ICT “helps students develop greater interest in learning,” and 92 percent agreed that ICT “helps students work at a level appropriate to their learning needs.” On average across other education systems participating in ICILS, at least 85 percent of teachers agreed with each of these statements (Fraillon et al. 2019).

Seventy-five percent of U.S. eighth-grade teachers in 2018 agreed that ICT “improves academic performance of students,” which was higher than the ICILS international average of 71 percent. The percentages of teachers who agreed with this statement varied across education systems, from three-quarters or more of teachers in Chile, Denmark, Kazakhstan, and Portugal to less than half of teachers in Finland and North Rhine-Westphalia (Germany).

 

Frequency of teachers’ use of ICT

Teachers’ reported use of ICT for instruction in physical classroom settings may provide insight into their level of experience as they transition to virtual settings during the coronavirus pandemic.

In 2018, half of U.S. eighth-grade teachers reported “using ICT at school when teaching” every day, which was not significantly different from the ICILS average of 48 percent. However, the U.S. percentage was lower than the percentages of teachers in Moscow (76 percent), Denmark (72 percent), and Finland (57 percent) (figure 2).


Figure 2. Percentage of eighth-grade teachers who reported using ICT at school every day when teaching, by education system: 2018

p < .05. Significantly different from the U.S. estimate at the .05 level of statistical significance.
¹ Met guidelines for sample participation rates only after replacement schools were included.
² National Defined Population covers 90 to 95 percent of National Target Population.
³ Did not meet the guidelines for a sample participation rate of 85 percent and not included in the international average.
⁴ Data collected at the beginning of the school year.
NOTE: ICT = information and communications technologies. The ICILS 2018 average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. Education systems are ordered by their percentages of teachers reporting using ICT at school when teaching from largest to smallest. Italics indicate the benchmarking participants.
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), The International Computer and Information Literacy Study (ICILS), 2018. Modified reproduction of figure 15 from U.S. Results from the 2018 ICILS Web Report.


For more information on teachers and technology, check out NCES’s ICILS 2018 website, the international ICILS website, and the earlier NCES blog “New Study on U.S. Eighth-Grade Students’ Computer Literacy.”

 

By Amy Rathbun, AIR, and Stephen Provasnik, NCES

 


[1] The ICILS average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. The United States did not meet the guidelines for a sample participation rate of 85 percent, so it is not included in the international average.

 

Reference

Fraillon, J., Ainley, J., Schulz, W., Friedman, T., and Duckworth, D. (2019). Preparing for Life in a Digital World: IEA International Computer and Information Literacy Study 2018 International Report. Amsterdam: International Association for the Evaluation of Educational Achievement.

New Study on U.S. Eighth-Grade Students’ Computer Literacy

In the 21st-century global economy, computer literacy and skills are an important part of an education that prepares students to compete in the workplace. The results of a recent assessment show us how U.S. students compare to some of their international peers in the areas of computer information literacy and computational thinking.

In 2018, the U.S. participated for the first time in the International Computer and Information Literacy Study (ICILS), along with 13 other education systems around the globe. The ICILS is a computer-based international assessment of eighth-grade students that measures outcomes in two domains: computer and information literacy (CIL)[1] and computational thinking (CT).[2] It compares U.S. students’ skills and experiences using technology to those of students in other education systems and provides information on teachers’ experiences, school resources, and other factors that may influence students’ CIL and CT skills.

ICILS is sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and is conducted in the United States by the National Center for Education Statistics (NCES).

The newly released U.S. Results from the 2018 International Computer and Information Literacy Study (ICILS) web report provides information on how U.S. students performed on the assessment compared with students in other education systems and describes students’ and teachers’ experiences with computers.


U.S. Students’ Performance

In 2018, U.S. eighth-grade students’ average score in CIL was higher than the average of participating education systems[3] (figure 1), while the U.S. average score in CT was not measurably different from the average of participating education systems.

 


Figure 1. Average computer and information literacy (CIL) scores of eighth-grade students, by education system: 2018p < .05. Significantly different from the U.S. estimate at the .05 level of statistical significance.

¹ Met guidelines for sample participation rates only after replacement schools were included.

² National Defined Population covers 90 to 95 percent of National Target Population.

³ Did not meet the guidelines for a sample participation rate of 85 percent and not included in the international average.

⁴ Nearly met guidelines for sample participation rates after replacement schools were included.

⁵ Data collected at the beginning of the school year.

NOTE: The ICILS computer and information literacy (CIL) scale ranges from 100 to 700. The ICILS 2018 average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. Education systems are ordered by their average CIL scores, from largest to smallest. Italics indicate the benchmarking participants.

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), the International Computer and Information Literacy Study (ICILS), 2018.


 

Given the importance of students’ home environments in developing CIL and CT skills (Fraillon et al. 2019), students were asked about how many computers (desktop or laptop) they had at home. In the United States, eighth-grade students with two or more computers at home performed better in both CIL and CT than their U.S. peers with fewer computers (figure 2). This pattern was also observed in all participating countries and education systems.

 


Figure 2. Average computational thinking (CT) scores of eighth-grade students, by student-reported number of computers at home and education system: 2018

p < .05. Significantly different from the U.S. estimate at the .05 level of statistical significance.

¹ Met guidelines for sample participation rates only after replacement schools were included.

² National Defined Population covers 90 to 95 percent of National Target Population.

³ Did not meet the guidelines for a sample participation rate of 85 percent and not included in the international average.

⁴ Nearly met guidelines for sample participation rates after replacement schools were included.

NOTE: The ICILS computational thinking (CT) scale ranges from 100 to 700. The number of computers at home includes desktop and laptop computers. Students with fewer than two computers include students reporting having “none” or “one” computer. Students with two or more computers include students reporting having “two” or “three or more” computers. The ICILS 2018 average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. Education systems are ordered by their average scores of students with two or more computers at home, from largest to smallest. Italics indicate the benchmarking participants.

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), the International Computer and Information Literacy Study (ICILS), 2018.


 

U.S. Students’ Technology Experiences

Among U.S. eighth-grade students, 72 percent reported using the Internet to do research in 2018, and 56 percent reported completing worksheets or exercises using information and communications technology (ICT)[4] every school day or at least once a week. Both of these percentages were higher than the respective ICILS averages (figure 3). The learning activities least frequently reported by U.S eighth-grade students were using coding software to complete assignments (15 percent) and making video or audio productions (13 percent).

 


Figure 3. Percentage of eighth-grade students who reported using information and communications technology (ICT) every school day or at least once a week, by activity: 2018

p < .05. Significantly different from the U.S. estimate at the .05 level of statistical significance.

¹ Did not meet the guidelines for a sample participation rate of 85 percent and not included in the international average.

NOTE: The ICILS 2018 average is the average of all participating education systems meeting international technical standards, with each education system weighted equally. Activities are ordered by the percentages of U.S. students reporting using information and communications technology (ICT) for the activities, from largest to smallest.

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), the International Computer and Information Literacy Study (ICILS), 2018.


 

Browse the full U.S. Results from the 2018 International Computer and Information Literacy Study (ICILS) web report to learn more about how U.S. students compare with their international peers in their computer literacy skills and experiences.

 

By Yan Wang, AIR, and Linda Hamilton, NCES

 

[1] CIL refers to “an individual's ability to use computers to investigate, create, and communicate in order to participate effectively at home, at school, in the workplace, and in society” (Fraillon et al. 2019).

[2] CT refers to “an individual’s ability to recognize aspects of real-world problems which are appropriate for computational formulation and to evaluate and develop algorithmic solutions to those problems so that the solutions could be operationalized with a computer” (Fraillon et al. 2019). CT was an optional component in 2018. Nine out of 14 ICILS countries participated in CT in 2018.

[3] U.S. results are not included in the ICILS international average because the U.S. school level response rate of 77 percent was below the international requirement for a participation rate of 85 percent.

[4] Information and communications technology (ICT) can refer to desktop computers, notebook or laptop computers, netbook computers, tablet devices, or smartphones (except when being used for talking and texting).

 

Reference

Fraillon, J., Ainley, J., Schulz, W., Duckworth, D., and Friedman, T. (2019). IEA International Computer and Information Literacy Study 2018: Assessment Framework. Cham, Switzerland: Springer. Retrieved October 7, 2019, from https://link.springer.com/book/10.1007%2F978-3-030-19389-8.