IES Blog

Institute of Education Sciences

You’ve Been Asked to Participate in a Study

Dear reader,

You’ve been asked to participate in a study.

. . . I know what you’re thinking. Oh, great. Another request for my time. I am already so busy.

Hmm, if I participate, what is my information going to be used for? Well, the letter says that collecting data from me will help researchers study education, and it says something else about how the information I provide would “inform education policy . . .”

But what does that mean?

If you’re a parent, student, teacher, school administrator, or district leader, you may have gotten a request like this from me or a colleague at the National Center for Education Statistics (NCES). NCES is one of 13 federal agencies that conducts survey and assessment research in order to help federal, state, and local policymakers better understand public needs and challenges. It is the U.S. Department of Education’s (ED’s) statistical agency and fulfills a congressional mandate to collect, collate, analyze, and report statistics on the condition of American education. The law also directs NCES to do the same about education across the globe.

But how does my participation in a study actually support the role Congress has given NCES?

Good question. When NCES conducts a study, participants are asked to provide information about themselves, their students or child/children, teachers, households, classrooms, schools, colleges, or other education providers. What exactly you will be asked about is based on many considerations, including previous research or policy needs. For example, maybe a current policy might be based on results from an earlier study, and we need to see if the results are still relevant. Maybe the topic has not been studied before and data are needed to determine policy options. In some cases, Congress has charged NCES with collecting data for them to better understand education in general.

Data collected from participants like you are combined so that research can be conducted at the group level. Individual information is not the focus of the research. Instead, NCES is interested in the experiences of groups of people or groups of institutions—like schools—based on the collected data. To protect respondents, personally identifiable information like your name (and other information that could identify you personally) is removed before data are analyzed and is never provided to others. This means that people who participate in NCES studies are grouped in different ways, such as by age or type of school attended, and their information is studied to identify patterns of experiences that people in these different groups may have had.

Let’s take a look at specific examples that show how data from NCES studies provide valuable information for policy decisions.

When policymakers are considering how data can inform policy—either in general or for a specific law under consideration—data from NCES studies play an important role. For example, policymakers concerned that students in their state/district/city often struggle to pay for college may be interested in this question:

“What can education data tell me about how to make college more affordable?”

Or policymakers further along in the law development process might have more specific ideas about how to help low-income students access college. They may have come across research linking programs such as dual enrollment—when high school students take college courses—to college access for underrepresented college students. An example of this research is provided in the What Works Clearinghouse (WWC) dual-enrollment report produced by ED’s Institute for Education Sciences (IES), which shows that dual-enrollment programs are effective at increasing students’ access to and enrollment in college and attainment of degrees. This was found to be the case especially for students typically underrepresented in higher education.   

Then, these policymakers might need more specific questions answered about these programs, such as:

What is the benefit of high school students from low-income households also taking college courses?”

Thanks to people who participate in NCES studies, we have the data to address such policy questions. Rigorous research using data from large datasets, compiled from many participants, can be used to identify differences in outcomes between groups. In the case of dual-enrollment programs, college outcomes for dual-enrollment participants from low-income households can be compared with those of dual-enrollment participants from higher-income households, and possible causes of those differences can be investigated.

The results of these investigations may then inform enactment of laws or creation of programs to support students. In the case of dual enrollment, grant programs might be set up at the state level for districts and schools to increase students’ local access to dual-enrollment credit earning.

This was very close to what happened in 2012, when I was asked by analysts in ED’s Office of Planning, Evaluation, and Policy Development to produce statistical tables with data on students’ access to career and technical education (CTE) programs. Research, as reviewed in the WWC dual-enrollment report, was already demonstrating the benefits of dual enrollment for high school students. Around 2012, ED was considering a policy that would fund the expansion of dual enrollment specifically for CTE. The reason I was asked to provide tables on the topic was my understanding of two important NCES studies, the Education Longitudinal Study of 2002 (ELS:2002) and the High School Longitudinal Study of 2009 (HSLS:09). Data provided by participants in those studies were ideal for studying the question. The tables were used to evaluate policy options. Based on the results, ED, through the President, made a budget request to Congress to support dual-enrollment policies. Ultimately, dual-enrollment programs were included in the Strengthening Career and Technical Education for the 21st Century Act (Perkins V).  

The infographic below shows that this scenario—in which NCES data provided by participants like you were used to provide information about policy—has happened on different scales for different policies many times over the past few decades. The examples included are just some of those from the NCES high school longitudinal studies. NCES data have been used countless times in its 154-year history to improve education for American students. Check out the full infographic (PDF) with other examples.


Excerpt of full infographic showing findings and actions for NCES studies on Equity, Dropout Prevention, and College and Career Readiness


However, it’s not always the case that a direct line can be drawn between data from NCES studies and any one policy. Research often informs policy indirectly by educating policymakers and the public they serve on critical topics. Sometimes, as in the dual-enrollment and CTE programs research question I investigated, it can take time before policy gets enacted or a new program rolls out. This does not lessen the importance of the research, nor the vital importance of the data participants provide that underpin it.

The examples in the infographic represent experiences of actual individuals who took the time to tell NCES about themselves by participating in a study.  

If you are asked to participate in an NCES study, please consider doing so. People like you, schools like yours, and households in your town do matter—and by participating, you are helping to inform decisions and improve education across the country.

 

By Elise Christopher, NCES

Working Toward a Successful National Data Collection: The ECLS Field Test

The National Center for Education Statistics (NCES) conducts some of the most complex education surveys in the world, and we work hard to make these surveys as effective and efficient as possible. One way we make sure our surveys are successful is by conducting multiple tests before we fully launch a national data collection.

Even prior to a field test, NCES develops survey materials and procedures using much smaller-scale cognitive laboratory testing and focus-group processes. These initial development procedures help ensure that materials are clear and procedures are understood before we conduct field testing with larger and more representative groups of respondents. Then, we launch the field tests to test data-collection operations and survey processes and procedures. Field tests are small-scale surveys that include a range of respondents and are designed to test the survey questionnaires and survey administration procedures in a real-world situation prior to the launch of a major study. The field test results allow us to make any necessary adjustments before starting the national data collection. Field tests also allow us to test specific survey items and ensure that they are valid and reliable. Without a field test, we could risk spending the public’s time and money on large data-collection efforts that do not produce the intended information.

NCES is about to begin the Early Childhood Longitudinal Study, Kindergarten Class of 2022–23 (ECLS-K:2023) with a field test early this year. The ECLS-K:2023 will focus on children’s early school experiences, beginning with preschool and continuing through fifth grade. From the spring of 2022 through the spring of 2028, we will collect national study data from children and their parents, teachers, and school administrators to answer questions about children’s early learning and development, transition into kindergarten and beyond, and experiences in the elementary grades. 

Although the ECLS-K:2023 will be similar in many ways to prior ECLS kindergarten studies, we are adding a round of data collection prior to the children’s kindergarten year—the national spring 2022 preschool round. For this preschool survey, we’ll send an invitation to participate to a sample of residential addresses within selected areas of the United States. Potential participants will first be asked to fill out a brief screener questionnaire. If they report that an ECLS-eligible child is in the household, they will be asked additional important questions about early childhood topics, such as their child’s literacy, language, math, and social skills; activities done with the child in the home (e.g., singing songs, playing games, reading); and characteristics of any early care and education (i.e., child care) arrangements for the child.   

Because the ECLS-K:2023 preschool data need to be comprehensive and reliable so that they can inform public discussions and policies related to early elementary education, it’s crucial that we test our procedures and questions for this new preschool round by conducting a field test in early 2020.  

If you receive a letter about participating in the 2020 ECLS field test, you’re being selected to represent thousands of households like yours and provide NCES with the data we need to make decisions about how to best conduct the ECLS-K:2023. The participation of all the selected households who receive our mailings, even those without children, is essential for a successful field test and, ultimately, a successful ECLS-K:2023.

If you are selected for the ECLS field test and have any questions about participating, please visit the participant information page

For more information on the ECLS-K:2023 or its 2020 field test, please email the ECLS study team.

For information about other ECLS program studies, please visit https://nces.ed.gov/ecls/.

 

By Jill Carlivati McCarroll

From Data Collection to Data Release: What Happens?

In today’s world, much scientific data is collected automatically from sensors and processed by computers in real time to produce instant analytic results. People grow accustomed to instant data and expect to get things quickly.

At the National Center for Education Statistics (NCES), we are frequently asked why, in a world of instant data, it takes so long to produce and publish data from surveys. Although improvements in the timeliness of federal data releases have been made, there are fundamental differences in the nature of data compiled by automated systems and specific data requested from federal survey respondents. Federal statistical surveys are designed to capture policy-related and research data from a range of targeted respondents across the country, who may not always be willing participants.

This blog is designed to provide a brief overview of the survey data processing framework, but it’s important to understand that the survey design phase is, in itself, a highly complex and technical process. In contrast to a management information system, in which an organization has complete control over data production processes, federal education surveys are designed to represent the entire country and require coordination with other federal, state, and local agencies. After the necessary coordination activities have been concluded, and the response periods for surveys have ended, much work remains to be done before the survey data can be released.

Survey Response

One of the first sources of potential delays is that some jurisdictions or individuals are unable to fill in their surveys on time. Unlike opinion polls and online quizzes, which use anyone who feels like responding to the survey (convenience samples), NCES surveys use rigorously formulated samples meant to properly represent specific populations, such as states or the nation as a whole. In order to ensure proper representation within the sample, NCES follows up with nonresponding sampled individuals, education institutions, school districts, and states to ensure the maximum possible survey participation within the sample. Some large jurisdictions, such as the New York City school district, also have their own extensive survey operations to conclude before they can provide information to NCES. Before the New York City school district, which is larger than about two-thirds of all state education systems, can respond to NCES surveys, it must first gather information from all its schools. Receipt of data from New York City and other large districts is essential to compiling nationally representative data.

Editing and Quality Reviews

Waiting for final survey responses does not mean that survey processing comes to a halt. One of the most important roles NCES plays in survey operations is editing and conducting quality reviews of incoming data, which take place on an ongoing basis. In these quality reviews, a variety of strategies are used to make cost-effective and time-sensitive edits to the incoming data. For example, in the Integrated Postsecondary Education Data System (IPEDS), individual higher education institutions upload their survey responses and receive real-time feedback on responses that are out of range compared to prior submissions or instances where survey responses do not align in a logical way. All NCES surveys use similar logic checks in addition to a range of other editing checks that are appropriate to the specific survey. These checks typically look for responses that are out of range for a certain type of respondent.

Although most checks are automated, some particularly complicated or large responses may require individual review. For IPEDS, the real-time feedback described above is followed by quality review checks that are done after collection of the full dataset. This can result in individualized follow up and review with institutions whose data still raise substantive questions. 

Sample Weighting

In order to lessen the burden on the public and reduce costs, NCES collects data from selected samples of the population rather than taking a full census of the entire population for every study. In all sample surveys, a range of additional analytic tasks must be completed before data can be released. One of the more complicated tasks is constructing weights based on the original sample design and survey responses so that the collected data can properly represent the nation and/or states, depending on the survey. These sample weights are designed so that analyses can be conducted across a range of demographic or geographic characteristics and properly reflect the experiences of individuals with those characteristics in the population.

If the survey response rate is too low, a “survey bias analysis” must be completed to ensure that the results will be sufficiently reliable for public use. For longitudinal surveys, such as the Early Childhood Longitudinal Study, multiple sets of weights must be constructed so that researchers using the data will be able to appropriately account for respondents who answered some but not all of the survey waves.

NCES surveys also include “constructed variables” to facilitate more convenient and systematic use of the survey data. Examples of constructed variables include socioeconomic status or family type. Other types of survey data also require special analytic considerations before they can be released. Student assessment data, such as the National Assessment of Educational Progress (NAEP), require that a number of highly complex processes be completed to ensure proper estimations for the various populations being represented in the results. For example, just the standardized scoring of multiple choice and open-ended items can take thousands of hours of design and analysis work.

Privacy Protection

Release of data by NCES carries a legal requirement to protect the privacy of our nation’s children. Each NCES public-use dataset undergoes a thorough evaluation to ensure that it cannot be used to identify responses of individuals, whether they are students, parents, teachers, or principals. The datasets must be protected through item suppression, statistical swapping, or other techniques to ensure that multiple datasets cannot be combined in such a way as to identify any individual. This is a time-consuming process, but it is incredibly important to protect the privacy of respondents.

Data and Report Release

When the final data have been received and edited, the necessary variables have been constructed, and the privacy protections have been implemented, there is still more that must be done to release the data. The data must be put in appropriate formats with the necessary documentation for data users. NCES reports with basic analyses or tabulations of the data must be prepared. These products are independently reviewed within the NCES Chief Statistician’s office.

Depending on the nature of the report, the Institute of Education Sciences Standards and Review Office may conduct an additional review. After all internal reviews have been conducted, revisions have been made, and the final survey products have been approved, the U.S. Secretary of Education’s office is notified 2 weeks in advance of the pending release. During this notification period, appropriate press release materials and social media announcements are finalized.

Although NCES can expedite some product releases, the work of preparing survey data for release often takes a year or more. NCES strives to maintain a balance between timeliness and providing the reliable high-quality information that is expected of a federal statistical agency while also protecting the privacy of our respondents.  

 

By Thomas Snyder

Data Tools for College Professors and Students

Ever wonder what parts of the country produce the most English majors? Want to know which school districts have the most guidance counselors? The National Center for Education Statistics (NCES) has all the tools you need to dig into these and lots of other data!

Whether you’re a student embarking on a research project or a college professor looking for a large data set to use for an assignment, NCES has you covered. Below, check out the tools you can use to conduct searches, download datasets, and generate your own statistical tables and analyses.

 

Conduct Publication Searches

Two search tools help researchers identify potential data sources for their study and explore prior research conducted with NCES data. The Publications & Products Search Tool can be used to search for NCES publications and data products. The Bibliography Search Tool, which is updated continually, allows users to search for individual citations from journal articles that have been published using data from most surveys conducted by NCES.

Key reference publications include the Digest of Education Statistics, which is a comprehensive library of statistical tabulations, and The Condition of Education, which highlights up-to-date trends in education through statistical indicators.

 

Learn with Instructional Modules

The Distance Learning Dataset Training System (DLDT) is an interactive online tool that allows users to learn about NCES data across the education spectrum. DLDT’s computer-based training introduces users to many NCES datasets, explains their designs, and offers technical considerations to facilitate successful analyses. Please see the NCES blog Learning to Use the Data: Online Dataset Training Modules for more details about the DLDT tool.
 




Download and Access Raw Data Files

Users have several options for conducting statistical analyses and producing data tables. Many NCES surveys release public-use raw data files that professors and students can download and analyze using statistical software packages like SAS, STATA, and SPSS. Some data files and syntax files can also be downloaded using NCES data tools:

  • Education Data Analysis Tool (EDAT) and the Online Codebook allow users to download several survey datasets in various statistical software formats. Users can subset a dataset by selecting a survey, a population, and variables relevant to their analysis.
  • Many data files can be accessed directly from the Surveys & Programs page by clicking on the specific survey and then clicking on the “Data Products” link on the survey website.

 

Generate Analyses and Tables

NCES provides several online analysis tools that do not require a statistical software package:

  • DataLab is a tool for making tables and regressions that features more than 30 federal education datasets. It includes three powerful analytic tools:
    • QuickStats—for creating simple tables and charts.
    • PowerStats—for creating complex tables and logistic and linear regressions.
    • TrendStats—for creating complex tables spanning multiple data collection years. This tool also contains the Tables Library, which houses more than 5,000 published analysis tables by topic, publication, and source.



  • National Assessment of Educational Progress (NAEP) Data Explorer can be used to generate tables, charts, and maps of detailed results from national and state assessments. Users can identify the subject area, grade level, and years of interest and then select variables from the student, teacher, and school questionnaires for analysis.
  • International Data Explorer (IDE) is an interactive tool with data from international assessments and surveys, such as the Program for International Student Assessment (PISA), the Program for the International Assessment of Adult Competencies (PIAAC), and the Trends in International Mathematics and Science Study (TIMSS). The IDE can be used to explore student and adult performance on assessments, create a variety of data visualizations, and run statistical tests and regression analyses.
  • Elementary/Secondary Information System (ElSi) allows users to quickly view public and private school data and create custom tables and charts using data from the Common Core of Data (CCD) and Private School Universe Survey (PSS).
  • Integrated Postsecondary Education Data System (IPEDS) Use the Data provides researcher-focused access to IPEDS data and tools that contain comprehensive data on postsecondary institutions. Users can view video tutorials or use data through one of the many functions within the portal, including the following:
    • Data Trends—Provides trends over time for high-interest topics, including enrollment, graduation rates, and financial aid.
    • Look Up an Institution—Allows for quick access to an institution’s comprehensive profile. Shows data similar to College Navigator but contains additional IPEDS metrics.
    • Statistical Tables—Equips power users to quickly get data and statistics for specific measures, such as average graduation rates by state.

 

 

New Data Available on Prevalence of Recognized Student Acceptance Groups

Schools can help to foster students’ understanding of diversity and create environments where students feel safe and welcome. One way to do this is by organizing student groups whose purpose is to promote acceptance of other students. The School Survey on Crime and Safety (SSOCS) collected data on the presence of recognized student acceptance groups during the 2015–16 school year from a nationally representative sample of 3,500 K–12 public schools. The questionnaire asked whether schools had student groups that promote acceptance of students’ sexual orientations[i] and gender identities,[ii] of students with disabilities, and of cultural diversity.

Among all public schools, groups that promote acceptance of students with disabilities were most common. Some 27 percent of schools reported having a student group that promotes acceptance of students with disabilities during the 2015­–16 school year, compared with 21 percent that reported having a student group that promotes acceptance of cultural diversity and 12 percent that reported having a student group that promotes acceptance of students’ sexual orientations and gender identities.


Figure 1. Percentage of public schools reporting the presence of recognized student acceptance groups, by school level and purpose of student group: School year 2015–16

1Primary schools are defined as schools in which the lowest grade is not higher than grade 3 and the highest grade is not higher than grade 8. Middle schools are defined as schools in which the lowest grade is not lower than grade 4 and the highest grade is not higher than grade 9. High schools are defined as schools in which the lowest grade is not lower than grade 9 and the highest grade is not higher than grade 12. The total also includes combined schools. Combined schools are schools that have a combination of grades that cannot be categorized as primary, middle, or high schools, including K–12 schools. 
2Sexual orientation was defined for respondents as one’s emotional or physical attraction to the same and/or opposite sex. Gender identity was defined for respondents as one’s inner sense of one’s own gender, which may or may not match the sex assigned at birth. Different people choose to express their gender identity differently. For some, gender may be expressed through, for example, dress, grooming, mannerisms, speech patterns, and social interactions. Gender expression usually ranges between masculine and feminine, and some transgender people express their gender consistent with how they identify internally, rather than in accordance with the sex they were assigned at birth. An example of a student group to promote acceptance of students' sexual orientations and gender identities provided to respondents was a Gay-Straight Alliance.
3An example of a student group to promote acceptance of students with disabilities provided to respondents was Best Buddies.
4An example of a student group to promote acceptance of cultural diversity provided to respondents was a Cultural Awareness Club.
NOTE: Responses were provided by the principal or the person most knowledgeable about school crime and policies to provide a safe environment. Although rounded numbers are displayed, the figures are based on unrounded data. 
SOURCE: U.S. Department of Education, National Center for Education Statistics (NCES), 2015–16 School Survey on Crime and Safety (SSOCS), 2016. See table 37.


All three types of acceptance groups were more common in high schools than in middle or primary schools. The most common type of student acceptance group varied by school level. For example, among middle schools, the percentage of schools that reported having a student group that promotes acceptance of students with disabilities (32 percent) was higher than the percentage that reported having a student group that promotes acceptance of cultural diversity (22 percent) or acceptance of students’ sexual orientations and gender identities (12 percent). This pattern was similar for primary schools. Among high schools, the percentage of schools that reported having a student group that promotes acceptance of students’ sexual orientations and gender identities (50 percent) was higher than the percentage that reported having a student group that promotes acceptance of students with disabilities (45 percent).

During the 2015–16 school year, all three types of student acceptance groups were more commonly found in schools in cities and suburbs than in schools in rural areas. For example, 30 percent of schools in cities and 26 percent of schools in suburbs reported having a student group that promotes acceptance of cultural diversity, compared with 10 percent of schools in rural areas.


Figure 2. Percentage of public schools reporting the presence of recognized student acceptance groups, by purpose of student group and school locale: School year 2015–16

1Sexual orientation was defined for respondents as one's emotional or physical attraction to the same and/or opposite sex. Gender identity was defined for respondents as one's inner sense of one's own gender, which may or may not match the sex assigned at birth. Different people choose to express their gender identity differently. For some, gender may be expressed through, for example, dress, grooming, mannerisms, speech patterns, and social interactions. Gender expression usually ranges between masculine and feminine, and some transgender people express their gender consistent with how they identify internally, rather than in accordance with the sex they were assigned at birth. An example of a student group to promote acceptance of students' sexual orientations and gender identities provided to respondents was a Gay-Straight Alliance.
2An example of a student group to promote acceptance of students with disabilities provided to respondents was Best Buddies.
3An example of a student group to promote acceptance of cultural diversity provided to respondents was a Cultural Awareness Club.
NOTE: Responses were provided by the principal or the person most knowledgeable about school crime and policies to provide a safe environment.
SOURCE: U.S. Department of Education, National Center for Education Statistics (NCES), 2015–16 School Survey on Crime and Safety (SSOCS), 2016. See table 37.


You can find more information on school crime and safety in NCES publications, including Crime, Violence, Discipline, and Safety in U.S. Public Schools: Findings From the School Survey on Crime and Safety: 2015–16 and the 2017 Indicators of School Crime and Safety.

 

By Rachel Hansen, NCES, and Melissa Diliberti, AIR


[i]Sexual orientation was defined for respondents as one’s emotional or physical attraction to the same and/or opposite sex.

[ii]Gender identity was defined for respondents as one’s inner sense of one’s own gender, which may or may not match the sex assigned at birth. Different people choose to express their gender identity differently. For some, gender may be expressed through, for example, dress, grooming, mannerisms, speech patterns, and social interactions. Gender expression usually ranges between masculine and feminine, and some transgender people express their gender consistent with how they identify internally, rather than in accordance with the sex they were assigned at birth.