NCES Blog

National Center for Education Statistics

From Data Collection to Data Release: What Happens?

In today’s world, much scientific data is collected automatically from sensors and processed by computers in real time to produce instant analytic results. People grow accustomed to instant data and expect to get things quickly.

At the National Center for Education Statistics (NCES), we are frequently asked why, in a world of instant data, it takes so long to produce and publish data from surveys. Although improvements in the timeliness of federal data releases have been made, there are fundamental differences in the nature of data compiled by automated systems and specific data requested from federal survey respondents. Federal statistical surveys are designed to capture policy-related and research data from a range of targeted respondents across the country, who may not always be willing participants.

This blog is designed to provide a brief overview of the survey data processing framework, but it’s important to understand that the survey design phase is, in itself, a highly complex and technical process. In contrast to a management information system, in which an organization has complete control over data production processes, federal education surveys are designed to represent the entire country and require coordination with other federal, state, and local agencies. After the necessary coordination activities have been concluded, and the response periods for surveys have ended, much work remains to be done before the survey data can be released.

Survey Response

One of the first sources of potential delays is that some jurisdictions or individuals are unable to fill in their surveys on time. Unlike opinion polls and online quizzes, which use anyone who feels like responding to the survey (convenience samples), NCES surveys use rigorously formulated samples meant to properly represent specific populations, such as states or the nation as a whole. In order to ensure proper representation within the sample, NCES follows up with nonresponding sampled individuals, education institutions, school districts, and states to ensure the maximum possible survey participation within the sample. Some large jurisdictions, such as the New York City school district, also have their own extensive survey operations to conclude before they can provide information to NCES. Before the New York City school district, which is larger than about two-thirds of all state education systems, can respond to NCES surveys, it must first gather information from all its schools. Receipt of data from New York City and other large districts is essential to compiling nationally representative data.

Editing and Quality Reviews

Waiting for final survey responses does not mean that survey processing comes to a halt. One of the most important roles NCES plays in survey operations is editing and conducting quality reviews of incoming data, which take place on an ongoing basis. In these quality reviews, a variety of strategies are used to make cost-effective and time-sensitive edits to the incoming data. For example, in the Integrated Postsecondary Education Data System (IPEDS), individual higher education institutions upload their survey responses and receive real-time feedback on responses that are out of range compared to prior submissions or instances where survey responses do not align in a logical way. All NCES surveys use similar logic checks in addition to a range of other editing checks that are appropriate to the specific survey. These checks typically look for responses that are out of range for a certain type of respondent.

Although most checks are automated, some particularly complicated or large responses may require individual review. For IPEDS, the real-time feedback described above is followed by quality review checks that are done after collection of the full dataset. This can result in individualized follow up and review with institutions whose data still raise substantive questions. 

Sample Weighting

In order to lessen the burden on the public and reduce costs, NCES collects data from selected samples of the population rather than taking a full census of the entire population for every study. In all sample surveys, a range of additional analytic tasks must be completed before data can be released. One of the more complicated tasks is constructing weights based on the original sample design and survey responses so that the collected data can properly represent the nation and/or states, depending on the survey. These sample weights are designed so that analyses can be conducted across a range of demographic or geographic characteristics and properly reflect the experiences of individuals with those characteristics in the population.

If the survey response rate is too low, a “survey bias analysis” must be completed to ensure that the results will be sufficiently reliable for public use. For longitudinal surveys, such as the Early Childhood Longitudinal Study, multiple sets of weights must be constructed so that researchers using the data will be able to appropriately account for respondents who answered some but not all of the survey waves.

NCES surveys also include “constructed variables” to facilitate more convenient and systematic use of the survey data. Examples of constructed variables include socioeconomic status or family type. Other types of survey data also require special analytic considerations before they can be released. Student assessment data, such as the National Assessment of Educational Progress (NAEP), require that a number of highly complex processes be completed to ensure proper estimations for the various populations being represented in the results. For example, just the standardized scoring of multiple choice and open-ended items can take thousands of hours of design and analysis work.

Privacy Protection

Release of data by NCES carries a legal requirement to protect the privacy of our nation’s children. Each NCES public-use dataset undergoes a thorough evaluation to ensure that it cannot be used to identify responses of individuals, whether they are students, parents, teachers, or principals. The datasets must be protected through item suppression, statistical swapping, or other techniques to ensure that multiple datasets cannot be combined in such a way as to identify any individual. This is a time-consuming process, but it is incredibly important to protect the privacy of respondents.

Data and Report Release

When the final data have been received and edited, the necessary variables have been constructed, and the privacy protections have been implemented, there is still more that must be done to release the data. The data must be put in appropriate formats with the necessary documentation for data users. NCES reports with basic analyses or tabulations of the data must be prepared. These products are independently reviewed within the NCES Chief Statistician’s office.

Depending on the nature of the report, the Institute of Education Sciences Standards and Review Office may conduct an additional review. After all internal reviews have been conducted, revisions have been made, and the final survey products have been approved, the U.S. Secretary of Education’s office is notified 2 weeks in advance of the pending release. During this notification period, appropriate press release materials and social media announcements are finalized.

Although NCES can expedite some product releases, the work of preparing survey data for release often takes a year or more. NCES strives to maintain a balance between timeliness and providing the reliable high-quality information that is expected of a federal statistical agency while also protecting the privacy of our respondents.  

 

By Thomas Snyder

Now Available! New Nationally Representative Data on the Socioemotional Development of Elementary School Students

In an earlier blog post, we shared that one of our survey programs—the Early Childhood Longitudinal Study (ECLS) program—was collecting data on socioemotional development to better understand how different academic and nonacademic factors may influence a child’s early schooling experiences. New data are now available from the spring 2016 public-use dataset for the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011). This file contains data from every round of the ECLS-K:2011, from kindergarten through fifth grade.

For decades, the National Center for Education Statistics and other researchers have used ECLS data to examine questions about elementary school students’ socioemotional development. For instance, as seen in the excerpt below, an earlier wave of data was used to develop an indicator in the America’s Children report that looks at first-time kindergartners’ scores on socioemotional scales and how these students may victimize their peers. ECLS data are rich with information that can be used to analyze the influence of family, school, community, and individual factors on students’ development, early learning, and performance in school.

In the most recent ECLS program study, the ECLS-K:2011 collected information on its sample of kindergartners during the 2010–11 school year and then at least once during every academic year thereafter until 2015–16, when most of the students were in fifth grade. The ECLS-K:2011 data allow researchers to study how students’ socioemotional skills develop over time through reports from the students themselves and from key people in those students’ lives, including their parents, before- and after-school care providers, teachers, and school administrators.

Here’s a peek into the socioemotional development measures included the ECLS-K:2011:

  • Students completed questionnaires about their relationships with peers, social distress, peer victimization, and satisfaction with different aspects of their lives.
  • Teachers used their experiences with students in their classrooms to provide information about students’ approaches to learning (e.g., eagerness to learn, self-direction, attentiveness), social skills, and problem behaviors, as well as their own closeness and conflict with students.
  • Parents provided separate reports on much of the same information reported by teachers to provide a richer picture of their child’s development through a different lens.

For more information on the measures of socioemotional development included in the ECLS-K:2011, please see our study instruments or email the ECLS study team. Also, keep an eye out for future online training modules for the ECLS-K:2011, which will be released in fall 2019 or early 2020. To be alerted about the release of the free online trainings, email the ECLS study team at ECLS@ed.gov and ask to be added to the ECLS listserv.

 


Excerpt from America’s Children: Key National Indicators of Well-Being 2017


 

By Jill Carlivati McCarroll and Gail M. Mulligan

 

What is the Forum on Child and Family Statistics?

By Grace Kena

The Federal Interagency Forum on Child and Family Statistics, is a working group of Federal agencies that collect, analyze, and report data on issues related to the well-being of children and their families. The Forum on Child and Family Statistics’ mission is to promote coordination and collaboration among member agencies and to improve efforts to collect and report Federal data on children and families. This forum is unique in that it compiles key findings across many domains of children’s lives. 

The National Center for Education Statistics (NCES) has been involved with the Forum on Child and Family Statistics since the early stages of its development. Founded in 1994, the Forum on Child and Family Statistics was formally established by Executive Order No. 13045 in 1997. The Forum’s main activity is to produce the report, America's Children: Key National Indicators of Well-Being, which is a collection of national indicators of child well-being. Through the report, the Forum aims to improve the reporting of Federal data on children and families; make these data available in an easy-to-use, nontechnical format; and stimulate discussions among policymakers and the public, and between the statistical and policy communities.

Using Federal data, the America’s Children series presents a set of key indicators on aspects of children’s lives that measure their well-being and influence the likelihood that a child will become a well-educated, economically secure, productive, and healthy adult. While there are many, interrelated aspects of children’s well-being, America’s Children reports on seven major domains:  family and social environment, economic circumstances, health care, physical environment and safety, behavior, education, and health. Currently, 23 agencies contribute to the report, including NCES, the Environmental Protection Agency, the Bureau of Justice Statistics, the Bureau of Labor Statistics, the Economic Research Service, the U. S. Census Bureau, and the National Center for Health Statistics and the Substance Abuse and Mental Health Services Administration. 

The Forum on Child and Family Statistics has published the America’s Children report since 1997. Beginning in 2004, the Forum started producing a brief report, America’s Children in Brief: Key National Indicators of Well-Being in even-numbered years; the full report is still published in odd years. Although this shortened version of the report focuses on selected indicators, data for all indicators are updated on the website each year. In 2014, the Forum published a one-time, special issue report titled America’s Young Adults. In addition to producing reports, the Forum collaborates with partner and other organizations on a number of research projects and in supporting conferences, workshops, and policy seminars. Most recently, NCES experts participated in a day-long workshop on Measuring and Reporting Social-Emotional Development in Early Childhood. NCES experts also authored the 2013 special feature on the academic knowledge and skills of kindergarten students using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K: 2011).
 
The 2015 America’s Children report shows several improvements in children’s well-being. The number of babies born prematurely has continued to decline, and recently, the percentage of children with asthma has decreased. High school completion rates have increased, particularly for Hispanic students. On the other hand, some aspects have not improved. The percentage of children experiencing a major depressive episode has continued to increase over the past several years. 

This year’s report also contains a special feature on health care quality, which provides information on well-child and well-adolescent visits, preschool vision screenings, asthma management plans, and access to care.

Learn more about the Forum on Child and Family Statistics and its activities, and the 2015 America’s Children report at the website. Also, tune in to a recent podcast describing findings from the latest report.

Does the Department of Education collect information on young children’s social and emotional development?

By Jill Carlivati McCarroll and Gail M. Mulligan

Yes, we do! During their early years, children are developing socially and emotionally. This includes the development of social skills, relationships, and regulation of emotions. Children’s socioemotional development can affect school experiences and outcomes, so it makes sense that the Department of Education is interested in this topic.

Researchers are using NCES’s Early Childhood Longitudinal Studies (ECLS) to examine questions about socioemotional development, for instance, how children’s growth in this area is related to background characteristics such as race/ethnicity and parents’ educational attainment, as well as home and school experiences. The ECLS studies collect information from the children themselves, as well as from their parents, their care providers, and their teachers. Being longitudinal, the ECLS data allow researchers to study how children’s socioemotional skills develop over time. Additionally, these surveys are some of the only nationally representative studies with data on children in these age groups.

The Early Childhood Longitudinal Study, Birth Cohort (ECLS-B) followed a group of children born in 2001 until they entered kindergarten. The ECLS-B was designed to describe children’s earliest experiences and relationships, and the first home visit data collection occurred when the children were only 9 months old. Socioemotional development was measured in several ways in this study. During home visits, researchers observed the children’s interactions with a parent during specific tasks, such as while the parent was reading a book aloud to the child, and reported on the children’s attentiveness, interest, affect, and social engagement. The quality of the children’s attachment relationship to their parent was also measured at age 2. When the children were in kindergarten, their teachers provided information about the children’s socioemotional skills. 

Socioemotional development has also been measured in ECLS studies that have followed groups of kindergartners over time: the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) and the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011). Teachers provided information about children’s social skills, problem behaviors, learning behaviors, and their own closeness and conflict with the students. Parents provided their own reports on much of the same information. One analysis of data from the teacher reports shows that children who enter kindergarten on time and those who had a delayed entry show positive “approaches to learning” (for example, eagerness to learn, self-direction, and attentiveness) more often than children who repeat kindergarten.

In later rounds of the ECLS-K and ECLS-K:2011, when the children were older, they were asked to provide information about themselves. In the ongoing ECLS-K:2011, children are reporting on aspects of socioemotional development such as their relationships with peers, social distress, peer victimization, and their satisfaction with different aspects of their lives.

For more information on the measures of socioemotional development included in the ECLS studies, please see our homepage, review our online training modules for the ECLS-B and ECLS-K, or email the ECLS study team.