IES Blog

Institute of Education Sciences

Taking Education Research Out of the Lab and Into the Real World

The Institute of Education Sciences is committed to supporting research that develops and tests solutions to the challenges facing education in the United States and working to share the results of that research with a broad audience of policymakers and practitioners. In this blog post, Katherine Pears (pictured right), a principal investigator from the Oregon Social Learning Center, shares how an IES-funded grant supported not only the development and testing of an intervention, but its implementation in classrooms.

Sometimes when I tell people that I am a research scientist, they say "Ok, but what do you do?" What they are really asking is whether the work I am doing is helping students and families in actual schools and community. It’s a good question and the short answer is “Yes!” but moving programs from the research lab to children, families and schools takes time and funding. Here is how it worked with my program.

I wanted to help children at risk for school failure to start kindergarten with skills to help them to do better. I teamed up with Dr. Phil Fisher and others, and we built on years of research and program development at the Oregon Social Learning Center (OSLC) to create a program called Kids in Transition to School (KITS). This is a summer program that continues into the first few weeks of school and focuses not only on letters and numbers, but also on teaching children how to get along with others and how to learn (such as focusing their attention). We also train the KITS teachers to use positive teaching strategies and help parents learn the same techniques to increase the chances that children will succeed.

However, before a school will put a program like KITS in place, they need to have some proof that it will actually work. That is why we sought external funding to evaluate the program.  With funding from the Institute of Education Sciences (IES), as well as the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) and the National Institute on Drug Abuse (NIDA), we were able to test KITS with three different groups of children at high risk for difficulties in school: children in foster care, children with developmental disabilities and behavior problems, and children from low-income neighborhoods. 

Across these studies, KITS led to positive outcomes for children and the parents. Funding from IES helped us show that the KITS Program worked with different groups of children and that it was a good candidate for use in school settings. In many cases, this is where the process of moving programs from research to practice can get bogged down due to a gap between researchers and the people in the education systems that could use the programs. IES is strongly committed to bridging that gap by supporting partnerships between researchers and practitioners. In our IES-funded study, we partnered with two of our local school districts and United Way of Lane County (UWLC). Working with the schools and community agencies helped us to both test KITS and develop plans for how the districts could keep the program going after the funding for the research study ended.

Our partnership with UWLC also enabled us to start a new project to bring KITS to more schools. With funding from the Corporation for National and Community Service (CNCS) Social Innovation Fund, UWLC offered grants to education foundations in Lane County, Oregon to implement KITS in the local school districts. The Social Innovation Fund allows promising programs to be brought to scale (i.e., made available to large numbers of people) by providing funding for community agencies to start and develop plans for sustaining these programs. In 2015, the KITS Program had about 30 educators serving about 120 children and families in 6 school districts in Lane County, Oregon. By summer 2017, two years into the grant, the KITS Program had trained 125 educators to serve approximately 435 children and families in 13 school districts in the county. That’s more than a threefold increase in the number of children and families served, as well as the number of teachers trained in the KITS Program.

We will continue to work with partners to evaluate the effects of the KITS program and make it available to more school districts. However, without the funding from IES and other federal agencies, we would not have been able to both test the program and partner with schools to make KITS a part of their everyday practice. 

 

IES-supported Technology to be Used in Hundreds of Schools

A technology-based instructional tool—developed and evaluated through IES funding—will now be put it to use in hundreds of schools across the country with the goal of improving students’ literacy outcomes. The United2Read Project was recently awarded a five-year Education Innovation and Research (EIR) expansion grant from the U.S. Department of Education. The EIR grants provide funding to develop, expand, and evaluate innovative, evidence-based programs designed to improve student achievement.

A2i (pictured right) includes a series of assessments to measure component literacy skills of students and provide information that teachers can use to individualize literacy instruction in Kindergarten through Grade 3.  The assessments cover a wide range of literacy skills, including vocabulary, decoding, word reading, spelling, sentence and paragraph writing, comprehension, and inferencing.

A2i assessments are given throughout the school year to monitor student progress, and A2i’s algorithms are updated in real-time to provide teachers with recommendations for instruction for each student and changes to grouping.

This project demonstrates the critical role IES plays in supporting research to develop innovative teaching and learning products and test those products for efficacy.  Over the past 13 years, A2i was developed, evaluated, and scaled with a progression of awards from IES, as well as the National Institutes of Health/National Institute of Child Health and Human Development (NIH/NICHD).

  • With a 2004 IES research grant, researchers at the University of Michigan and Florida State University conducted basic research to understand how the effects of different types of literacy instruction (e.g., phonics vs. meaning focused) depend on children’s constellation of language, phonological awareness, decoding and encoding, and comprehension skills. The team then used the findings from this initial work to build and test an initial version of A2i with first-grade students. The research demonstrated that instruction tailored to students’ skills and learning needs, and adjusted over time, is more effective than one-size-fits-all approaches.
  • With a 2007 IES research grant, the researchers expanded A2i into second- and third-grade classrooms. They also tested the effects of implementing the system with second-grade students who had received A2i-informed instruction in Grade 1, as well as those who had not.
  • With a 2013 IES research grant and grant support from NIH/NICHD, researchers at Florida State and Arizona State conducted seven randomized controlled studies examining the impact of A2i on student reading. The results suggest that individualizing literacy instruction with A2i recommendations led to stronger literacy gains for K-3 students. On average, students who used A2i across multiple years ended third grade reading at a grade 5 level. The research included students who qualified for the National School Lunch Program and who received special education services.
  • With a 2013 IES research grant to Arizona State University (which was later transferred in 2016 to University of California, Irvine) and a 2014 award from the ED/IES Small Business Innovation Research (SBIR) Program to Learning Ovations, an education technology development firm, the researchers upgraded the underlying data architecture of A2i to enable scale and implementation in classrooms across the nation. Research results demonstrated that the more teachers viewed student test scores, the greater the students’ literacy skill gains.
     

(Findings from all publications on the A2i software and related professional development are posted on Learning Ovations website.) 

The new EIR expansion grant was awarded to a consortium of researchers and developers, including researchers at UC Irvine, Learning Ovations, Digital Promise (a non-profit), and MDRC, a national evaluation firm. At least 300 schools and 100,000 students will be served through the grant, which will also support a large-scale effectiveness trial to measure the impact of the project on student reading achievement.

Ed Metz is a Research Scientist at IES, where he leads the SBIR and the Education Technology Research Grants programs.

Erin Higgins is an Education Research Analyst at IES, where she leads the Cognition and Student Learning Research Grants program.

Data on New Topics in the School Survey on Crime and Safety Shed Light on Emerging Areas of Interest

By Rachel Hansen, NCES; and Melissa Diliberti and Jana Kemp, AIR

For more than 15 years, the National Center for Education Statistics has administered the School Survey on Crime and Safety (SSOCS) to provide timely, high-quality data on crime and safety in U.S. public schools. Information collected on SSOCS includes the frequency and nature of crime; disciplinary actions; crime prevention and the involvement of law enforcement; and challenges to reducing and preventing crime. Conducted with a nationally representative sample of public schools, the sixth, and most recent, administration of SSOCS took place during the 2015–16 school year. The first report highlighting key findings from that survey was released in 2017.

For the 2015–16 survey, we included new and expanded questions on several topics to address emerging policy issues and to identify common practices in school safety, including:

  • School law enforcement, including questions on how schools involve sworn law enforcement officers in daily activities and whether schools outline the responsibilities of these officers at school. For instance, one new item asks whether law enforcement officers routinely wear a body camera, while another item asks if the school has a formalized policy defining officers’ use of firearms while at school;
  • Preventative measures used in public schools, including new questions on more recent security practices. For example, one new item asks schools to report whether they have a threat assessment team to identity students who might be a potential risk for violent or harmful behavior;
  • Preparations for crisis situations, such as whether schools drill students on the use of evacuation, lockdown, and shelter-in-place procedures. Other new items ask whether schools have panic buttons that directly connect to law enforcement and whether they have classroom doors that can be locked from the inside;
  • Student involvement in crime prevention, such as whether schools use peer mediation, student court, restorative circles, or social emotional learning training for students as part of a formal program intended to prevent or reduce violence; and
  • Staff training in discipline policies and practices, including those related to bullying and cyberbullying or strategies for students displaying signs of mental health disorders.

While previous administrations of SSOCS have asked schools to report the number of hate crimes that occurred during a given school year, the 2015–16 questionnaire asked schools to also report the bias (e.g., national origin or ethnicity, gender identity, etc.) that may have motivated these hate crimes. For the first time, the SSOCS questionnaire also asked schools to report the number of arrests that occurred at school.

In addition to these new and expanded questions, SSOCS continues to collect detailed information on schools’ safety practices, the number and type of crime incidents (e.g., sexual assault, physical attack or fight) that occur at school, and the extent to which schools involve law enforcement, parents, and other community groups in their efforts to reduce and prevent crime. To allow for trend comparisons, many items included on SSOCS questionnaires have remained consistent between survey administrations.

Due to the sensitive nature of SSOCS data, researchers must apply for a restricted-use license to access the SSOCS:2016 restricted-use data file. A public-use data file, with some variables removed, was released in March of 2018. Public-use data files from previous SSOCS administrations are also available on the SSOCS website and in DataLab

 

What is the difference between the ACGR and the AFGR?

By Joel McFarland

NCES and the Department of Education have released national and state-level Average Cohort Graduation Rates for the 2015-16 school year. You can see the data on the NCES website (as well as data from 2010-11 through 2015-16).

In recent years, NCES has released two widely-used annual measures of high school completion: the Adjusted Cohort Graduation Rate (ACGR) and the Averaged Freshman Graduation Rate (AFGR). Both measure the percent of public school students who attain a regular high school diploma within 4 years of starting 9th grade. However, they also differ in important ways. This post provides an overview of how each measure is calculated and why they may result in different rates.

What is the Adjusted Cohort Graduation Rate (ACGR)?

The ACGR was first collected for 2010-11 and is a newer graduation rate measure. To calculate the ACGR, states identify the “cohort” of first-time 9th graders in a particular school year, and adjust this number by adding any students who transfer into the cohort after 9th grade and subtracting any students who transfer out, emigrate to another country, or pass away. The ACGR is the percentage of the students in this cohort who graduate within four years. States calculate the ACGR for individual schools and districts and for the state as a whole using detailed data that track each student over time. In many states, these student-level records have become available at a state level only in recent years. As an example, the ACGR formula for 2012-13 was calculated like this:

Average Cohort Graduation Rate calculation

What is the Averaged Freshman Graduation Rate (AFGR)?

The AFGR uses aggregate student enrollment data to estimate the size of an incoming freshman class, which is compared to the number of high school diplomas awarded 4 years later. The incoming freshman class size is estimated by summing 8th grade enrollment in year one, 9th grade enrollment for the next year, and 10th grade enrollment for the year after, and then dividing by three. The averaging of the enrollment counts helps to smooth out the enrollment bump typically seen in 9th grade. The AFGR estimate is less accurate than the ACGR, but it can be estimated as far back as the 1960s since it requires only aggregate annual counts of enrollment and graduate data. As an example, the AFGR formula for 2012-13 was:

Average Freshman Graduation Rate calculation

Why do they produce different rates?

There are several reasons the AFGR and ACGR do not match exactly.

  • The AFGR’s estimate of the incoming freshman class is fixed, and is not adjusted to account for students entering or exiting the cohort during high school. As a result it is very sensitive to migration trends. If there is net out-migration after the initial cohort size is estimated, the AFGR will understate the graduation rate relative to the ACGR. If there is net in-migration, the AFGR will overstate the graduation rate;
  • The diploma count used in the AFGR includes any students who graduate with a regular high school diploma in a given school year, which may include students who took more or less than four years to graduate. The ACGR includes only those students who graduate within four years of starting ninth grade. This can cause the AFGR to be inflated relative to the ACGR; and
  • The AFGR’s averaged enrollment base is sensitive to the presence of 8th and 9th grade dropouts. Students who drop out in the 8th grade in one year are not eligible to be first-time freshmen the next year, but are included in the calculation of the AFGR enrollment base. At the same time, 9th grade dropouts should be counted as first-time 9th graders, but are excluded from the 10th grade enrollment counts used in the AFGR enrollment base. Since more students typically drop out in 9th grade than in 8th grade, the overall impact is likely to underestimate the AFGR enrollment base relative to the true ACGR cohort.

At the national level, these factors largely balance out, and the AFGR closely tracks the ACGR. For instance, in 2012-13, there was less than one percentage point difference between the AFGR (81.9%) and the ACGR (81.4%). At the state level, especially for small population subgroups, there is often more variation between the two measures.

On the NCES website you can access the most recently available data for each measure, including 2016-17 adjusted cohort graduation rates and 2012-13 averaged freshman graduation rates. You can find more data on high school graduation and dropout rates in the annual report Trends in High School Dropout and Completion Rates in the United States.

This blog was originally posted on July 15, 2015 and was updated on February 2, 2016, December 4, 2017, and January 24, 2019.

Using Game-Based Technologies for Civic Education

It is clear that civic education is a priority in the United States. All U.S. states require students to take a civics or government course and many provide opportunities for students to perform community service or service learning to practice active citizenship.

Screenshot of ECO

However, reports illustrate that many young people are not being adequately prepared as citizens. For example, in the most recent National Assessment of Education Progress (NAEP) in Civics only one-quarter of students reached “proficient” in knowledge of key facts on the U.S. Constitution or the functions of government.

Amid calls to strengthen civic education, IES has funded several interventions that are leveraging technological innovation and game design to engage students. These projects are mainly funded through two IES programs—the Small Business Innovation Research (SBIR) Program and Education Research Grants in Education Technology.

Among the projects IES has funded:

  • ECO (pictured above) is a game-based virtual environment where students collaboratively build a shared civilization and, in the process, apply democratic skills such as making rules and laws, analyzing data, and deliberation (watch video);
  • Discussion Maker is a role-playing game where students debate a civic issue, such as freedom of speech. The technology assigns each student a role, facilitates analysis and using evidence to make an argument, and organizes face-to-face debates and deliberation (watch video);
  • GlobalED2 is a role-playing game addressing a simulated global crisis (e.g., major oil spill or water scarcity) where students act as representatives of different countries governments. Students analyze the issue from the perspective of their country and negotiate with other countries to create a solution (watch video);
  • EcoMUVE is a 3D multi-user virtual pond or forest environment where students apply inquiry-based practices to understand causal patterns in ecological science (watch video); andScreenshot of Up from the Dust
  • Mission US’s Up from the Dust (pictured right) is a historical fiction adventure style game where students take on the role wheat farmers during the Great Depression in Texas 1929, and in doing so guide decisions by learning about the interaction of agriculture, the environment, and how government policies affect the economy (watch video).

Specific design elements of these games offer new ways to stimulate young people’s civic knowledge, skills, and engagement. For example:

All of these interventions employ game-based learning to motivate and engage students in new ways. For example, ECO is an unscripted “sandbox” where students create unique and personalized worlds, while Up from the Dust follows an adventure-based historical story. GlobalED2 and Discussion Maker employ role-playing and EcoMUVE guides inquiry-based virtual and real-world exploration.

Additionally, these programs all seek to build citizenship skills with cross-disciplinary content. For example, ECO, EcoMUVE, and GlobalED2 focus building citizen-science skills such as inquiry and analysis; Discussion Maker can apply content from almost any course within democratic debate; and Up from the Dust immerses students in a storyline with history, economics, and government concepts.

At the same time, these interventions also promote collaborative civic learning by simulating democratic processes for a whole class of students, both virtually and face-to-face. In ECO, students collaborate to create their own government with which they need to maintain through rules and laws. Discussion Maker and GlobalED2 use analysis, deliberation, and debates for individuals and groups of students. In EcoMUVE, students can conduct inquiry-based learning within the virtual environment. In Up from the Dust students engage in group discussions after gameplay to assess how specific decisions influenced results.

Several of these games also allow for feasible classroom implementation of what would otherwise be complex interventions. For example, some of the interventions provide real-time cues to students as they progress through the game, allowing a teacher to focus on facilitating overall instruction rather than coordinating every step of the game for every student.

To learn more about these projects, and to learn more about upcoming funding opportunities for the research and development and evaluation of technologies that support civic learning, visit the IES website or follow IES on Facebook and Twitter @IESResearch

Ed Metz is a Research Scientist at IES, where he leads the SBIR and the Education Technology Research Grants programs.