Inside IES Research

Notes from NCER & NCSER

A Lightbulb Moment: How IES Sparks Research, Teaching, and Practice

We know a lot about how people learn and the strategies and principles that promote learning and retention, but much of it gets stuck in translation between research and practice. Through the Cognition and Student Learning program, the Institute of Education Sciences (IES) supports projects that try to bridge that gap. Pooja Agarwal has been a researcher for multiple Cognition and Student Learning projects. She was also a Harry S. Truman Foundation Scholarship recipient, which gave her the opportunity to work as an intern at IES for a summer. She received her PhD from Washington University of St. Louis and is currently an Assistant Professor at Berklee College of Music and founder of RetrievalPractice.org. Here she shares some reflections on how IES and its grants programs have influenced her career and the field, culminating in a book she recently published with collaborator Patrice Bain, Powerful Teaching: Unleash the Science of Learning.

 

Fifteen years ago, I was double majoring in cognitive neuroscience and elementary education at Washington University in St. Louis. One semester, I was taking a psychology class on human memory on one side of campus and a class on K–12 social studies teaching methods on the other side of campus. I felt frustrated that the psychology class was too esoteric, thinking that’s not how memory works in the “real world.” Meanwhile, I felt the social studies methods class was too anecdotal; we were being told to teach the way the professor taught without any evidence to support their methods.

Around the same time, the Institute of Education Sciences (IES) was born, bridging the gap between research and practice. I was enthralled by the legislative authorization bill (yes, I was that college student). The more I read about IES, the more I was convinced that transformation of the education system through research was not only possible–it was starting to happen. Simply put, this was my coveted “lightbulb moment.”

Subsequently, I spent a summer as an intern at IES, and the following fall, my mentor Henry L. Roediger, III (“Roddy”) and colleagues received a Cognition and Student Learning (CASL) grant from IES (2006-2010). This opportunity was a perfect fit with my growing passion: I would be embedded in K–12 classrooms leading rigorous research on learning and memory.

Our research centered around extending a laboratory-based principle, retrieval practice, into the classroom. Retrieval practice is the process by which learners recall or retrieve information they have previously learned, which subsequently improves their long-term retention for that information. For example, do you know the fourth president of the United States? Your mental struggle is referred to as a “desirable difficulty,” which will help you remember the name of the president (it’s listed at the end of this blog).

Our first experiments on retrieval practice were conducted with Patrice Bain, a 6th grade world history teacher at Columbia Middle School in Columbia, Illinois. Initially, we compared student performance after lessons with brief quizzes vs. lessons without quizzes. Importantly, Patrice’s curriculum stayed the same; we simply included frequent retrieval practice. In one set of experiments, for example, retrieval practice boosted grades from a C to an A level, with benefits lasting nine months later, until the end of the school year. By year two, we were collecting data on a scale we never had in the lab, in various grade levels and content areas. We were fortunate to continue our research with a second CASL grant (2011-2014), publishing numerous peer-review publications, presenting at academic conferences, and creating a practice guide for educators on the research and implementation of retrieval practice in the classroom. I recently completed a review of the literature on retrieval practice, screening more than 2,000 abstracts and narrowing them down to 50 selected experiments. The majority of experiments demonstrated that retrieval practice consistently boosted student learning, regardless of type (for example, multiple-choice or short answer), spacing over time, or education level.

To get this information into the hands of educators around the world, my fellow cognitive scientists and I have written and disseminated 6 practice guides available in 6 languages, which have been downloaded more than 100,000 times. I continue to develop a community of more than 15,000 educators around the world via social media (Twitter and Facebook), a weekly newsletter, and articles and podcasts. Of course, our own research on retrieval practice informs my own teaching as a college professor on a daily basis.

Most recently, I co-authored a book, Powerful Teaching: Unleash the Science of Learning, with aforementioned collaborator Patrice Bain. Educators are given the impossible challenge of seeking out good research, making sense of it, and applying it in the classroom. It is impossible because this research isn’t accessible–literally and figuratively. In line with IES’s mission, we aimed to increase access to cognitive science research and make it applicable for today’s classrooms.

In Powerful Teaching, we focused on four teaching strategies we call Power Tools: retrieval practice, spacing, interleaving, and feedback-driven metacognition, all of which are supported by research from the CASL grants program.

  1. Retrieval practice boosts learning by pulling information out of students’ heads, rather than cramming information into students’ heads.
  2. Spacing boosts learning by spreading lessons and retrieval opportunities out over time, so learning is not crammed all at once. In this way, forgetting is a good thing for learning.
  3. Interleaving boosts learning by mixing up closely related topics, encouraging discrimination between similarities and differences.
  4. Feedback-driven metacognition boosts learning by providing the opportunity for students to know what they know and know what they don’t know.

The four Power Tools are flexible, practical, and quick to implement. By focusing on just a few carefully selected strategies, educators are empowered to harness cognitive science, without being stretched too thin. We have found these elements–accessibility and feasibility–to be critical if educators are to implement research-based strategies in their classrooms.

I say all this because I want to emphasize that IES–and the people behind the scenes–is much more than a granting agency. IES provides opportunities and support for applied research in education that can inform practice and make a difference in the classroom. Sometimes, all it takes is a lightbulb moment to spark a transformation.

 

Endnote: The fourth president of the United States was James Madison.


Pooja K. Agarwal, PhD (@RetrieveLearn) is a cognitive scientist, conducting research on how students learn since 2005. She is the author of the book Powerful Teaching: Unleash the Science of Learning and an Assistant Professor of Psychology at the Berklee College of Music in Boston. Pooja is also the Founder of RetrievalPractice.org, a source of research-based teaching strategies for more than 15,000 educators around the world.

Using Mistakes as a Vehicle for Learning in Mathematics: From Research to Practice at Scale in Education

Every student makes mistakes. But not every student is given the opportunity to learn from mistakes. Left unaddressed, the mathematical misconceptions that underlie many mistakes can keep students from progressing in mathematics.

 

At the request of districts in the Minority Student Achievement Network (MSAN), a Strategic Education Research Partnership (SERP) team was convened in 2007 to address a widening achievement gap in Algebra I. The team was charged with identifying an intervention strategy, subject to several district constraints:

  1. The solution would need to be applied to all students in the regular classroom to avoid the stereotype threat associated with separating students based on performance and to protect the intervention from budget cuts that target supplemental, after-school, and summer programs first.
  2. A new curriculum was off the table because it would create upheaval for a time and would be followed by a decline in student performance during the period of adjustment.
  3. Extensive teacher training was considered undesirable because it would be costly and because algebra teachers consider themselves more expert in mathematics teaching than central office staff who would be requiring the training.

 

Julie Booth joined the partnership, and with funding from IES, led the iterative development and testing of worked example assignments that, with the input of teachers and administrators, fit within the routines of the classroom. The result—AlgebraByExample—consists of 42 uniquely designed assignments that address misconceptions, harness the power of explanation, and use mistakes as a vehicle for learning.

Typical math assignments require students to solve problems on their own. If a student’s work is incorrect, the student may never focus on what went wrong. ByExample assignments also give students problems to solve, but they first provide a solution to a similar problem that is marked right or wrong. Students are prompted with questions that target common misconceptions and errors before solving a similar problem on their own. Each assignment contains several strategically designed item pairs:

 

 

Designed in collaboration with teachers from districts in several states, the assignments can be easily incorporated into any Algebra I curriculum and teachers can choose in what way and in what order to use them. The assignments were tested in randomized trials in classrooms in eight districts with more than 6,000 students. Not only did students using AlgebraByExample improve an average of 7 percentage points on an assessment of standardized test items, students at the lower end of the distribution improved the most. The PDF downloads of the assignments are freely available for anyone to use.

The success of AlgebraByExample  led to  further IES funding of MathByExample for Grades 4 and 5 and GeometryByExample for high school geometry .

 

Resources:

AlgebraByExample website

MathByExample website

Booth et al, 2015

NSF Stem for All Video Submission 2019

 

Interview with Dr. Suzanne Donovan (SERP), Dr. Julie Booth (Temple University), and Allie Huyghe (SERP), the developers of the ByExample interventions.

 

 

Was it part of the original plan to develop an intervention that could one day be used at scale in schools?

Yes. SERP partnerships begin with problems of practice nominated by district partners, but the partnership agreement distinguishes SERP from a consultant. The intention from the start is to frame the problem and design a solution that can be used at scale. SERP has developed in-house, user-centered design expertise so that resources (such as the ByExample products) developed through partnerships meet the needs of teachers and students. Products scale when they improve the experience of teachers and students. Both the model and the internal design capacity allow SERP to move from problem framing through research, development, and dissemination of a product with IES grant funding.

 

Describe the initial research and development that occurred.

Dr. Julie Booth drafted initial assignments drawing on the mathematics misconceptions literature. SERP held regular partnership meetings with teachers and administrators at which assignments were reviewed and additional misconceptions were nominated for attention in the assignments. Administrators agreed to randomization of the assignments across classrooms and within-teacher. Assignments were first tested in individual topic blocks and revised in accordance with student performance data, observations, and teacher feedback. A year-long pilot study was then conducted using the full set of assignments.

 

Beyond IES or ED grants, what additional funding was needed to develop the intervention?

For the ByExample work, additional funding was provided by the Goldman Sachs Foundation in the initial phase to support partnership formation, problem framing, and the solution generation. IES grants funded the research and development, along with initial dissemination activities to make the materials available to the public. We were also able to develop an online platform to allow for digital use with the IES grant funds.

 

What model was used for dissemination and sustainability?

The assignments are available as free downloads on SERP’s website, and as printed workbooks through SERP’s partner print-on-demand company. They have been publicized through online communications, journal articles, presentations at conferences of various types, social media, and word of mouth. There will be a small fee for use of the digital platform to support its maintenance, but the PDFs will remain as free downloads. We have been able to sustain the collaboration of the partnership team by responding to requests from educators to expand the approach to other grade levels and submitting additional proposals to IES that have been awarded.

 

What advice would you provide to researchers who are looking to move their research from the lab to market? What steps should they take? What resources should they look for?

First, I would note that it is difficult to persuade educators to use a product that solves a problem they don’t believe they have. Listen to educators and apply research expertise to address the challenges that they experience on a day-to-day basis. Design for ease of use by teachers. No matter how good your strategy or marketing is, if it’s too much work for an already busy teacher to use, you may get uptake by a few committed teachers, but not at scale. Finally, pay attention to where teachers get their information. For AlgebraByExample, we got a big boost from the Marshall Report, produced by a teacher for other teachers to call attention to usable research.  

 

In one sentence, what would you say is most needed for gaining traction and wide scale use by educators?

Design for the routines of the classroom.

 


Suzanne Donovan, PhD, is the founding Executive Director of the SERP Institute, an education research, development, and implementation organization incubated at the National Academies. SERP leads collaborations of educators, researchers, and designers to generate research-based, scalable, and sustainable solutions to critical problems of practice. 

Julie Booth, PhD, is a Professor of STEM Education and Psychology and the Deputy Dean of Academic and Faculty Affairs at Temple University’s College of Education and Human Development. Her work focuses on translating between cognitive science and education to better understand students’ learning and improve instruction, primarily in mathematics education. She is currently an Executive Editor for the Journal of Experimental Education.

Allie Huyghe is the Assistant Director of the SERP Institute, where she manages several projects, including the IES-funded MathbyExample and GeometryByExample projects. She is also intricately involved with other SERP areas of work, participating in the design of materials from early development through release to the public.

 

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov) and Edward Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This is the fifth in an ongoing series of blog posts examining moving from university research to practice at scale in education.​

 

 

Investing in Next Generation of Education Technologies to Personalize Learning and Inform Instructional Practice

The Institute of Education Sciences Small Business Innovation Research program (ED/IES SBIR) funds entrepreneurial developers to create the next generation of education technology for students, teachers, and administrators in general and special education. The program emphasizes an iterative research and development process and pilot studies to evaluate the feasibility, usability, and promise of new products to improve educational outcomes. The program also focuses on commercialization after development is complete, so that the products can reach schools and be sustained over time.

In recent years, millions of students in tens of thousands of schools around the country have used technologies developed through ED/IES SBIR. And in the past four months, about one million students and teachers used the technologies for remote teaching and learning, as many ED/IES SBIR-supported developers made their products available at no cost in response to the COVID-19 pandemic and the closure of schools.

 

ED/IES SBIR Announces its 2020 Awards

This week, ED/IES SBIR announced the results of its 2020 award competition. Of the 22 new awards, 16 are for prototype development and 6 are for full-scale development. IES also announced two additional awards through a special topic solicitation in postsecondary education. Read about these awards here.

 

 

Each of the new awards supports a project to develop a product to personalize the student learning experience or generate information that educators can use to guide practice.

Most of the projects are developing a software component (for example, algorithms, artificial intelligence, machine Learning, or natural language processing) that continually adjusts the difficulty of content, provides prompts to support individual students if support is needed, or generates real-time actionable information for educators to track student progress and adjust instruction accordingly. Other projects are developing technologies to promote student learning through self-directed, hands-on, simulated, and immersive experiences. If the future of education includes a blend of in-class and remote learning due to public health crises, or for whatever reasons, technologies such as these will be ready to keep the learning going.

The projects address different ages of students and content areas.

In science, LightHaus is fully developing a virtual reality (VR) intervention for students to explore plant heredity; LightUp is fully developing an augmented reality (AR) app for students to perform hands-on physical science investigations with their own on-device camera; and Myriad Sensors is developing a prototype artificial intelligence formative assessment system that generates feedback in real time as students do hands-on laboratory experiments.

In math, Muzology is creating a prototype for students to create music videos to learn algebra, and Teachley is creating a prototype transmedia kit with videos, comics, and pictures to enhance teaching and learning of hard to learn concepts.

In engineering and computer science, Parametric Studios is fully developing an augmented reality puzzle game for early learners, and Liminal eSports, Makefully, and Beach Day Studios are creating prototype components that each provide feedback to students as they engage in activities to learn to code.

In English Language Arts, Analytic Measures and Hoogalit are each employing natural language processing to develop new prototypes to facilitate speech acquisition, and Learning Ovations is developing a prototype data engine to make recommendations for what individual children should read.

For English learners, KooApps is developing an artificial intelligence prototype to support vocabulary acquisition, and Kings Peak Technologies is employing machine learning to generate passages that blend English and Spanish words together to improve reading comprehension.

For early learners, Cognitive Toybox is fully developing an observation and game-based school readiness assessment.

For postsecondary students, Hats & Ladders is fully developing a social skills game to foster career readiness skills.

In special education, Attainment Company is developing a prototype to support student’s self-management, and Alchemie is developing a prototype of an augmented reality science experience for visually impaired students.

To support school administrators and teachers, LearnPlatform is fully developing a dashboard that generates reports with insights for teachers to implement education technology interventions, and Zuni Learning Tree, Teachley and LiveSchool are developing prototype dashboards to organize and present results on student progress and performance in real time.

 

Stay tuned for updates on Twitter and Facebook as IES continues to support innovative forms of technology.


Written by Edward Metz (Edward.Metz@ed.gov), Program Manager, ED/IES SBIR

Addressing Persistent Disparities in Education Through IES Research

Spring 2020 has been a season of upheaval for students and educational institutions across the country. Just when the conditions around the COVID-19 pandemic began to improve, the longstanding symptoms of a different wave of distress resurfaced. We are seeing and experiencing the fear, distrust, and confusion that are the result of systemic racism and bigotry. For education stakeholders, both the COVID-19 pandemic and the civil unrest unfolding across the country accentuate the systemic inequities in access, opportunities, resources, and outcomes that continue to exist in education.

IES acknowledges these inequities and is supporting rigorous research that is helping to identify, measure, and address persistent disparities in education.

In January (back when large gatherings were a thing), IES hosted its Annual Principal Investigator’s (PI) Meeting with the theme of Closing the Gaps for All Learners. The theme underscored IES's objective of supporting research that improves equity in education access and outcomes. Presentations from IES-funded projects focusing on diversity, equity, and inclusion were included throughout the meeting and can be found here. In addition, below are highlights of several IES-funded studies that are exploring, developing, or evaluating programs, practices, and policies that education stakeholders can implement to help reduce bias and inequities in schools.

 

 

 

  • The Men of Color College Achievement (MoCCA) Project - This project addresses the problem of low completion rates for men of color at community colleges through an intervention that provides incoming male students of color with a culturally relevant student success course and adult mentors. In partnership with the Community College of Baltimore County, the team is engaged in program development, qualitative data collections to understand student perspectives, and an evaluation of the success course/mentorship intervention. This project is part of the College Completion Network and posts resources for supporting men of color here.

 

  • Identifying Discrete and Malleable Indicators of Culturally Responsive Instruction and Discipline—The purpose of this project is to use the culturally responsive practices (CRP) framework from a promising intervention, Double Check, to define and specify discrete indicators of CRPs; confirm and refine teacher and student surveys and classroom direct observation tools to measure these discrete indicators; and develop, refine, and evaluate a theory of change linking these indicators of CRPs with student academic and behavioral outcomes.

 

 

  • The Early Learning Network (Supporting Early Learning From Preschool Through Early Elementary School Grades Network)—The purpose of this research network is to examine why many children—especially children from low-income households or other disadvantaged backgrounds—experience academic and social difficulties as they begin elementary school. Network members are identifying factors (such as state and local policies, instructional practices, and parental support) that are associated with early learning and achievement from preschool through the early elementary school grades.
    • At the January 2020 IES PI Meeting, Early Learning Network researchers presented on the achievement gaps for early learners. Watch the video here. Presentations, newsletters, and other resources are available on the Early Learning Network website.

 

  • Reducing Achievement Gaps at Scale Through a Brief Self-Affirmation Intervention—In this study, researchers will test the effectiveness at scale of a low-cost, self-affirmation mindset intervention on the achievement, behavior, and attitudes of 7th grade students, focusing primarily on Black and Hispanic students. These minority student groups are susceptible to the threat of conforming to or being judged by negative stereotypes about the general underperformance of their racial/ethnic group ("stereotype threat"). A prior evaluation of this intervention has been reviewed by the What Works Clearinghouse and met standards without reservations.

 

 

IES seeks to work with education stakeholders at every level (for example, students, parents, educators, researchers, funders, and policy makers) to improve education access, equity, and outcomes for all learners, especially those who have been impacted by systemic bias. Together, we can do more.

This fall, IES will be hosting a technical working group on increasing the participation of researchers and institutions that have been historically underutilized in federal education research activities. If you have suggestions for how IES can better support research to improve equity in education, please contact us: NCER.Commissioner@ed.gov.  


Written by Christina Chhin (Christina.Chhin@ed.gov), National Center for Education Research (NCER).  

This is the fourth in a series of blog posts that stems from the 2020 Annual Principal Investigators Meeting. The theme of the meeting was Closing the Gaps for All Learners and focused on IES’s objective to support research that improves equity in access to education and education outcomes. Other posts in this series include Why I Want to Become an Education Researcher, Diversify Education Sciences? Yes, We Can!, and Closing the Opportunity Gap Through Instructional Alternatives to Exclusionary Discipline.

A2i: From Research to Practice at Scale in Education

This blog post is part of an interview series with education researchers who have successfully scaled their interventions.

Assessment-to-Instruction (A2i) is an online Teacher Professional Support System that guides teachers in providing Kindergarten to Grade 3 students individualized literacy instruction and assessments. Students complete the assessments independently online without the teacher taking time away from instruction. A2i generates instantaneous teacher reports with precise recommendations for each student and group recommendations. See a video demo here. Between 2003 and 2017, researchers at Florida State University (FSU) and Arizona State University (ASU), led by Carol Connor, developed and evaluated A2i with the support of a series of awards from IES and the National Institutes of Health. Findings from all publications on the A2i are posted here.

While results across seven controlled studies demonstrated the effectiveness of A2i, feedback from practitioners in the field demonstrated that implementation often required substantial amounts of researcher support and local district adaptation, and that the cost was not sustainable for most school district budgets. In 2014, the development firm Learning Ovations, led by Jay Connor, received an award from the Department of Education (ED) and IES’s Small Business Innovation Research program (ED/IES SBIR) to develop an technologically upgraded and commercially viable version of A2i to be ready to be used at scale in classrooms around the country. In 2018, with the support of a five-year Education Innovation and Research (EIR) expansion grant from ED totaling $14.65 million, A2i is now used in more than 110 schools across the country, with plans for further expansion. 

 

Interview with Carol Connor (CC) and Jay Connor (JC)

From the start of the research in the early 2000s, was it always the goal to develop a reading intervention that would one day be used on a wide scale?
CC: Yes and no. First, we had to answer the question as to whether individualization was effective in achieving student literacy outcomes. Once the research established that, we knew that this work would have wide-scale application.

When did you start thinking about a plan for distribution
CC: Before embarking on the cumulative results studies, in 2008, Jay said that we needed to know who the “customer” was… i.e., how purchasing decisions were made at scale.  His 2008 Phase I ED/IES SBIR was critical in shifting our research focus from individual classrooms to school districts as the key scaling point. 

Did you work with a technology transfer office at the university?
CC: Only to the extent of contractually clarifying intellectual property (IP) ownership and licensing. 

Who provided the support on the business side?
CC: Jay, who has an MBA/JD and has been a senior officer in two Fortune 100 companies was very instrumental in guiding our thinking of this evolution from important research to practical application. 


 Do you have any agreement about the IP with the university? What were the biggest challenges in this area?

JC: Yes, Learning Ovations has a 60-year renewable exclusive licensing agreement with FSU Foundation. FSU couldn’t have been better to work with.  Though there were expected back-and-forth elements of the original negotiations, it was clear that we shared the central vision of transforming literacy outcomes.  They continue to be a meaningful partner.

When and why was Learning Ovations first launched?
JC: In order to pursue SBIR funding we needed to be a for-profit company.  At first, I used my consulting business – Rubicon Partners LLP – as the legal entity for a 2008 Phase I award from ED/IES SBIR.  When we considered applying (and eventually won) a Fast Track Phase I & II award from SBIR in 2014, it was clear that we needed to create a full C – Corp that could expand with the scaling of the business, thus Learning Ovations was formed.

Who has provided you great guidance on the business side over the year? What did they say and do? 
JC: Having run large corporate entities and worked with small business start-ups in conjunction with Arizona State University (Skysong) and the University of California, Irvine (Applied Innovation at The Cove) and having taught entrepreneurship at The Paul Merage School of Business at UC Irvine, I had the experience or network to connect to whatever business guidance we needed.  Further, having attended a number of reading research conferences with Carol, I was quite conversant in the literacy language both from the research side and from the district decision maker’s side.

How do you describe the experience of commercializing the A2i? What were the biggest achievements and challenges in terms of preparing for commercialization?

JC: Having coached scores of entrepreneurs at various stages, I can safely say that there is no harder commercialization than one that must stay faithful to the underlying research.  A key strategy for most new businesses: being able to pivot as you find a better (easier) solution.  It is often circumscribed by the “active ingredients” of the underlying research.  Knowing this, we imbued Learning Ovations with a very strong outcomes mission – all children reading at, or above, grade level by 3rd grade.  This commitment to outcomes certainty is only assured by staying faithful to the research.  Thus, a possible constraint, became our uncontroverted strength.

Do you have advice for university researchers seeking to move their laboratory research in education into wide-spread practice? 
JC:  Start with the end in mind.  As soon as you envision wide-scale usage, learn as much as you can about the present pain and needs of your future users and frame your research questions to speak to this.  Implementation should not be an after-the-fact consideration; build it into how you frame your research questions. On one level you are asking simultaneously “will this work with my treatment group” AND “will this help me understand/deliver to my end-user group.”  I can’t imagine effective research being graphed onto a business after the fact.  One key risk that we see a number of researchers make is thinking in very small fragments whereas application (i.e., the ability to go to scale) is usually much more systemic and holistic.

In one sentence, what would say is most needed for gaining traction in the marketplace?
JC: If not you, as a researcher, someone on your team of advisors needs to know the target marketplace as well as you know the treatment protocols in your RCT.

____________

Carol Connor is a Chancellor’s Professor in the UC Irvine School of Education. Prior she was a professor of Psychology and a Senior Learning Scientist at the Learning Sciences Institute at ASU. Carol’s research focuses on teaching and learning in preschool through fifth grade classrooms – with a particular emphasis on reading comprehension, executive functioning, and behavioral regulation development, especially for low-income children.

Joseph “Jay” Connor, JD/MBA, is the Founder/CEO of Learning Ovations, Inc, the developer of the platform that has enabled the A2i intervention to scale.  Jay has 20+ years of experience in senior business management at the multi-billion dollar corporate level, and has experience in the nonprofit and public policy arenas.

This interview was produced by Edward Metz of the Institute of Education Sciences.