Inside IES Research

We know a lot about how people learn and the strategies and principles that promote learning and retention, but much of it gets stuck in translation between research and practice. Through the Cognition and Student Learning program, the Institute of Education Sciences (IES) supports projects that try to bridge that gap. Pooja Agarwal has been a researcher for multiple Cognition and Student Learning projects. She was also a Harry S. Truman Foundation Scholarship recipient, which gave her the opportunity to work as an intern at IES for a summer. She received her PhD from Washington University of St. Louis and is currently an Assistant Professor at Berklee College of Music and founder of RetrievalPractice.org. Here she shares some reflections on how IES and its grants programs have influenced her career and the field, culminating in a book she recently published with collaborator Patrice Bain, Powerful Teaching: Unleash the Science of Learning.

Fifteen years ago, I was double majoring in cognitive neuroscience and elementary education at Washington University in St. Louis. One semester, I was taking a psychology class on human memory on one side of campus and a class on K–12 social studies teaching methods on the other side of campus. I felt frustrated that the psychology class was too esoteric, thinking that’s not how memory works in the “real world.” Meanwhile, I felt the social studies methods class was too anecdotal; we were being told to teach the way the professor taught without any evidence to support their methods.

Around the same time, the Institute of Education Sciences (IES) was born, bridging the gap between research and practice. I was enthralled by the legislative authorization bill (yes, I was that college student). The more I read about IES, the more I was convinced that transformation of the education system through research was not only possible–it was starting to happen. Simply put, this was my coveted “lightbulb moment.”

Subsequently, I spent a summer as an intern at IES, and the following fall, my mentor Henry L. Roediger, III (“Roddy”) and colleagues received a Cognition and Student Learning (CASL) grant from IES (2006-2010). This opportunity was a perfect fit with my growing passion: I would be embedded in K–12 classrooms leading rigorous research on learning and memory.

Our research centered around extending a laboratory-based principle, retrieval practice, into the classroom. Retrieval practice is the process by which learners recall or retrieve information they have previously learned, which subsequently improves their long-term retention for that information. For example, do you know the fourth president of the United States? Your mental struggle is referred to as a “desirable difficulty,” which will help you remember the name of the president (it’s listed at the end of this blog).

Our first experiments on retrieval practice were conducted with Patrice Bain, a 6th grade world history teacher at Columbia Middle School in Columbia, Illinois. Initially, we compared student performance after lessons with brief quizzes vs. lessons without quizzes. Importantly, Patrice’s curriculum stayed the same; we simply included frequent retrieval practice. In one set of experiments, for example, retrieval practice boosted grades from a C to an A level, with benefits lasting nine months later, until the end of the school year. By year two, we were collecting data on a scale we never had in the lab, in various grade levels and content areas. We were fortunate to continue our research with a second CASL grant (2011-2014), publishing numerous peer-review publications, presenting at academic conferences, and creating a practice guide for educators on the research and implementation of retrieval practice in the classroom. I recently completed a review of the literature on retrieval practice, screening more than 2,000 abstracts and narrowing them down to 50 selected experiments. The majority of experiments demonstrated that retrieval practice consistently boosted student learning, regardless of type (for example, multiple-choice or short answer), spacing over time, or education level.

To get this information into the hands of educators around the world, my fellow cognitive scientists and I have written and disseminated 6 practice guides available in 6 languages, which have been downloaded more than 100,000 times. I continue to develop a community of more than 15,000 educators around the world via social media (Twitter and Facebook), a weekly newsletter, and articles and podcasts. Of course, our own research on retrieval practice informs my own teaching as a college professor on a daily basis.

Most recently, I co-authored a book, Powerful Teaching: Unleash the Science of Learning, with aforementioned collaborator Patrice Bain. Educators are given the impossible challenge of seeking out good research, making sense of it, and applying it in the classroom. It is impossible because this research isn’t accessible–literally and figuratively. In line with IES’s mission, we aimed to increase access to cognitive science research and make it applicable for today’s classrooms.

In Powerful Teaching, we focused on four teaching strategies we call Power Tools: retrieval practice, spacing, interleaving, and feedback-driven metacognition, all of which are supported by research from the CASL grants program.

1. Retrieval practice boosts learning by pulling information out of students’ heads, rather than cramming information into students’ heads.
2. Spacing boosts learning by spreading lessons and retrieval opportunities out over time, so learning is not crammed all at once. In this way, forgetting is a good thing for learning.
3. Interleaving boosts learning by mixing up closely related topics, encouraging discrimination between similarities and differences.
4. Feedback-driven metacognition boosts learning by providing the opportunity for students to know what they know and know what they don’t know.

The four Power Tools are flexible, practical, and quick to implement. By focusing on just a few carefully selected strategies, educators are empowered to harness cognitive science, without being stretched too thin. We have found these elements–accessibility and feasibility–to be critical if educators are to implement research-based strategies in their classrooms.

I say all this because I want to emphasize that IES–and the people behind the scenes–is much more than a granting agency. IES provides opportunities and support for applied research in education that can inform practice and make a difference in the classroom. Sometimes, all it takes is a lightbulb moment to spark a transformation.

Endnote: The fourth president of the United States was James Madison.

Pooja K. Agarwal, PhD (@RetrieveLearn) is a cognitive scientist, conducting research on how students learn since 2005. She is the author of the book Powerful Teaching: Unleash the Science of Learning and an Assistant Professor of Psychology at the Berklee College of Music in Boston. Pooja is also the Founder of RetrievalPractice.org, a source of research-based teaching strategies for more than 15,000 educators around the world.

Editor’s Note: This Inside IES Blog is crossed-posted on Homeroom, the official blog of the U.S. Department of Education.

On July 30, 2020, NASA launched a rocket from Cape Canaveral Air Force Station in Florida on a journey to Mars. The rocket is carrying a rover named Perseverance and a helicopter named Ingenuity, both of which will land inside Mars's Jezero Crater on February 18, 2021. While on Mars, Perseverance and Ingenuity will collect the first Martian soil and rock samples for future return to Earth, search for signs of extinct or extant life, characterize the planet’s climate and geology, and pave the way for human exploration of the Red Planet with the help of new technologies and scientific instruments.

Perseverance and Ingenuity were named by students through a national Kindergarten to Grade 12 student competition run by NASA in partnership with Future Engineers and Battelle Education.

The student whose entry won the prize to name the rover is Alexander Mather, a seventh grader from Lake Braddock Secondary School in Burke, Virginia. Alexander submitted the name Perseverance and included the following in his essay:

“Curiosity. Insight. Spirit. Opportunity. If you think about it, all of these names of past Mars rovers are qualities we possess as humans. We are always curious, and seek opportunity. We have the spirit and insight to explore the Moon, Mars, and beyond. But, if rovers are to be the qualities of us as a race, we missed the most important thing. Perseverance.”

Watch the March 5 program where the winning name was revealed here:

The student whose entry won the prize to name the helicopter is Vaneeza Rupani, a junior at Tuscaloosa County High School in Northport, Alabama. Vanessa submitted the name Ingenuity and included the following in her essay:

"The ingenuity and brilliance of people working hard to overcome the challenges of interplanetary travel are what allow us all to experience the wonders of space exploration. Ingenuity is what allows people to accomplish amazing things, and it allows us to expand our horizons to the edges of the universe."

Watch the video trailer featuring the naming of the Mars helicopter: 

About the “Name the Rover” Contest

Not only did the contest help NASA pick a new name for the rover, it also engaged U.S. students in the engineering and scientific work that makes Mars exploration possible, stimulated interest in science, technology, engineering, and mathematics (STEM), and inspired the next generation of STEM leaders.

After launching the competition in August 2019, students from 50 states, U.S. territories, and military bases submitted over 28,000 essays. More than 4,500 volunteer judges narrowed the pool to 155 semifinalists. From these, NASA chose nine finalists—Clarity, Courage, Endurance, Fortitude, Ingenuity, Perseverance, Promise, Tenacity, and Vision—and opened a public poll in which anyone could vote. After considering these poll results, NASA officials chose the two names.

To manage the competition, NASA used a web-based platform developed by Burbank, California-based Future Engineers.  This platform was created with the support of a 2017 award from the U.S. Department of Education and Institute of Education Sciences’ Small Business Innovation Research program (ED/IES SBIR).  Future Engineers built this platform to be an online hub for classrooms and educators to access free, project-based STEM activities and to provide a portal where students submit and compete in different kinds of maker and innovation challenges across the country. The Mars 2020 “Name the Rover” contest was the first naming challenge issued on the platform. We look forward to more student challenges to come!

Edward Metz (Edward.Metz@ed.gov) is a research scientist at the Institute of Education Sciences in the US Department of Education.

Bob Collom is an integration lead in the Mars Exploration Program at NASA Headquarters.

About ED/IES SBIR

The U.S. Department of Education’s Small Business Innovation Research program, administered by the Institute of Education Sciences (IES), funds projects to develop education technology products designed to support students, teachers, or administrators in general or special education. The program emphasizes rigorous and relevant research to inform iterative development and to evaluate whether fully developed products show promise for leading to the intended outcomes. The program also focuses on commercialization once the award period ends so that products can reach students and teachers and be sustained over time. ED/IES SBIR-supported products are currently used by millions of students in thousands of schools around the country.

About NASA’s Mars Exploration Program (MEP)

NASA’s Mars Exploration Program (MEP) in the Planetary Science Division is a science-driven program that seeks to understand whether Mars was, is, or can be, a habitable world. To find out, we need to understand how geologic, climatic, and other processes have worked to shape Mars and its environment over time, as well as how they interact today. To that end, all of our future missions will be driven by rigorous scientific questions that will continuously evolve as we make new discoveries. MEP continues to explore Mars and to provide a continuous flow of scientific information and discovery through a carefully selected series of robotic orbiters, landers and mobile laboratories interconnected by a high-bandwidth Mars/Earth communications network. The Mars 2020 Project at NASA’s Jet Propulsion Laboratory manages rover development for NASA’s Science Mission Directorate. NASA’s Launch Services Program, based at the agency’s Kennedy Space Center in Florida, is responsible for launch management.

With a 2010 IES research grant, researchers at Teachers College, Columbia University conducted basic research and created prototype software programs for children in mathematics. In 2011, three members of the research team launched a startup and submitted a successful proposal to IES’s Small Business Innovation Research programs. With awards in 2012 and 2013, the developers created a suite of math game apps that support fact fluency and promote math strategy development. The apps all connect with a teacher dashboard that provides in-depth reports in real time and supports differentiation in math instruction. In 2013, Teachley’s Addimal Adventure won an Apple Design Award as one of the 12 best apps of 2014. Since their commercial launch in 2014, Teachley Apps have been downloaded 1.5 million times, and the Teachley suite of products are currently used in all 50 states and 2,000 schools.

Interview with Kara Carpenter, co-founder of Teachley

The three co-founders of Teachley were all classroom teachers before you met at Teachers College as graduate students in 2010. What led to your decision to go to graduate school to earn PhDs as researchers?

While teaching 2nd grade, I had the opportunity to receive professional development focused on elementary math content, and I became fascinated with how children develop their mathematical thinking. Years later, when I was getting a master’s in curriculum & teaching at Teachers College, I pursued a work study opportunity with Professor Herb Ginsburg, who focuses on early childhood math thinking. At the time in 2009, my cofounder Rachael Labrecque was already working with Professor Ginsburg, and the three of us submitted an application to IES to develop math software for young learners. That fall, I went back to classroom teaching, but when the application was funded in 2010, I decided to take the leap and accept a research fellowship to pursue a PhD. My other co-founder, Dana Pagar, joined our research team that fall, and the three of us decided to start Teachley in 2012 to bring all the great research on how kids learn math into marketable products.

Tell us about the research projects that you were involved with in graduate school.

We worked on a project developing math software for grades pre-K to 3, called MathemAntics. We developed dozens of activities and conducted small learning studies along the way. In the third year, we conducted an RCT with approximately 400 students in grades PreK - 2. Each of our dissertations involved different elements of the project. Mine focused on teaching and detecting kids’ single-digit addition strategies. Dana’s focused on continuous versus discrete blocks, while Rachael studied teachers’ preparedness to integrate technology into their classrooms.

How did you come up with the idea to develop apps that would be used in schools on a wide scale basis?
Originally, we were looking for a company who might want to take these research findings and turn them into commercial products. We were meeting with various business leaders, and one of them turned to us and said, “You should do this. You should start a company to bring your ideas to market.” That’s the push we needed to think of ourselves as potential startup founders.

How did you find out about the SBIR program at the US Department of Education’s Institute of Education Sciences? How important was the first SBIR award for launching Teachley?

Once we decided to start Teachley, we knew that SBIR would be a great resource for us. The MathemAntics project had actually started out as an NIH SBIR Phase I with a different company. That first ED/IES SBIR award is the reason that Teachley became a company. Without that funding, we would not have been able to prove ourselves capable of bringing a product to market. Institutional investors aren’t taking those kinds of risks, and angel investment is too tied into social networks and who you know.

Was Teachers College supportive of its graduate students starting a small business and getting an award to develop apps? Did anyone at the university offer advice or guidance on how to operate a small business?

Leaving the university was tricky because we had research fellowships when we started the company. However, the Teachers College president at the time, Susan Fuhrman, and the provost, Tom James, were supportive of our startup. We speak and participate in various discussions and events at Teachers College, which keeps us connected to the university and the research.

How does Teachley ensure that research is integrated into your development and validation process?

Before developing any new product idea, we look to the research to see what’s already been learned about the topic, especially as it relates to struggling learners. During the early stages of development, we rely on close observations of students as they use pencil/paper mockups and early software builds. As a team, we closely review videos of students working through problems, looking to find better, more intuitive ways to support students’ thinking. Once we have a functional prototype, we use more formal evaluative techniques to determine our impact on student learning.

What models have you used to commercialize Teachley on a widespread basis?

We have tried out many different revenue models. Initially, we tried publishing the games for free and charging schools for the formative assessment data. However, we soon found that bundling the games and data together into a single subscription worked better for schools. With our latest game, Market Bay, we are trying a new model where educators create a free account, and parents subscribe to have access at home. Schools who subscribe to Teachley get home access to Market Bay and our other games for all of their families.

Have you raised funds from venture capitalists? Why or why not?

Not yet. Raising money from venture capitalists can put you on a succeed-or-fail-fast treadmill that isn’t always a great fit for the education market. Many investors are looking for a 70x return within just a few years or they abandon ship. Developing great educational software takes time for both the iterative design process and the research to prove your effectiveness. We are just now at the stage where raising venture capital may soon make sense because we have enough content to scale our school/district sales.

When COVID-19 emerged and schools closed, you made your apps freely available to teachers and students in their classes, and 15,000 teachers and students were able to access your products. What was that experience like?

Teachers are looking for digital products that will deeply engage students and support true learning. We’re a great fit. However, schools across the country are suffering budget shortfalls at the same time as they need to spend more to ensure they meet safety standards. We’re working with schools and teachers to find alternative ways to fund our program, from parent organizations to Donors Choose to corporate partnerships.

None of you had had formal business training prior to founding Teachley. Do you have advice for those who are interested in starting an entrepreneurial small business to develop education technology that can be used in schools?

My advice would be to know your users and implementation deeply. If you don’t have a background in teaching, spend time volunteering in schools. Become a close observer of children and their thinking, so you can create products that support and bring out children’s genius.

 ____________________________________________________________________________

Kara Carpenter is cofounder of Teachley (@teachley), an edtech startup focused on promoting deep math thinking and learning. Kara has over 10 years of teaching experience and was a National Board Certified Teacher with a PhD in Cognition and Learning from Teachers College, Columbia University. Her dissertation went on to become an Apple Design Award winning app, Addimal Adventure.

This interview was produced by Ed Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This post is the sixth in an ongoing series of blog posts examining moving from university research to practice at scale in education.

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

STAGES FOR CONDUCTING A COST ANALYSIS

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

*More resources available here.

The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

Three IES-funded investigators were presented with awards from the Society for Prevention Research (SPR) last week. We recognize and applaud these investigators (pictured above from left to right) Elizabeth Stormshak, Dorothy Espelage, and Patrick Tolan.

Dr. Elizabeth Stormshak, Philip H. Knight Chair and Department Head for Counseling Psychology and Human Services at University of Oregon, received the Translational Science Award from SPR. This award is given to an individual or a team of individuals in recognition for contributions to the field of prevention science through translational research. Dr. Stormshak’s research focuses on understanding risk factors in early and middle childhood associated with the development of problem behavior in late adolescence, including substance use and delinquency. She also studies the process of disseminating evidence-based interventions into real world community settings. She has been the Principal Investigator on multiple IES grants, including a 2018 NCSER-funded project to examine the long-term efficacy of the Kindergarten Family Check-Up (FCU), a school-based, family-centered intervention intended to prevent student social and behavioral problems. This grant is a follow-up and extension to her recently completed randomized controlled trial of Kindergarten FCU, funded by NCER, which found positive impacts on student behavior and academic outcomes during and up to 3 years after the transition to kindergarten. The primary aims of the newer grant are to determine the long-term impact of receiving the original kindergarten intervention and the effects of a middle school booster session of FCU on students' behavior and academic outcomes. Dr. Stormshak is also the director of a NCSER-funded postdoctoral training program focused on the prevention of school-based social and behavioral problems.

Dr. Dorothy Espelage, William C. Friday Distinguished Professor of Education at the University of North Carolina, Chapel Hill, received SPR’s Prevention Science Award. This award is given for the application of scientific methods to develop and test prevention strategies. Dr. Espelage is a leading expert on school safety and has led multiple research studies on school-based violence and bullying. She has been involved in several IES grants. Most recently, she is serving as the Principal Investigator of a 2019 NCSER-funded project to develop and test a professional development program aimed at enhancing elementary school teachers' knowledge and skills for identifying, mitigating, and preventing bullying among students with and without disabilities. Dr. Espelage also led an exploratory study funded by NCER to better understand how teacher practices influence elementary school students’ interpersonal relationships in the classroom and related behavioral outcomes. 

Dr. Patrick H. Tolan, Charles S. Robb Professor of Education at the University of Virginia in the Curry School of Education and in the Department of Psychiatry and Neurobehavioral Sciences in the School of Medicine, and Director Emeritus of the YouthNex Center in the UVA Center to Promote Effective Youth Development, received the Advances in Culture and Diversity in Prevention Science Award as one of three founding members of the Boys of Color Collaborative. This award is given for contributions to the field of prevention science in the area of community and culture. Dr. Tolan’s research career spans 34 years with a focus on program evaluation for promoting positive youth development and preventing youth violence. Dr. Tolan is currently the Principal Investigator of a 2019 Follow-Up study of an IES efficacy study of the integration of two prevention programs, Good Behavior Game and My Teaching Partner (GBG+MTP), for teachers who have recently entered the teaching profession.

Congratulations to the award recipients!

This blog was co-authored by Jackie Buckley (NCSER), Katie Taylor (NCSER), Emily Doolittle (NCER), and Amy Sussman (NCSER).