Inside IES Research

Many programs across the Federal government, such as the ED/IES Small Business Innovation Research (SBIR) and the IES Research Grants programs, fund projects to develop and evaluate new forms of education technology and interventions that can be implemented to support instruction and learning at schools and for remote learning. More than 150 of these technologies were demoed in January 2020 at the ED Games Expo, a showcase for learning games and technologies developed with support from IES and more than 30 other Federal programs.

Since the global outbreak of COVID19 and the closure of schools across the United States and the world, a group of government-supported developers and researchers responded to provide resources to educators, students, and families to facilitate remote learning. More than 50 developers and researchers offered 88 learning games and technologies at no cost through the end of the school year for use in distance learning settings with internet access (see this blog for the list). In addition, many of the developers and researchers provided technical assistance directly to individual teachers to support implementation at a distance, and many created new materials and worked to refine and adapt their products to optimize usability and feasibility for fully remote use. More than a million students and thousands of educators used these learning technologies during the spring.

In April and May 2020, more than 70 developers and researchers partnered to produce and participate in a series of free day-long virtual events, which were called “unconferences.” The events featured presentations on innovative models and approaches to teaching and learning remotely and provided an in-depth look at the learning games and technologies created by the presenters. More than 25,000 educators attended these virtual events in real-time, hundreds asked questions and made comments through chats during the events, and many thousands more have accessed these videos after the events. See this blog for the list of archived videos.

A New Resource: Guides to Education Technologies that are Ready Now

As schools begin re-opening for the new school year, a group of 70 developers and researchers have collaborated to produce a new series of Guides to Education Technologies. The guides present information on government-supported education technology products that are ready now for in-class and remote learning. All the resources are web-based and can be used on either computers, tablets, or personal devices. The resources in the guides include a mix of no-cost products as well as ones that are fee-based.  

With awards from government programs, all of the resources were developed through an iterative process with feedback from teachers and students, and most were evaluated through small pilot studies to measure the promise of the technologies to support improvements in student learning and relevant educational outcomes. All the products were used and demonstrated to be feasible for use in remote settings in the spring after the onset of the pandemic.

The guides present resources appropriate for young children through postsecondary students in education and special education, for English learners, and for teachers in education and special education across a wide range of educational topics. Many of the technologies personalize learning by adjusting content to students as they go and present information to educators to inform instruction.

The Guides focus on the following areas and can be accessed below:

• Early Learning
• Mathematics
• Science and Engineering
• Social Studies
• Special Education

Stay tuned to the Inside IES Blog for more information and resources about the response to the COVID-19 in education.

Edward Metz (Edward.Metz@ed.gov) is a research scientist and the program manager for the Small Business Innovation Research Program at the US Department of Education’s Institute of Education Sciences.

Editor’s Note: This Inside IES Blog is crossed-posted on Homeroom, the official blog of the U.S. Department of Education.

On July 30, 2020, NASA launched a rocket from Cape Canaveral Air Force Station in Florida on a journey to Mars. The rocket is carrying a rover named Perseverance and a helicopter named Ingenuity, both of which will land inside Mars's Jezero Crater on February 18, 2021. While on Mars, Perseverance and Ingenuity will collect the first Martian soil and rock samples for future return to Earth, search for signs of extinct or extant life, characterize the planet’s climate and geology, and pave the way for human exploration of the Red Planet with the help of new technologies and scientific instruments.

Perseverance and Ingenuity were named by students through a national Kindergarten to Grade 12 student competition run by NASA in partnership with Future Engineers and Battelle Education.

The student whose entry won the prize to name the rover is Alexander Mather, a seventh grader from Lake Braddock Secondary School in Burke, Virginia. Alexander submitted the name Perseverance and included the following in his essay:

“Curiosity. Insight. Spirit. Opportunity. If you think about it, all of these names of past Mars rovers are qualities we possess as humans. We are always curious, and seek opportunity. We have the spirit and insight to explore the Moon, Mars, and beyond. But, if rovers are to be the qualities of us as a race, we missed the most important thing. Perseverance.”

Watch the March 5 program where the winning name was revealed here:

The student whose entry won the prize to name the helicopter is Vaneeza Rupani, a junior at Tuscaloosa County High School in Northport, Alabama. Vanessa submitted the name Ingenuity and included the following in her essay:

"The ingenuity and brilliance of people working hard to overcome the challenges of interplanetary travel are what allow us all to experience the wonders of space exploration. Ingenuity is what allows people to accomplish amazing things, and it allows us to expand our horizons to the edges of the universe."

Watch the video trailer featuring the naming of the Mars helicopter: 

About the “Name the Rover” Contest

Not only did the contest help NASA pick a new name for the rover, it also engaged U.S. students in the engineering and scientific work that makes Mars exploration possible, stimulated interest in science, technology, engineering, and mathematics (STEM), and inspired the next generation of STEM leaders.

After launching the competition in August 2019, students from 50 states, U.S. territories, and military bases submitted over 28,000 essays. More than 4,500 volunteer judges narrowed the pool to 155 semifinalists. From these, NASA chose nine finalists—Clarity, Courage, Endurance, Fortitude, Ingenuity, Perseverance, Promise, Tenacity, and Vision—and opened a public poll in which anyone could vote. After considering these poll results, NASA officials chose the two names.

To manage the competition, NASA used a web-based platform developed by Burbank, California-based Future Engineers.  This platform was created with the support of a 2017 award from the U.S. Department of Education and Institute of Education Sciences’ Small Business Innovation Research program (ED/IES SBIR).  Future Engineers built this platform to be an online hub for classrooms and educators to access free, project-based STEM activities and to provide a portal where students submit and compete in different kinds of maker and innovation challenges across the country. The Mars 2020 “Name the Rover” contest was the first naming challenge issued on the platform. We look forward to more student challenges to come!

Edward Metz (Edward.Metz@ed.gov) is a research scientist at the Institute of Education Sciences in the US Department of Education.

Bob Collom is an integration lead in the Mars Exploration Program at NASA Headquarters.

About ED/IES SBIR

The U.S. Department of Education’s Small Business Innovation Research program, administered by the Institute of Education Sciences (IES), funds projects to develop education technology products designed to support students, teachers, or administrators in general or special education. The program emphasizes rigorous and relevant research to inform iterative development and to evaluate whether fully developed products show promise for leading to the intended outcomes. The program also focuses on commercialization once the award period ends so that products can reach students and teachers and be sustained over time. ED/IES SBIR-supported products are currently used by millions of students in thousands of schools around the country.

About NASA’s Mars Exploration Program (MEP)

NASA’s Mars Exploration Program (MEP) in the Planetary Science Division is a science-driven program that seeks to understand whether Mars was, is, or can be, a habitable world. To find out, we need to understand how geologic, climatic, and other processes have worked to shape Mars and its environment over time, as well as how they interact today. To that end, all of our future missions will be driven by rigorous scientific questions that will continuously evolve as we make new discoveries. MEP continues to explore Mars and to provide a continuous flow of scientific information and discovery through a carefully selected series of robotic orbiters, landers and mobile laboratories interconnected by a high-bandwidth Mars/Earth communications network. The Mars 2020 Project at NASA’s Jet Propulsion Laboratory manages rover development for NASA’s Science Mission Directorate. NASA’s Launch Services Program, based at the agency’s Kennedy Space Center in Florida, is responsible for launch management.

With a 2010 IES research grant, researchers at Teachers College, Columbia University conducted basic research and created prototype software programs for children in mathematics. In 2011, three members of the research team launched a startup and submitted a successful proposal to IES’s Small Business Innovation Research programs. With awards in 2012 and 2013, the developers created a suite of math game apps that support fact fluency and promote math strategy development. The apps all connect with a teacher dashboard that provides in-depth reports in real time and supports differentiation in math instruction. In 2013, Teachley’s Addimal Adventure won an Apple Design Award as one of the 12 best apps of 2014. Since their commercial launch in 2014, Teachley Apps have been downloaded 1.5 million times, and the Teachley suite of products are currently used in all 50 states and 2,000 schools.

Interview with Kara Carpenter, co-founder of Teachley

The three co-founders of Teachley were all classroom teachers before you met at Teachers College as graduate students in 2010. What led to your decision to go to graduate school to earn PhDs as researchers?

While teaching 2nd grade, I had the opportunity to receive professional development focused on elementary math content, and I became fascinated with how children develop their mathematical thinking. Years later, when I was getting a master’s in curriculum & teaching at Teachers College, I pursued a work study opportunity with Professor Herb Ginsburg, who focuses on early childhood math thinking. At the time in 2009, my cofounder Rachael Labrecque was already working with Professor Ginsburg, and the three of us submitted an application to IES to develop math software for young learners. That fall, I went back to classroom teaching, but when the application was funded in 2010, I decided to take the leap and accept a research fellowship to pursue a PhD. My other co-founder, Dana Pagar, joined our research team that fall, and the three of us decided to start Teachley in 2012 to bring all the great research on how kids learn math into marketable products.

Tell us about the research projects that you were involved with in graduate school.

We worked on a project developing math software for grades pre-K to 3, called MathemAntics. We developed dozens of activities and conducted small learning studies along the way. In the third year, we conducted an RCT with approximately 400 students in grades PreK - 2. Each of our dissertations involved different elements of the project. Mine focused on teaching and detecting kids’ single-digit addition strategies. Dana’s focused on continuous versus discrete blocks, while Rachael studied teachers’ preparedness to integrate technology into their classrooms.

How did you come up with the idea to develop apps that would be used in schools on a wide scale basis?
Originally, we were looking for a company who might want to take these research findings and turn them into commercial products. We were meeting with various business leaders, and one of them turned to us and said, “You should do this. You should start a company to bring your ideas to market.” That’s the push we needed to think of ourselves as potential startup founders.

How did you find out about the SBIR program at the US Department of Education’s Institute of Education Sciences? How important was the first SBIR award for launching Teachley?

Once we decided to start Teachley, we knew that SBIR would be a great resource for us. The MathemAntics project had actually started out as an NIH SBIR Phase I with a different company. That first ED/IES SBIR award is the reason that Teachley became a company. Without that funding, we would not have been able to prove ourselves capable of bringing a product to market. Institutional investors aren’t taking those kinds of risks, and angel investment is too tied into social networks and who you know.

Was Teachers College supportive of its graduate students starting a small business and getting an award to develop apps? Did anyone at the university offer advice or guidance on how to operate a small business?

Leaving the university was tricky because we had research fellowships when we started the company. However, the Teachers College president at the time, Susan Fuhrman, and the provost, Tom James, were supportive of our startup. We speak and participate in various discussions and events at Teachers College, which keeps us connected to the university and the research.

How does Teachley ensure that research is integrated into your development and validation process?

Before developing any new product idea, we look to the research to see what’s already been learned about the topic, especially as it relates to struggling learners. During the early stages of development, we rely on close observations of students as they use pencil/paper mockups and early software builds. As a team, we closely review videos of students working through problems, looking to find better, more intuitive ways to support students’ thinking. Once we have a functional prototype, we use more formal evaluative techniques to determine our impact on student learning.

What models have you used to commercialize Teachley on a widespread basis?

We have tried out many different revenue models. Initially, we tried publishing the games for free and charging schools for the formative assessment data. However, we soon found that bundling the games and data together into a single subscription worked better for schools. With our latest game, Market Bay, we are trying a new model where educators create a free account, and parents subscribe to have access at home. Schools who subscribe to Teachley get home access to Market Bay and our other games for all of their families.

Have you raised funds from venture capitalists? Why or why not?

Not yet. Raising money from venture capitalists can put you on a succeed-or-fail-fast treadmill that isn’t always a great fit for the education market. Many investors are looking for a 70x return within just a few years or they abandon ship. Developing great educational software takes time for both the iterative design process and the research to prove your effectiveness. We are just now at the stage where raising venture capital may soon make sense because we have enough content to scale our school/district sales.

When COVID-19 emerged and schools closed, you made your apps freely available to teachers and students in their classes, and 15,000 teachers and students were able to access your products. What was that experience like?

Teachers are looking for digital products that will deeply engage students and support true learning. We’re a great fit. However, schools across the country are suffering budget shortfalls at the same time as they need to spend more to ensure they meet safety standards. We’re working with schools and teachers to find alternative ways to fund our program, from parent organizations to Donors Choose to corporate partnerships.

None of you had had formal business training prior to founding Teachley. Do you have advice for those who are interested in starting an entrepreneurial small business to develop education technology that can be used in schools?

My advice would be to know your users and implementation deeply. If you don’t have a background in teaching, spend time volunteering in schools. Become a close observer of children and their thinking, so you can create products that support and bring out children’s genius.

 ____________________________________________________________________________

Kara Carpenter is cofounder of Teachley (@teachley), an edtech startup focused on promoting deep math thinking and learning. Kara has over 10 years of teaching experience and was a National Board Certified Teacher with a PhD in Cognition and Learning from Teachers College, Columbia University. Her dissertation went on to become an Apple Design Award winning app, Addimal Adventure.

This interview was produced by Ed Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This post is the sixth in an ongoing series of blog posts examining moving from university research to practice at scale in education.

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

STAGES FOR CONDUCTING A COST ANALYSIS

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

*More resources available here.

The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

Every student makes mistakes. But not every student is given the opportunity to learn from mistakes. Left unaddressed, the mathematical misconceptions that underlie many mistakes can keep students from progressing in mathematics.

At the request of districts in the Minority Student Achievement Network (MSAN), a Strategic Education Research Partnership (SERP) team was convened in 2007 to address a widening achievement gap in Algebra I. The team was charged with identifying an intervention strategy, subject to several district constraints:

1. The solution would need to be applied to all students in the regular classroom to avoid the stereotype threat associated with separating students based on performance and to protect the intervention from budget cuts that target supplemental, after-school, and summer programs first.
2. A new curriculum was off the table because it would create upheaval for a time and would be followed by a decline in student performance during the period of adjustment.
3. Extensive teacher training was considered undesirable because it would be costly and because algebra teachers consider themselves more expert in mathematics teaching than central office staff who would be requiring the training.

Julie Booth joined the partnership, and with funding from IES, led the iterative development and testing of worked example assignments that, with the input of teachers and administrators, fit within the routines of the classroom. The result—AlgebraByExample—consists of 42 uniquely designed assignments that address misconceptions, harness the power of explanation, and use mistakes as a vehicle for learning.

Typical math assignments require students to solve problems on their own. If a student’s work is incorrect, the student may never focus on what went wrong. ByExample assignments also give students problems to solve, but they first provide a solution to a similar problem that is marked right or wrong. Students are prompted with questions that target common misconceptions and errors before solving a similar problem on their own. Each assignment contains several strategically designed item pairs:

Designed in collaboration with teachers from districts in several states, the assignments can be easily incorporated into any Algebra I curriculum and teachers can choose in what way and in what order to use them. The assignments were tested in randomized trials in classrooms in eight districts with more than 6,000 students. Not only did students using AlgebraByExample improve an average of 7 percentage points on an assessment of standardized test items, students at the lower end of the distribution improved the most. The PDF downloads of the assignments are freely available for anyone to use.

The success of AlgebraByExample  led to  further IES funding of MathByExample for Grades 4 and 5 and GeometryByExample for high school geometry .

Resources:

AlgebraByExample website

MathByExample website

Booth et al, 2015

NSF Stem for All Video Submission 2019

Interview with Dr. Suzanne Donovan (SERP), Dr. Julie Booth (Temple University), and Allie Huyghe (SERP), the developers of the ByExample interventions.

Was it part of the original plan to develop an intervention that could one day be used at scale in schools?

Yes. SERP partnerships begin with problems of practice nominated by district partners, but the partnership agreement distinguishes SERP from a consultant. The intention from the start is to frame the problem and design a solution that can be used at scale. SERP has developed in-house, user-centered design expertise so that resources (such as the ByExample products) developed through partnerships meet the needs of teachers and students. Products scale when they improve the experience of teachers and students. Both the model and the internal design capacity allow SERP to move from problem framing through research, development, and dissemination of a product with IES grant funding.

Describe the initial research and development that occurred.

Dr. Julie Booth drafted initial assignments drawing on the mathematics misconceptions literature. SERP held regular partnership meetings with teachers and administrators at which assignments were reviewed and additional misconceptions were nominated for attention in the assignments. Administrators agreed to randomization of the assignments across classrooms and within-teacher. Assignments were first tested in individual topic blocks and revised in accordance with student performance data, observations, and teacher feedback. A year-long pilot study was then conducted using the full set of assignments.

Beyond IES or ED grants, what additional funding was needed to develop the intervention?

For the ByExample work, additional funding was provided by the Goldman Sachs Foundation in the initial phase to support partnership formation, problem framing, and the solution generation. IES grants funded the research and development, along with initial dissemination activities to make the materials available to the public. We were also able to develop an online platform to allow for digital use with the IES grant funds.

What model was used for dissemination and sustainability?

The assignments are available as free downloads on SERP’s website, and as printed workbooks through SERP’s partner print-on-demand company. They have been publicized through online communications, journal articles, presentations at conferences of various types, social media, and word of mouth. There will be a small fee for use of the digital platform to support its maintenance, but the PDFs will remain as free downloads. We have been able to sustain the collaboration of the partnership team by responding to requests from educators to expand the approach to other grade levels and submitting additional proposals to IES that have been awarded.

What advice would you provide to researchers who are looking to move their research from the lab to market? What steps should they take? What resources should they look for?

First, I would note that it is difficult to persuade educators to use a product that solves a problem they don’t believe they have. Listen to educators and apply research expertise to address the challenges that they experience on a day-to-day basis. Design for ease of use by teachers. No matter how good your strategy or marketing is, if it’s too much work for an already busy teacher to use, you may get uptake by a few committed teachers, but not at scale. Finally, pay attention to where teachers get their information. For AlgebraByExample, we got a big boost from the Marshall Report, produced by a teacher for other teachers to call attention to usable research.  

In one sentence, what would you say is most needed for gaining traction and wide scale use by educators?

Design for the routines of the classroom.

Suzanne Donovan, PhD, is the founding Executive Director of the SERP Institute, an education research, development, and implementation organization incubated at the National Academies. SERP leads collaborations of educators, researchers, and designers to generate research-based, scalable, and sustainable solutions to critical problems of practice. 

Julie Booth, PhD, is a Professor of STEM Education and Psychology and the Deputy Dean of Academic and Faculty Affairs at Temple University’s College of Education and Human Development. Her work focuses on translating between cognitive science and education to better understand students’ learning and improve instruction, primarily in mathematics education. She is currently an Executive Editor for the Journal of Experimental Education.

Allie Huyghe is the Assistant Director of the SERP Institute, where she manages several projects, including the IES-funded MathbyExample and GeometryByExample projects. She is also intricately involved with other SERP areas of work, participating in the design of materials from early development through release to the public.

This interview was produced by Christina Chhin (Christina.Chhin@ed.gov) and Edward Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This is the fifth in an ongoing series of blog posts examining moving from university research to practice at scale in education.​