Inside IES Research

Notes from NCER & NCSER

Announcing the 8th Annual ED Games Expo: June 1 to 5, 2021

A Free All-Virtual Showcase of Game-Changing Innovations in EdTech developed through ED and Programs Across Government

The ED Games Expo is an annual showcase of game-changing innovations in education technology (EdTech) developed through programs at the Department of Education (ED) and across the federal government. Since 2013, the Expo has been an in-person event at venues across Washington, D.C. Because of the COVID-19 national emergency, the 2021 ED Games Expo is moving online, from June 1 – 5, for an entirely virtual experience. Hosting virtually provides the unique opportunity to engage a national audience and to present content mindful of the pandemic and useful for educational programming in the summer and going forward.  

 

ED Games Expo: Featured Resources

A new set of YouTube playlists and an accompanying PDF guide will be released on June 1 to present video trailers for more than 150 participating government-supported learning games and technologies. These learning games and technologies are appropriate for children and students in early childhood to post-secondary education and special education, and cover a range of topics across STEM, reading, social studies, civics, healthy development, and others. Nearly all the resources are research-based – meaning studies demonstrate the usability, feasibility, and promise of leading to the intended outcomes. Many of the education technologies at ED Games Expo will be available to students and educators who are learning in-person or remotely at no cost during June 2021. Attendees will also have the opportunity to engage in virtual Q&A with developers during and after the Expo to learn more.

 

ED Games Expo: A Range of Online Events

The 2021 ED Games Expo Agenda presents the lineup for 35 online events to be broadcast during the weeklong Expo. The events are designed for a wide audience across the education technology ecosystem, including educators, students, parents and caregivers, developers, researchers, and other stakeholders.

Events include:

  • Master Classes for Educators: Eight Expo developers present use-case examples and guidance for implementing innovative education technology interventions to support in-person or remote learning across many different topics.
  • How the Learning Game was Made: Five teams of learning game developers inspire and prompt students to think about the many skills and careers involved in creating a learning game.
  • Showcase Events: More than 20 government agencies and offices that invest in education technology are broadcasting events to showcase their projects and initiatives. Just a few highlights from the week include events on: innovations in early learning and special education, learning games to combat disinformation, models to support remote tutoring, a live kick-off for a new NASA national student challenge, an esports competition with students from Historically Black Colleges and Universities, and forums where government leaders, experts, and practitioners reflect on the role of education technology during the era of COVID-19.
  • A Unique Kick Off Show: This year’s Expo will kick off on June 1, 2021 at 8pm ET with a unique virtual event featuring a few of our favorite children’s TV characters and puppet friends created through ED funded projects.

 

All Expo events are free and accessible to the public to watch online. Content from all events will be archived and available to watch on demand via YouTube after the event. Follow the ED Games Expo on social media @USEdGov and by using the #EDGamesExpo hashtag.

 

We hope you will join us in June!


This Inside IES Blog is crossed-posted on Homeroom, the official blog of the U.S. Department of Education.

Edward Metz is the Program Manager for the Small Business Innovation Research Program at the Institute of Education Sciences in the US Department of Education. Contact Edward.Metz@ed.gov for more information or with questions.

Teachley’s Game Apps for Mathematics: From Research to Practice at Scale in Education

With a 2010 IES research grant, researchers at Teachers College, Columbia University conducted basic research and created prototype software programs for children in mathematics. In 2011, three members of the research team launched a startup and submitted a successful proposal to IES’s Small Business Innovation Research programs. With awards in 2012 and 2013, the developers created a suite of math game apps that support fact fluency and promote math strategy development. The apps all connect with a teacher dashboard that provides in-depth reports in real time and supports differentiation in math instruction. In 2013, Teachley’s Addimal Adventure won an Apple Design Award as one of the 12 best apps of 2014. Since their commercial launch in 2014, Teachley Apps have been downloaded 1.5 million times, and the Teachley suite of products are currently used in all 50 states and 2,000 schools.

Interview with Kara Carpenter, co-founder of Teachley

 

 

The three co-founders of Teachley were all classroom teachers before you met at Teachers College as graduate students in 2010. What led to your decision to go to graduate school to earn PhDs as researchers?

While teaching 2nd grade, I had the opportunity to receive professional development focused on elementary math content, and I became fascinated with how children develop their mathematical thinking. Years later, when I was getting a master’s in curriculum & teaching at Teachers College, I pursued a work study opportunity with Professor Herb Ginsburg, who focuses on early childhood math thinking. At the time in 2009, my cofounder Rachael Labrecque was already working with Professor Ginsburg, and the three of us submitted an application to IES to develop math software for young learners. That fall, I went back to classroom teaching, but when the application was funded in 2010, I decided to take the leap and accept a research fellowship to pursue a PhD. My other co-founder, Dana Pagar, joined our research team that fall, and the three of us decided to start Teachley in 2012 to bring all the great research on how kids learn math into marketable products.

 

Tell us about the research projects that you were involved with in graduate school.

We worked on a project developing math software for grades pre-K to 3, called MathemAntics. We developed dozens of activities and conducted small learning studies along the way. In the third year, we conducted an RCT with approximately 400 students in grades PreK - 2. Each of our dissertations involved different elements of the project. Mine focused on teaching and detecting kids’ single-digit addition strategies. Dana’s focused on continuous versus discrete blocks, while Rachael studied teachers’ preparedness to integrate technology into their classrooms.

 

How did you come up with the idea to develop apps that would be used in schools on a wide scale basis?
Originally, we were looking for a company who might want to take these research findings and turn them into commercial products. We were meeting with various business leaders, and one of them turned to us and said, “You should do this. You should start a company to bring your ideas to market.” That’s the push we needed to think of ourselves as potential startup founders.

 

How did you find out about the SBIR program at the US Department of Education’s Institute of Education Sciences? How important was the first SBIR award for launching Teachley?

Once we decided to start Teachley, we knew that SBIR would be a great resource for us. The MathemAntics project had actually started out as an NIH SBIR Phase I with a different company. That first ED/IES SBIR award is the reason that Teachley became a company. Without that funding, we would not have been able to prove ourselves capable of bringing a product to market. Institutional investors aren’t taking those kinds of risks, and angel investment is too tied into social networks and who you know.

 

Was Teachers College supportive of its graduate students starting a small business and getting an award to develop apps? Did anyone at the university offer advice or guidance on how to operate a small business?

Leaving the university was tricky because we had research fellowships when we started the company. However, the Teachers College president at the time, Susan Fuhrman, and the provost, Tom James, were supportive of our startup. We speak and participate in various discussions and events at Teachers College, which keeps us connected to the university and the research.

 

How does Teachley ensure that research is integrated into your development and validation process?

Before developing any new product idea, we look to the research to see what’s already been learned about the topic, especially as it relates to struggling learners. During the early stages of development, we rely on close observations of students as they use pencil/paper mockups and early software builds. As a team, we closely review videos of students working through problems, looking to find better, more intuitive ways to support students’ thinking. Once we have a functional prototype, we use more formal evaluative techniques to determine our impact on student learning.

 

What models have you used to commercialize Teachley on a widespread basis?

We have tried out many different revenue models. Initially, we tried publishing the games for free and charging schools for the formative assessment data. However, we soon found that bundling the games and data together into a single subscription worked better for schools. With our latest game, Market Bay, we are trying a new model where educators create a free account, and parents subscribe to have access at home. Schools who subscribe to Teachley get home access to Market Bay and our other games for all of their families.

 

Have you raised funds from venture capitalists? Why or why not?

Not yet. Raising money from venture capitalists can put you on a succeed-or-fail-fast treadmill that isn’t always a great fit for the education market. Many investors are looking for a 70x return within just a few years or they abandon ship. Developing great educational software takes time for both the iterative design process and the research to prove your effectiveness. We are just now at the stage where raising venture capital may soon make sense because we have enough content to scale our school/district sales.

 

When COVID-19 emerged and schools closed, you made your apps freely available to teachers and students in their classes, and 15,000 teachers and students were able to access your products. What was that experience like?

Teachers are looking for digital products that will deeply engage students and support true learning. We’re a great fit. However, schools across the country are suffering budget shortfalls at the same time as they need to spend more to ensure they meet safety standards. We’re working with schools and teachers to find alternative ways to fund our program, from parent organizations to Donors Choose to corporate partnerships.

 

None of you had had formal business training prior to founding Teachley. Do you have advice for those who are interested in starting an entrepreneurial small business to develop education technology that can be used in schools?

My advice would be to know your users and implementation deeply. If you don’t have a background in teaching, spend time volunteering in schools. Become a close observer of children and their thinking, so you can create products that support and bring out children’s genius.

 

 ____________________________________________________________________________

Kara Carpenter is cofounder of Teachley (@teachley), an edtech startup focused on promoting deep math thinking and learning. Kara has over 10 years of teaching experience and was a National Board Certified Teacher with a PhD in Cognition and Learning from Teachers College, Columbia University. Her dissertation went on to become an Apple Design Award winning app, Addimal Adventure.

This interview was produced by Ed Metz (Edward.Metz@ed.gov) of the Institute of Education Sciences. This post is the sixth in an ongoing series of blog posts examining moving from university research to practice at scale in education.

 

 

ASSISTments: From Research to Practice at Scale in Education

ASSISTments is a free, web-based formative assessment platform for teachers and students in Grades 3 through 12. The tool is designed for teachers to easily assign students math problems from OER textbooks such as Illustrative Math and EngageNY, existing item banks, or items they have developed on their own. ASSISTments will continually assess students as they solve math problems and provide immediate feedback and the chance to try again. The computer-generated reports provide teachers with information to make real-time adjustments to their instruction. Teachers can use it with their school’s existing learner management systems, such as Google Classroom and Canvas. Watch a video here.

 

 

Over the past 13 years, ASSISTments was developed and evaluated with the support of a series of IES and National Science Foundation awards. With a 2003 IES award to Carnegie Mellon University and Worcester Polytechnic Institute (WPI), researchers created the first version of ASSISTments. The system was populated with Massachusetts high-stakes mathematics test questions and the tutoring for the questions was authored by WPI staff with assistance from local teachers. After students completed problems assigned by the teacher, reports provided teachers with information about question difficulty and the most commonly submitted wrong answers, initiating class discussions around the completed assignments. In 2007, researchers at WPI received an award to build additional functionalities in the ASSISTments program so that teachers could assign supports (called “skill builders”) to students to help them master content.  An additional eight grants allowed the researchers to create other features. 

With a 2012 IES research grant award, SRI evaluated the efficacy of the ASSISTments program as a homework tool for academic learning.  In the study, the researchers took all 7th grade textbooks in the State of Maine and added answers to homework problems into ASSISTments.  The results of the efficacy trial demonstrated that teachers changed their homework reviewing behavior, mathematical learning improved an extra three quarters of a year of schooling, and using ASSISTments reliably closed achievement gaps for students with different achievement levels. ASSISTments is currently being evaluated again through two IES studies, with over 120 schools, to attempt to replicate this result. To view all publications related to ASSISTments, see here.

As of 2020, ASSISTments has been used by approximately 60,000 students with over 12 million problems solved.

 

Interview with Neil Heffernan and Cristina Heffernan

From the start of the project, was it always a goal that ASSISTments would one day be used on a wide scale?

We created ASSISTments to help as many teachers and students as possible. After we learned that the ASSISTments intervention was effective, we set the goal to have every middle school student in the country get immediate feedback on their homework. We created ASSISTments to be used by real teachers and have been improving it with each grant. Because of the effectiveness of ASSISTments, we kept getting funded to make improvements allowing our user base to grow.

At what point was ASSISTments ready to be used at a large scale in schools?

We were ready in year one because of the simplicity of our software. Now that we integrated seamlessly with Google Classroom, most teachers can use the system without training!

ASSISTments is backed by a lot of research, which would make some think that it would be easy for many schools to adopt. What were (or are) the biggest obstacles to ASSISTments being used in more schools?

A big obstacle has been access to technology for all students. The current environment in schools is making that less and less of a barrier. Now, teachers are looking for effective ways to use the computers they have.      

What options did you consider to begin distributing ASSISTments?

We had major companies try to buy us out, but we turned them all down. We knew the value was being in control so we could run research studies, let others run research studies and AB test new ideas. It was important to us to keep ASSISTments free to teachers. It is also a necessity since we crowdsource from teachers.

How do you do marketing?

Our biggest obstacle is marketing. But we are lucky to have just received $1 million in funding from a philanthropy to create a nonprofit to support the work of making our product accessible. Foundation funding has allowed us to hire staff members to write marketing materials including a new website, op-eds, blog posts and press releases. In addition to our internal marketing staff member, we work extensively with The Learning Agency to get press and foundation support for ASSISTments.

What costs are associated with the launch and distribution of ASSISTments, including marketing? Will a revenue model needed sustain ASSISTments over time?

When creating ASSISTments, we didn’t want a traditional business model based on schools paying. Our vision for future growth, instead, focused on crowdsourcing ideas from teachers and testing them. We are trying to replicate the Wikimedia platform idea created by Jimmy Wales. He crowdsources the content that makes up the encyclopedia, so it must be free. We envision using ASSISTments to help us crowdsource hints and explanations for all the commonly used questions in middle school mathematics.  

Do you have any agreement about the IP with the universities where ASSISTments was developed?

The ASSISTments Foundation was founded in 2019 and supports our project work in tandem with Worcester Polytechnic Institute due to our integration with research. The close relationship takes care of any issues that would arise with intellectual property. Additionally, the fact that we are a nonprofit helps address these issues.

How do you describe the experience of commercializing ASSISTments? What would you say is most needed for gaining traction in the marketplace?

Even though we are free, we do have several competitors. To gain traction, we have found that word of mouth is an effective disseminator and our positive efficacy trial result. Currently, there are many teachers on Facebook sharing how much they like ASSISTments. We also attend conferences and are working on an email campaign to get new users onboard.

Do you have advice for university researchers seeking to move their laboratory research into widespread practice?

Make sure your work is accessible and meaningful! We are solving a super-pervasive problem of homework in schools. Everyone finds meaning in making homework better.


Neil Heffernan (@NeilTHeffernan) is a professor of computer science and Director of Learning Sciences and Technologies at Worcester Polytechnic Institute.  He developed ASSISTments not only to help teachers be more effective in the classroom but also so that he could use the platform to conduct studies to improve the quality of education.  

Cristina Heffernan (@CristinaHeff) is the Lead Strategist for the ASSISTments Project at WPI. She began her career in education as a math teacher in the Peace Corps and after went on to teach middle school math in the US.  She began working with teachers while a graduate student at the University of Pittsburgh. As one of the co-founders of ASSISTments, Cristina has nurtured the system to be a tool for teachers to improve what they already do well in the classroom. 


This interview was produced by Edward Metz of the Institute of Education Sciences. This is the fourth in an ongoing series of blog posts examining moving from university research to practice at scale in education.​

Education Technology Platforms to Enable Efficient Education Research

Education research is often a slow and costly process. An even more difficult challenge is replicating research findings in a timely and cost-effective way to ensure that they are meaningful for the wide range of contexts and populations that make up our nation’s school system.

In a recent op-ed, IES Director Mark Schneider and Schmidt Futures Senior Director for Technology and Society Kumar Garg pitched the idea that digital learning platforms may be a way to accelerate the research enterprise. These platforms will enable researchers to try new ideas and replicate interventions quickly across many sites and with a wide range of student populations. They could also open the door for educators to get more involved in the research process. For example, Learn Platform supports districts as they make decisions about purchasing and implementing products in their schools, and ASSISTments provides infrastructure for researchers to conduct cheaper and faster studies than they would be able to do on their own.

IES Director Mark Schneider and NCER Commissioner Liz Albro recently attended a meeting sponsored by Schmidt Futures focused on these issues. Two major takeaways from the meeting: first, there is already progress on building and using platforms for testing interventions, and, second, platform developers are enthusiastic about integrating research capabilities into their work.

As we consider how we can support platform developers, researchers, and education personnel to co-design tools to enable more efficient, large scale research on digital learning platforms, several questions have arisen:  

  1. What digital learning platforms already have a large enough user base to support large scale research studies?
  2. Are there content areas or grade levels that are not well supported through digital learning platforms?
  3. What are the key features that a platform needs to have to support rigorous tests and rapid replication of research findings? 
  4. What are the barriers and challenges for companies interested in participating in this effort?
  5. What kinds of research questions can best be answered in this research environment?
  6. What kind of infrastructure needs to be developed around the platform to enable seamless collaborations between education stakeholders, researchers, and product developers?

We know there are some of you have already given these questions some thought. In addition, there are other questions and issues that we haven’t considered. We welcome your thoughts. Feel free to email us at Erin.Higgins@ed.gov and Elizabeth.Albro@ed.gov. And join NCER’s Virtual Learning Lab in their virtual workshop “Designing Online Learning Platforms to Enable Research” on April 17th, 3:00pm-5:00pm Eastern Time. Learn more about the workshop here.

A2i: From Research to Practice at Scale in Education

This blog post is part of an interview series with education researchers who have successfully scaled their interventions.

Assessment-to-Instruction (A2i) is an online Teacher Professional Support System that guides teachers in providing Kindergarten to Grade 3 students individualized literacy instruction and assessments. Students complete the assessments independently online without the teacher taking time away from instruction. A2i generates instantaneous teacher reports with precise recommendations for each student and group recommendations. See a video demo here. Between 2003 and 2017, researchers at Florida State University (FSU) and Arizona State University (ASU), led by Carol Connor, developed and evaluated A2i with the support of a series of awards from IES and the National Institutes of Health. Findings from all publications on the A2i are posted here.

While results across seven controlled studies demonstrated the effectiveness of A2i, feedback from practitioners in the field demonstrated that implementation often required substantial amounts of researcher support and local district adaptation, and that the cost was not sustainable for most school district budgets. In 2014, the development firm Learning Ovations, led by Jay Connor, received an award from the Department of Education (ED) and IES’s Small Business Innovation Research program (ED/IES SBIR) to develop an technologically upgraded and commercially viable version of A2i to be ready to be used at scale in classrooms around the country. In 2018, with the support of a five-year Education Innovation and Research (EIR) expansion grant from ED totaling $14.65 million, A2i is now used in more than 110 schools across the country, with plans for further expansion. 

 

Interview with Carol Connor (CC) and Jay Connor (JC)

From the start of the research in the early 2000s, was it always the goal to develop a reading intervention that would one day be used on a wide scale?
CC: Yes and no. First, we had to answer the question as to whether individualization was effective in achieving student literacy outcomes. Once the research established that, we knew that this work would have wide-scale application.

When did you start thinking about a plan for distribution
CC: Before embarking on the cumulative results studies, in 2008, Jay said that we needed to know who the “customer” was… i.e., how purchasing decisions were made at scale.  His 2008 Phase I ED/IES SBIR was critical in shifting our research focus from individual classrooms to school districts as the key scaling point. 

Did you work with a technology transfer office at the university?
CC: Only to the extent of contractually clarifying intellectual property (IP) ownership and licensing. 

Who provided the support on the business side?
CC: Jay, who has an MBA/JD and has been a senior officer in two Fortune 100 companies was very instrumental in guiding our thinking of this evolution from important research to practical application. 


 Do you have any agreement about the IP with the university? What were the biggest challenges in this area?

JC: Yes, Learning Ovations has a 60-year renewable exclusive licensing agreement with FSU Foundation. FSU couldn’t have been better to work with.  Though there were expected back-and-forth elements of the original negotiations, it was clear that we shared the central vision of transforming literacy outcomes.  They continue to be a meaningful partner.

When and why was Learning Ovations first launched?
JC: In order to pursue SBIR funding we needed to be a for-profit company.  At first, I used my consulting business – Rubicon Partners LLP – as the legal entity for a 2008 Phase I award from ED/IES SBIR.  When we considered applying (and eventually won) a Fast Track Phase I & II award from SBIR in 2014, it was clear that we needed to create a full C – Corp that could expand with the scaling of the business, thus Learning Ovations was formed.

Who has provided you great guidance on the business side over the year? What did they say and do? 
JC: Having run large corporate entities and worked with small business start-ups in conjunction with Arizona State University (Skysong) and the University of California, Irvine (Applied Innovation at The Cove) and having taught entrepreneurship at The Paul Merage School of Business at UC Irvine, I had the experience or network to connect to whatever business guidance we needed.  Further, having attended a number of reading research conferences with Carol, I was quite conversant in the literacy language both from the research side and from the district decision maker’s side.

How do you describe the experience of commercializing the A2i? What were the biggest achievements and challenges in terms of preparing for commercialization?

JC: Having coached scores of entrepreneurs at various stages, I can safely say that there is no harder commercialization than one that must stay faithful to the underlying research.  A key strategy for most new businesses: being able to pivot as you find a better (easier) solution.  It is often circumscribed by the “active ingredients” of the underlying research.  Knowing this, we imbued Learning Ovations with a very strong outcomes mission – all children reading at, or above, grade level by 3rd grade.  This commitment to outcomes certainty is only assured by staying faithful to the research.  Thus, a possible constraint, became our uncontroverted strength.

Do you have advice for university researchers seeking to move their laboratory research in education into wide-spread practice? 
JC:  Start with the end in mind.  As soon as you envision wide-scale usage, learn as much as you can about the present pain and needs of your future users and frame your research questions to speak to this.  Implementation should not be an after-the-fact consideration; build it into how you frame your research questions. On one level you are asking simultaneously “will this work with my treatment group” AND “will this help me understand/deliver to my end-user group.”  I can’t imagine effective research being graphed onto a business after the fact.  One key risk that we see a number of researchers make is thinking in very small fragments whereas application (i.e., the ability to go to scale) is usually much more systemic and holistic.

In one sentence, what would say is most needed for gaining traction in the marketplace?
JC: If not you, as a researcher, someone on your team of advisors needs to know the target marketplace as well as you know the treatment protocols in your RCT.

____________

Carol Connor is a Chancellor’s Professor in the UC Irvine School of Education. Prior she was a professor of Psychology and a Senior Learning Scientist at the Learning Sciences Institute at ASU. Carol’s research focuses on teaching and learning in preschool through fifth grade classrooms – with a particular emphasis on reading comprehension, executive functioning, and behavioral regulation development, especially for low-income children.

Joseph “Jay” Connor, JD/MBA, is the Founder/CEO of Learning Ovations, Inc, the developer of the platform that has enabled the A2i intervention to scale.  Jay has 20+ years of experience in senior business management at the multi-billion dollar corporate level, and has experience in the nonprofit and public policy arenas.

This interview was produced by Edward Metz of the Institute of Education Sciences.