Inside IES Research

Notes from NCER & NCSER

SELweb: From Research to Practice at Scale in Education

With a 2011 IES development grant, researchers at Rush University Medical Center, led by Clark McKown, created SELweb, a web-based system to assess the social-emotional skills in children in Kindergarten to Grade 3. The system (watch the video demo) includes illustrated and narrated modules that gauge children’s social acceptance with peers and assess their ability to understand others’ emotions and perspectives, solve social problems, and self-regulate. The system generates teacher reports with norm-referenced scores and classroom social network maps. Field trials with 8,881 children in seven states demonstrate that system produces reliable and valid measures of social-emotional skills. Findings from all publications on SELweb are posted here.

In 2016, with support from the university, McKown launched a company called xSEL Labs, to further develop and ready SELweb for use at scale and to facilitate the launch SELweb into the school marketplace. SELweb is currently used in 21 school districts in 16 states by over 90,000 students per year.

Interview with Clark McKown of Rush University Medical Center and xSEL Labs

 

From the start of the project, was it always a goal for SELweb to one day be ready to be used widely in schools?

CM: When we started our aspiration was to build a usable, feasible, scientifically sound assessment and it could be done. When the end of the grant got closer, we knew that unless we figured out another way to support the work, this would be yet another good idea that would wither on the vine after showing evidence of promise. In the last year and a half of the grant, I started thinking about how to get this into the hands of educators to support teaching and learning, and how to do it in a large-scale way.

 

By the conclusion of your IES grant to develop SELweb, how close were you to the version that is being used now in schools? How much more time and money was it going to take?

CM: Let me answer that in two ways. First is how close I thought we were to a scalable version. I thought we were pretty close. Then let me answer how close we really were. Not very close. We had built SELweb in a Flash based application that was perfectly suited to small-scale data collection and was economical to build. But for a number of reasons, there was no way that it would work at scale. So we needed capital, time, and a new platform. We found an outstanding technology partner, the 3C Institute, who have a terrific ed tech platform well-suited to our needs, robust, and scalable. And we received funding from the Wallace Foundation to migrate the assessment from the original platform to 3C’s. The other thing I have learned is that technology is not one and done. It requires continued investment, upkeep, and improvement.

What experiences led you to start a company? How were you able to do this as an academic researcher?

CM: I could tell you that I ran a children’s center, had a lot of program development experience, had raised funds, and all that would be true, and some of the skills I developed in those roles have transferred. But starting a company is really different than anything I’d done before. It’s exciting and terrifying. It requires constant effort, a willingness to change course, rapid decision-making, collaboration, and a different kind of creativity than the academy. Turns out I really like it. I probably wouldn’t have made the leap except that the research led me to something that I felt required the marketplace to develop further and to realize its potential. There was really only so far I could take SELweb in the academic context. And universities recognize the limitations of doing business through the university—that’s why they have offices of technology transfer—to spin off good ideas from the academy to the market. And it’s a feather in their cap when they help a faculty member commercialize an invention. So really, it was about finding out how to use the resources at my disposal to migrate to an ecosystem suited to continuing to improve SELweb and to get it into the hands of educators.

How did xSEL Labs pay for the full development of the version of SELweb ready for use at scale?

CM: Just as we were getting off the ground, we developed

 a partnership with a research funder (the Wallace Foundation) who was interested in using SELweb as an outcome measure in a large-scale field trial of an SEL initiative. They really liked SELweb, but it was clear that in its original form, it simply wouldn’t work at the scale they required. So we worked out a contract that included financial support for improving the system in exchange for discounted fees in the out years of the project.

What agreement did you make with the university in order to start your company and commercial SELweb?

CM: I negotiated a license for the intellectual property from Rush University with the university getting a royalty and a small equity stake in the company.

Did anyone provide you guidance on the business side?

CM: Yes. I lucked into a group of in-laws who happen to be entrepreneurs, some in the education space. And my wife has a sharp business mind. They were helpful. I also sought and found advisors with relevant expertise to help me think through the initial licensing terms, and then pricing, marketing, sales, product development, and the like. One of the nice things about business is that you aren’t expected to know everything. You do need to know how and when to reach out to others for guidance, and how to frame the issues so that guidance is relevant and helpful.

How do you describe the experience of commercializing SELWeb?

CM: Commercialization is, in my experience, an exercise in experimentation and successive approximations. How will you find time and money to test the waters? Commercialization is an exciting and challenging leap from the lab to the marketplace. In my experience, you can’t do it alone, and even with great partners, competitive forces and chance factors make success scale hard to accomplish. Knowing what you don’t know, and finding partners who can help, is critical.

I forgot who described a startup as a temporary organization designed to test whether a business idea is replicable and sustainable. That really rings true. The experience has been about leaving the safe confines of the university and entering the dynamic and endlessly interesting bazaar beyond the ivory tower to see if what I have to offer can solve a problem of practice.

In one sentence (or two!), what would say is most needed for gaining traction in the marketplace?

CM: Figure out who the customer is, what the customer needs, and how what you have to offer addresses those needs. Until you get that down, all the evidence in the world won’t lead to scale.

Do you have advice for university researchers seeking to move their laboratory research into wide-spread practice?

CM: It’s not really practical for most university researchers to shift gears and become an entrepreneur. So I don’t advise doing what I did, although I’m so glad I did. For most university researchers, they should continue doing great science, and when they recognize a scalable idea, consider commercialization as an important option for bringing the idea to scale. My impression is that academic culture often finds commerce to be alien and somewhat grubby, which can get in the way. The truth is, there are whip-smart people in business who have tremendous expertise. The biggest hurdle for many university researchers will be to recognize that they lack expertise in bringing ideas to market, they will need to find that expertise, respect it, and let go of some control as the idea, program, or product is shaped by market forces. It’s also a hard truth for researchers, but most of the world doesn’t care very much about evidence of efficacy. They have much more pressing problems of practice to attend to. Don’t get me wrong—evidence of efficacy is crucial. But for an efficacious idea to go to scale, usability and feasibility are the biggest considerations.

For academics, getting the product into the marketplace requires a new set of considerations, such as: Universities and granting mechanisms reward solo stars; the marketplace rewards partnerships. That is a big shift in mindset, and not easily accomplished. Think partnerships, not empires; listening more than talking.

Any final words of wisdom in moving your intervention from research to practice?

CM: Proving the concept of an ed tech product gets you to the starting line, not the finish. Going to scale benefits from, probably actually requires, the power of the marketplace. Figuring out how the marketplace works and how to fit your product into it is a big leap for most professors and inventors. Knowing the product is not the same as knowing how to commercialize it.

 ____________________________________________________________________________

Clark McKown is a national expert on social and emotional learning (SEL) assessments. In his role as a university faculty member, Clark has been the lead scientist on several large grants supporting the development and validation of SELweb, Networker, and other assessment systems. Clark is passionate about creating usable, feasible, and scientifically sound tools that help educators and their students.

This interview was produced by Ed Metz of the Institute of Education Sciences. This post is the third in an ongoing series of blog posts examining moving from university research to practice at scale in education.

IES Celebrates Computer Science Education Week and Prepares for the 2020 ED Games Expo at the Kennedy Center

This week is Computer Science Education Week! The annual event encourages students from Kindergarten to Grade 12 to explore coding, with a focus on increasing representation among girls, women, and minorities. The event honors the life of computer scientist Grace Hopper, who broke the mold in the 1940s as a programming pioneer. Coding and computer science events are occurring in schools and communities around the country to celebrate the week.

This week is also a great time to highlight the computer science and engineering projects that are coming to Washington, DC for the 2020 ED Games Expo at the Kennedy Center on the evening of January 9, 2020. Developed with the support of the Institute of Education Sciences and other federal government offices, the projects provide different types of opportunities for students to learn and practice computer science and engineering skills with an eye toward examining complex real-world problems.

At the Expo, expect to explore the projects listed below.

  1. In CodeSpark Academy’s Story Mode, children learn the ABCs of computer science with a word-free approach by programming characters called The Foos to create their own interactive stories. In development with a 2019 ED/IES SBIR award.
  2. In VidCode, students manipulate digital media assets such as photos, audio, and graphics to create special effects in videos to learn about the coding. A teacher dashboard is being developed through a 2019 ED/IES SBIR award.
  3. Future Engineers uses its platform to conduct STEM challenges for Kindergarten to Grade 12 students. Developed with a 2017 ED/IES SBIR award.
  4. Fab@School Maker Studio allows students to design and build geometric constructions, pop-ups, and working machines using low-cost materials and tools from scissors to inexpensive 3-D printers and laser cutters. Developed with initial funding in 2010 by ED/IES SBIR.
  5. In DESCARTES, students use engineering design and then create 3-D print prototypes of boats, gliders, and other machines. Developed through a 2017 ED/IES SBIR award.
  6. In Ghost School, students learn programming and software development skills by creating games. In development with a 2018 Education Innovation and Research grant at ED.
  7. In Tami’s Tower, children practice basic engineering to help Tami, a golden lion tamarin, reach fruit on an overhanging branch by building a tower with blocks of geometric shapes. Developed by the Smithsonian Institution.
  8. In the Wright’s First Flight, students learn the basics of engineering a plane through hands-on and online activities, then get a firsthand look at what it looked (and felt) like to fly it through a virtual reality simulation. Developed by the Smithsonian Institution.
  9. In EDISON, students solve  engineering problems with gamified design software and simulate designs in virtual and augmented reality. In development with support from the National Science Foundation. 
  10. May’s Journey is a narrative puzzle game world where players use beginning programming skills to solve puzzles and help May find her friend and discover what is happening to her world. Developed with support from the National Science Foundation. 
  11. In FLEET, students engineer ships for a variety of naval missions, test their designs, gather data, and compete in nationwide naval engineering challenges. Developed with support from the U.S. Navy’s Office of Naval Research.
  12. Muzzy Lane Author is a platform for authoring learning games and simulations without requiring any programming skills. Developed in part with a Department of Defense award.

About the ED Games Expo: The ED Games Expo is the Institute’s and the Department of Education's annual public showcase and celebration of educational learning games as well as innovative forms of learning technologies for children and students in education and special education. At the Expo, attendees walk around the Terrace Level Galleries at the Kennedy Center to discover and demo more than 150 learning games and technologies, while meeting face-to-face with the developers. The Expo is free and open to the public. Attendees must RSVP online to gain entry. For more information, please email Edward.Metz@ed.gov.

Edward Metz is the program manager for the ED/IES Small Business Innovation Research program.

Christina Chhin is the program officer for the Science, Technology, Engineering, and Mathematics (STEM) Education research program.

IES Makes Two New Awards for the Development of Web-based Tools to Inform Decision Making by Postsecondary Students

In June, the Institute of Education Sciences (IES) announced two new awards to technology firms to develop web-based tools that inform student decision making in postsecondary education. The projects will focus on generating a measure of the return of investment (ROI) for different educational training programs and careers so that high school and college students have access to data-driven information to guide their decisions.

The awards were made through a special topic offered by the IES Small Business Innovation Research (known as ED/IES SBIR) program, which funds the research and development of commercially viable education technology. (For information on the 21 awards made through the IES 2019 standard solicitation, read here.)

Background and Awards

While websites like College Scorecard and CareerOneStop provide information to explore training programs in colleges and occupations of interest, there is no tool that helps students understand the costs and benefits of individual postsecondary programs in an integrated, customizable, and user-friendly manner.  

The special topic SBIR solicitation requested proposals from small businesses to develop new ROI tools that would combine information on fees, time to complete, and projected earnings so that students can easily compare college and career pathways. The IES-funded ROI tools aim to improve student program completion rates, with higher employment and earnings, less education-related debt, and more satisfaction with their selected paths. The special topic SBIR solicitation offered up to $200,000 for firms to develop and evaluate a prototype of their ROI tool. 

Two awards were made through this special topic:

  • Illinois-based BrightHive, Inc. is developing a prototype of the Training, Education, and Apprenticeship Program Outcomes Toolkit (TEAPOT). Designed to inform student training and educational decision making over a variety of potential pathways, TEAPOT will improve the flow and accuracy of data resulting in improved estimates of the ROI for different postsecondary education pathways.  The team will develop a data interoperability system and simplified toolkit for states and local postsecondary and workforce development organizations. The toolkit will provide more high quality, consistent, and granular information on postsecondary outcomes. The prototype will calculate ROI using student information, programmatic information (with an emphasis on net program costs to allow for variations by program type at the same institution), and access to wage and employment data sets.
  • Virginia-based Vantage Point Consultants is developing a prototype of a user-contextualized ROI tool that prospective students will use to make meaning of lifetime costs and opportunity tradeoffs associated with different degree programs offered by postsecondary institutions. The ROI tool will incorporate information on student goals and academic, professional, and personal characteristics.  The prototype will include an interface to present information to guide decision making based on an ROI calculation that discounts earning cash-flows under current and future state career and education assumptions, while subtracting college cost. In the first phase of work, the project will use information from data partners including Burning Glass Technologies and from public sources at the Department of Labor and Department of Education.

After developing prototypes, researchers will analyze whether the tools function as intended and are feasible for students to use. Research will also test if the tool shows promise for producing a meaningful and accurate measure of ROI.  Both firms are eligible to apply for additional funding to complete the full-scale development of the ROI tool, including developing an interface to improve user experience and conducting additional validation research.

Stay tuned for updates on Twitter (@IESResearch) as IES projects drive innovative forms of technology.

Written by Edward Metz, program manager, ED/IES SBIR

Investing in the Next Generation of Education Technology

Millions of students in thousands of schools around the country have used technologies developed through the Small Business Innovation Research program (ED/IES SBIR) at the IES. The program emphasizes rapid research and development (R&D), with rigorous research informing iterative development and evaluating the promise of products for improving the intended outcomes. The program also focuses on the commercialization after development is complete so that products can reach schools and be sustainable over time.

At the end of June, ED/IES SBIR announced 21 new awards for technology products for students, teachers, or administrators in education and special education. (IES also announced two additional awards through a special topic solicitation in postsecondary education. Read about these awards here.) Of the 21 awards, 13 are for prototype development and 8 for full scale development (a YouTube playlist of the full scale development projects is available here). 

Many of the new 2019 projects continue education and technology trends that have emerged in recent years. These include the three trends below.

Trend 1: Bringing Next Generation Technologies for Classrooms
For educators, it can be challenging to integrate next generation technologies into classroom practice to improve teaching and learning. In the current group of awardees, many developers are seeking to make this happen. Schell Games is developing a content creation tool for students to create artistic performances in Virtual Reality (VR) and Gigantic Mechanic is designing a class-wide role-playing game facilitated by a tablet-based app. codeSpark is building a game for children to learn to code by creating story based narratives. Killer Snails, Lighthaus, and AP Ventures are all creating educational content for VR headsets and Parametric Studios, Innovation Design Labs, and LightUp are employing Augmented Reality (AR) to support learning STEM concepts. Aufero is bringing modern design principles to develop a traditional board game for students to gain foundational computer science and coding skills.

Trend 2: Personalized Learning

Several 2019 awards are building technologies to provide immediate feedback to personalize student learning. Graspable, Inc. and Apprendis are developing adaptive engines that formatively assess performance as students do activities in algebra and physical science, and Sirius Thinking is building a multimedia platform to guide and support pairs of students as they read passages. Charmtech is developing a prototype to support English learners in reading, Cognitive Toybox is creating a game-based school readiness assessment, Hats & Ladders is developing a social skills game, and IQ Sonics is refining a music-based app for children with or at risk for disabilities to practice speaking.

Trend 3: Platforms that Host and Present Data
School administrators and teachers are always seeking useful information and data to guide decision making and inform instruction. Education Modified is developing a platform for special education teachers to implement effective Individual Education Programs (IEPs) for students with or at risk for disabilities, and VidCode is developing a dashboard to offer teachers real-time performance metrics on coding activities to teachers. LearnPlatform is developing a prototype platform that generates reports to guide teachers in implementing new education technology interventions in classrooms, and Liminal eSports is developing a platform administrators and teachers can use to organize eSports activities where students participate in group game activities to learn.

Stay tuned for updates on Twitter and Facebook as IES continues to support innovative forms of technology.

Written by Edward Metz, Program Manager, ED/IES SBIR

Equity Through Innovation: New Models, Methods, and Instruments to Measure What Matters for Diverse Learners

In today’s diverse classrooms, it is both challenging and critical to gather accurate and meaningful information about student knowledge and skills. Certain populations present unique challenges in this regard – for example, English learners (ELs) often struggle on assessments delivered in English. On “typical” classroom and state assessments, it can be difficult to parse how much of an EL student’s performance stems from content knowledge, and how much from language learner status. This lack of clarity makes it harder to make informed decisions about what students need instructionally, and often results in ELs being excluded from challenging (or even typical) coursework.

Over the past several years, NCER has invested in several grants to design innovative assessments that will collect and deliver better information about what ELs know and can do across the PK-12 spectrum. This work is producing some exciting results and products.

  • Jason Anthony and his colleagues at the University of South Florida have developed the School Readiness Curriculum Based Measurement System (SR-CBMS), a collection of measures for English- and Spanish-speaking 3- to 5-year-old children. Over the course of two back-to-back Measurement projects, Dr. Anthony’s team co-developed and co-normed item banks in English and Spanish in 13 different domains covering language, math, and science. The assessments are intended for a variety of uses, including screening, benchmarking, progress monitoring, and evaluation. The team used item development and evaluation procedures designed to assure that both the English and Spanish tests are sociolinguistically appropriate for both monolingual and bilingual speakers.

 

  • Daryl Greenfield and his team at the University of Miami created Enfoque en Ciencia, a computerized-adaptive test (CAT) designed to assess Latino preschoolers’ science knowledge and skills. Enfoque en Ciencia is built on 400 Spanish-language items that cover three science content domains and eight science practices. The items were independently translated into four major Spanish dialects and reviewed by a team of bilingual experts and early childhood researchers to create a consensus translation that would be appropriate for 3 to 5 year olds. The assessment is delivered via touch screen and is equated with an English-language version of the same test, Lens on Science.

  • A University of Houston team led by David Francis is engaged in a project to study the factors that affect assessment of vocabulary knowledge among ELs in unintended ways. Using a variety of psychometric methods, this team explores data from the Word Generation Academic Vocabulary Test to identify features that affect item difficulty and explore whether these features operate similarly for current, former, as well as students who have never been classified as ELs. The team will also preview a set of test recommendations for improving the accuracy and reliability of extant vocabulary assessments.

 

  • Researchers led by Rebecca Kopriva at the University of Wisconsin recently completed work on a set of technology-based, classroom-embedded formative assessments intended to support and encourage teachers to teach more complex math and science to ELs. The assessments use multiple methods to reduce the overall language load typically associated with challenging content in middle school math and science. The tools use auto-scoring techniques and are capable of providing immediate feedback to students and teachers in the form of specific, individualized, data-driven guidance to improve instruction for ELs.

 

By leveraging technology, developing new item formats and scoring models, and expanding the linguistic repertoire students may access, these teams have found ways to allow ELs – and all students – to show what really matters: their academic content knowledge and skills.

 

Written by Molly Faulkner-Bond (former NCER program officer).