Location
Institute of Education Sciences (IES) Board Room
80 F Street NW
Washington, DC 20001

Participants

National Board for Education Sciences (NBES) Members Present
Bridget Terry Long, Ph.D., Chair
Anthony S. Bryk, Ed.D.
David Chard, Ph.D.
Darryl J. Ford, Ph.D.
Adam Gamoran, Ph.D.
Robert Granger, Ed.D. (by phone)
Larry V. Hedges, Ph.D.
Susanna Loeb, Ph.D.
Judith Singer, Ph.D.
Robert A. Underwood, Ed.D.
Hirokazu Yoshikawa, Ph.D.

NBES Members Absent
David Chard, Ph.D.

Ex-Officio Members Present
John Q. Easton, Ph.D., Director, IES, U.S. Department of Education (ED)
Thomas Brock, Ph.D., Commissioner, National Center for Education Research (NCER)
Sean P. “Jack” Buckley, Ph.D., Commissioner, National Center for Education Statistics (NCES)
Janice Earle, Ph.D. (for Joan Ferrini-Mundy, Ph.D.), Senior Program Director K–12 STEM Education, National Science Foundation (NSF), Directorate for Education and Human Resources
Brett Miller, Ph.D., Health Scientist Administrator, Child Development & Behavior Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), National Institutes of Health (NIH)
Ruth Curran Neild, Ph.D., Commissioner, National Center for Education Evaluation and Regional Assistance (NCEE)
Deborah Speece, Ph.D., Commissioner, National Center for Special Education Research (NCSER)

NBES Staff
Rebecca McGill-Wilkinson, Ph.D., Associate Research Scientist, NCER
Ellie Pelaez, Designated Federal Official

ED Staff
Elizabeth Albro, Ph.D., NCER, IES
Corinne Alfeld, Ph.D., IES
Sue Betka, IES
Teresa Cahalan, IES
Joy Lesnick, Ph.D., IES
Diana McCallum, Ph.D., IES
Joan McLaughlin, Ph.D., NCSER, IES
Anne Ricciuti, Ph.D., IES
Pamela Tripp-Melby, NCEE, IES
Elizabeth Warner, Ph.D., IES

Invited Presenters
Margaret Honey, Ph.D., President and Chief Executive Officer, New York Hall of Science (NYSCI)
Edward Metz, Ph.D., Program Officer, IES Small Business Innovation Research (SBIR) Program
Constance Steinkuehler, Ph.D., Senior Policy Analyst, Office of Science and Technology Policy (OSTP), Executive Office of the President; Associate Professor, University of Wisconsin—Madison

Members of the Public
Jodi Asbell-Clarke, TERC
Peggy Clements, Education Development Center, Inc.
Frank Davis, TERC
Scott Filter, Bridgepoint Education
John Fraser, NewKnowledge Organization
Jean Gossman, LRP Publications
Kim Hymes, Council for Exceptional Children
Carla Jacobs, Lewis-Burke Associates, LLC
Jim Kohlmoos, Edge Consulting, LLC
Michele McLaughlin, Knowledge Alliance
Sarah Mancoll, Society for Research in Child Development
Emily Markham, Association for Public Television Stations
Augustus Mays, WestEd
Gerald Sroufe, American Educational Research Association
Anna Suarez, Trivium Consulting
Alison Thompson, Lewis-Burke Associates, LLC

Call to Order
Bridget Terry Long, Ph.D., NBES Chair

Dr. Long called the meeting to order at 8:34 a.m. and called roll. Board members unanimously approved the agenda for the meeting.

Swearing-In of Reappointed Members
John Q. Easton, Ph.D., IES Director

Dr. Easton swore in a reappointed board member, Margaret R. (Peggy) McLeod, Ed.D.

Chair's Remarks
Bridget Terry Long, Ph.D., NBES Chair

Dr. Long said that board member David Chard, Ph.D., is facing a serious health issue and anticipates a long recovery. She encouraged board members to reach out to Dr. Chard.

Dr. Long announced that Deborah Speece, Ph.D., the NCSER commissioner, is leaving her position. Before becoming commissioner, Dr. Speece led innovative studies in classifying and diagnosing learning disabilities, among other topics, and her appointment as commissioner won strong support from the community. On behalf of the board, Dr. Long expressed appreciation for Dr. Speece and said she enjoyed working with her.

In the absence of an executive director of NBES, Rebecca McGill-Wilkinson, Ph.D., of NCER, has been very helpful as a researcher for the board. Working with Dr. Easton and Sue Betka of IES, Dr. Long has determined that the board can hire a consultant to work intermittently to plan meetings, consult with the board, and work on the annual report, as long as all inherently governmental business is handled by an employee of IES. The board does not advertise such consultant positions but rather identifies a candidate, determines the number of days the consultant can work, and negotiates an agreement. Dr. Long identified Melissa Bert, Ph.D., a recent graduate of the Harvard Graduate School of Education, for whom Dr. Long served as dissertation chair. Dr. Bert has strong experience in education policy and research, and Dr. Long believes she will be very helpful to the board. Dr. Long said the position may evolve as the board determines the level of support it needs.

Summarizing the agenda for the day, Dr. Long thanked those commissioners who provided written summaries of their centers' recent work to the board in advance. She framed the agenda in terms of the three themes that the board is continually addressing. Regarding research funding, the board has previously discussed new IES research topics and the peer-review process. Today, it will address how to evaluate the IES research portfolio—that is, whether IES is funding the right things to achieve maximum impact. Funding choices are especially important in the face of limited and possibly declining resources. Regarding impact through disseminating and scaling up promising practices, the board will hear from two commissioners about improving the use and dissemination of IES-funded products and research. Dr. Long said she hopes to have ongoing conversations between the board and the commissioners that facilitate continuous improvement at IES. Finally, in terms of advocating for support and use of research in policymaking, Dr. Long referred to the upcoming annual report and plans to meet with more Congressional staff this summer to highlight what IES is doing well and discuss the status of legislation reauthorizing IES.

Update: Recent Developments at IES
John Q. Easton, Ph.D., IES Director

Dr. Easton reiterated that Dr. Speece is moving on after fulfilling most of her 2-year appointment as commissioner. She has been named associate dean for research at the School of Education at Virginia Commonwealth University. Dr. Easton said that during Dr. Speece's tenure, NCSER started a lot of new work, and IES will miss her. Joan McLaughlin, Ph.D., the current deputy commissioner of NCSER, will serve as acting commissioner. Dr. Easton said he is pleased to have someone as capable as Dr. McLaughlin to fill the role.

Dr. Easton welcomed Janice Earle, Ph.D., of NSF, who has been working on a joint project with NCEE to develop a common evidence framework for grant-making that focuses on education for math and science. The project has been in the works for over 2 years. It resulted from the White House OSTP emphasizing science, technology, engineering, and math (STEM) education and calls for a common framework for STEM research by a cross-government committee on STEM education. It will address the Office of Innovation and Improvement Investing in Innovation (i3) fund's tiered evidence approach to grant-making and the Office of Management and Budget's (OMB's) push for better evaluation mechanisms across the federal government.

The resulting document is in the final stages of clearance. It was publicly presented at the American Educational Research Association conference and received excellent feedback. The document helps NSF and IES more broadly lay out a series of different research types to guide grant-making and will facilitate more learning and science—not just for STEM but broadly around social, behavioral, and educational research. For each type of research, the framework lays out “entrance criteria” (i.e., the evidence that should already be in hand) and “exit criteria” (i.e., what the research is expected to produce). There were concerns about a framework that would make research look very linear (when in fact it is more often iterative and categories overlap), but Dr. Easton said the framework has the right amount of structure without oversimplifying the research process. He concluded that he was very pleased with how well IES and NSF worked together, even though the process represented a culture shift for both agencies.

Dr. Earle said that even within NSF, programs had different criteria. The framework will provide more clarity internally and across the field, she said. She hoped the framework would be updated and revised as opportunities arise. NCEE Commissioner Ruth Curran Neild, Ph.D., who codirected the project with Dr. Earle, pointed out that the new document provides a common vocabulary and common expectations. She appreciated its emphasis on not expecting too much (e.g., from early exploratory studies) or too little (e.g., for large randomized controlled trials [RCTs] in which significant investments are made).

Hirokazu Yoshikawa, Ph.D., asked whether the NSF will apply the NSF/IES framework primarily through the Education Directorate or also through the Social, Behavioral, and Economic Sciences Directorate. Dr. Earle said it mostly applies to education research and development (R&D), but to the extent that the Social, Behavioral, and Economic Sciences Directorate funds education R&D, it would apply the framework to those studies, as well as to some engineering education and computer science, but not basic science research. The new framework will begin to standardize language across programs that are funding education R&D. How the framework will be applied to review criteria is not yet completely clear, said Dr. Earle.

Dr. Easton said that in mid-May, the OSTP hosted a meeting at the White House on growth mindset, a concept that has received a lot of attention in recent years. Carol Dweck, Ph.D., who pioneered the concept, attended, as did NBES members Anthony Bryk, Ed.D., and Adam Gamoran, Ph.D. The White House convened the meeting because the administration believes that relatively small psychological interventions can have long-term effects and is building the concept into its policymaking and research agenda. About a week after the meeting, First Lady Michelle Obama expressed the concept in a presentation at a school. Dr. Easton said that IES is interested not just in academic mindset but in other social and psychological influences that affect development, and that interest is reflected in requests for applications (RFAs).

Dr. Easton explained that ED has been operating under a continuing resolution for fiscal year (FY) 2013, and the sequester was enacted in March. The combination of these factors resulted in a 5.25 percent reduction in the budget over FY 2012 levels, amounting to about $31 million, spread over seven budget lines.

In FY 2013, NCER received 80 applications for awards that were judged outstanding or excellent by peer reviewers and therefore eligible for funding. Because of the fiscal limitations and IES's commitment to existing grants, funding for new grants was cut by about $10 million. As a result, 49 of the 80 eligible applications were funded, with total first-year costs of $22 million. If the sequester were not in place, NCER would have funded another 16 applications (but probably not all 80).

NCSER received 45 applications that were deemed excellent or outstanding. Sequestration removed about $2.5 million from the NCSER research budget, and NCSER had a budget reduction 2 years ago said Dr. Easton. As a result, only 18 (or 40 percent of applications eligible for funding) were funded. Without sequestration, NCSER could only have funded three more applications. Funding all eligible applications would have required another $14 million. The FY 2014 budget requests a $10 million increase for NCSER.

Budget complexities also affected the timing of awards, said Dr. Easton. IES conducts two grant competitions each year, reviewing one group in the fall and a second in the spring. Because of the uncertainty around federal funding available for awards, IES deferred decisionmaking about round-one applications until round-two applications were reviewed. Some applicants from round one did not learn the fate of their proposals until late May, leaving them in a precarious position, especially if they wanted to resubmit their proposals. Dr. Easton said that IES addressed the situation as quickly and fairly as it could. He noted that IES posted NCER RFAs in early May for 2014, but there will be no competition for FY 2014 for special education research and only one round of competition for NCER awards.

Discussion

In response to Dr. Gamoran, Dr. Easton explained that if more money becomes available in the next FY, IES will fund some of the NCSER eligible proposals from FY 2013 because it received many very good proposals. Dr. Gamoran said IES, especially NCSER, is a victim of its own success. Budgets were reduced because of the lack of proposals that met minimum criteria; now, thanks to Dr. Speece and others, we have good proposals but lack resources to fund them, Dr. Gamoran pointed out. He felt IES should communicate the positive part of the story. Dr. Easton said the President's budget for FY 2015 includes increased funding for IES.

Judith Singer, Ph.D., called the report sobering news. She suggested the situation offers an opportunity to at least discuss how to make more strategic funding decisions that promote IES priorities over the order of quality determined by peer review panels. Dr. Easton agreed but noted that at present, IES can only override the panels' determination of order in special circumstances. Dr. Long also agreed that the issue merits further discussion, as the panels' determination of order may reflect very small, even arbitrary differences in scores between applications. She pointed out that one role of the board is to oversee the peer review process.

Larry V. Hedges, Ph.D., asked whether continued budget uncertainty was the rationale for limiting NCER to one round of competition. Dr. Easton said the uncertainty was the primary reason, but IES is also testing the approach. Thomas Brock, Ph.D., noted that IES does not want to put applicants in the same position as it did this year if budgets are uncertain. In response to Dr. Long, Dr. Brock noted that the applicants who received funding were notified sooner than those who did not. All of this year's applicants now know their status, and the awards will be announced publicly later in June. Dr. Brock further clarified that applications not funded can always be resubmitted, and applicants are encouraged to talk with the program officers, who should be encouraging the resubmission of high-scoring applications.

The situation is different for NCSER, said Dr. Speece. NCSER is unable to give straightforward advice because there are so many questions about funding, so the best advice it can offer is to seek other funding or to resubmit unfunded applications to NCER, if appropriate. Before NCSER was created, many projects were funded through NCER, but some topics just do not fit into NCER categories, which makes it a tough situation, said Dr. Speece.

Brett Miller, Ph.D., asked whether IES has considered communicating with the grantee community as a whole—for example, by making a statement clarifying the motivation for the FY 2014 plans—to quell rumors and clarify expectations. Dr. Easton said the idea has been discussed but IES is being cautious, because uncertainty continues. Dr. Long pointed out that with Dr. Speece stepping down and NCSER not holding a competition for FY 2014, it may be appropriate to make it clear that NCSER is not being eliminated.

Dr. Long asked for more clarity about the impact of the budget cuts. Dr. Easton responded that the cuts were spread equally across budget lines and mostly affected grant funding so far. For other activities, such as contracts, IES can opt to delay awarding a contract or stretch out the work. The statistics budget was reduced by $5.7 million, Dr. Easton noted. NCES has delayed data collection from several surveys and canceled a teacher compensation survey. The National Assessment of Educational Progress had to cut $6.8 million from its budget, but the National Assessment Governing Board influences that program and sets the assessment schedule in collaboration with NCES and IES. As a result, two 2014 assessments for 4th and 12th graders will be canceled but the same assessments will continue for 8th graders.

Darryl J. Ford, Ph.D., asked how the budget cuts affected hiring and infrastructure. Dr. Easton responded that administrative costs are a separate budget line. IES had a lot of retirements and offered some buyouts. Staff levels are down from a peak of 203 to about 168, and IES has permission to backfill for duties already committed on a case-by-case basis. At current staffing levels, people feel stretched, said Dr. Easton.

Finally, Dr. Easton noted that the board has two vacancies, and the candidates proposed for those vacancies did not make it through the appointment process. A list of possible candidates has been submitted but there appears to be a blockage. The two open slots expire in November 2014, so any candidates confirmed would only be appointed for a short term. IES is accepting nominations for board members.

Commissioner Updates
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild said the work of the center falls into three categories:

• Producing research evidence
• Reviewing and summarizing research evidence
• Providing access to, and support for, the research evidence

Over the past 3 months, NCEE has been preparing to review studies of evidence-based grant-making to better illustrate what a sustainable, systematic process looks like. The i3 program provides a good example of evidence-based grant-making. It offers three categories of awards—development, validation, and scale-up, each with entrance and exit criteria. The maximum funding amount is tied to the amount of existing supporting evidence. There is an expectation that the funded project will evaluate its accomplishments, build on existing evidence, and contribute to the evidence base.

In addition to a new round of i3 this year, other evidence-based grant competitions are the Arts Model and the Strengthening Institutions Program. Dr. Neild said ED's General Administrative Regulations (EDGAR) are being revised, making it easier for grant-makers to use the evidence-based approach. However, the approach raises issues about capacity, because more review is required. NCEE is addressing questions of sustainability, such as how much to budget for evidence review within a competition, how to scale up to capacity for such review while maintaining quality, how to broaden the impact of reviews, and how to leverage the process to increase the capacity of What Works Clearinghouse (WWC) reviewers in the field.

NCEE named Pamela Tripp-Melby to head the National Library of Education and the Education Resources Information Center (ERIC). Dr. Neild explained that the National Library of Education primarily serves ED and provides a range of information services, such as full citations for evidence used in grant competition applications, information searches for key offices, and input into systematic reviews by the WWC and others. The collection is being winnowed to become more research-focused. Dr. Neild emphasized that the library occupies a real, physical space and has 10 full-time employees.

In February 2013, the OSTP required agencies to make the results of grant-funded research available to the public, specifically findings in peer-reviewed publications and digital data. IES instituted a public access policy for peer-reviewed work beginning with FY 2012 grantees, and the OSTP directive provided an opportunity to expand the policy ED-wide. In the process, NCEE is addressing some complex issues, such as licensing and the accessibility of information to people with disabilities. Elizabeth Albro, Ph.D., of NCER, is working with Ms. Tripp-Melby on the effort. Dr. Albro noted that every journal has different policies, which makes public access challenging, but ED is making good progress. She anticipated that a draft policy would be ready by August. The effort has been a good opportunity to work across ED and discuss disseminating findings. Dr. Neild added that grantees are encouraged to upload their submitted manuscripts to ERIC, but ideally, publishers will provide the final manuscripts.

Evidence planning for FY 2014 is underway. The process, implemented about 2 years ago to make planning more systematic, facilitates meetings across ED program offices to exchange information, identify needs and opportunities, highlight “teachable moments,” and discuss what kinds of evidence are needed and when.

Discussion

Robert Granger, Ed.D., asked whether addressing capacity and evidence challenges refers to how claims about evidence made by applicants are vetted. Dr. Neild said competitions are just gearing up now, so there has not been much discussion, but the issue points to challenges in organizational capacity. Currently, NCEE is discussing how to build evidence using the evidence-based grant-making approach. The next step is to establish a sustainable process, building capacity in the field and directing work to the reviewers in the field. At present, most of the conversation has occurred between ED and OMB, said Dr. Neild. Dr. Granger said that other agencies, such as the U.S. Department of Health and Human Services (HHS), have similar competitions but use different approaches, and it may be helpful to consider how they assess evidence provided by applicants.

Dr. Singer wondered whether focusing on vetting evidence is the best use of IES resources, because published evidence has already gone through peer review. It appears that IES is setting up a process to do what reviewers in the field should already be doing. Dr. Neild noted that the IES commits no resources to the effort other than staff time.

Dr. Yoshikawa suggested thinking about when internal reviews might be useful to the field and vice versa. For example, President Obama spoke about preschool for all and a rigorous curriculum, and IES may have good information for the field about what constitutes a rigorous curriculum. Dr. Speece noted that NCSER and NCER have been working on synthesizing research around topics for which IES has already generated a lot of evidence. For example, one synthesis in the pipeline addresses early childhood development and early intervention, and another looks at reading. The mechanism is not very nimble but provides insight on what we have learned and where to invest resources, said Dr. Speece. Dr. Easton said that IES has a formal relationship with the ED Policy Office for providing evidence for policy development. Dr. Neild said there are other feedback mechanisms around implementation and impact.

Dr. Gamoran noted that increasing reviewer capacity for WWC review has two benefits: it increases the capacity of the WWC and also disseminates knowledge to the field more broadly.

National Center for Education Research
Thomas Brock, Ph.D., NCER Commissioner

Dr. Brock said NCER has been focusing on determining the awards for the FY 2013 competition. A theme across all of NCER's work is to encourage continuous improvement in an era of constrained resources. Dr. Brock anticipates lower budgets for grants for the coming year. Starting in FY 2014, as a result of conversations with the board, IES is requesting dissemination plans as part of Education Research Programs' applications, the largest program supported by NCER.

For the research training program, IES is placing more emphasis on involving fellows in partnerships and collaborations. Some programs face decreased funding, said Dr. Brock, but analysis showed that universities almost always extend the grants for 1 or 2 years, so the reduction probably aligns with what universities actually have been spending. Previous research training grantees have been asked to demonstrate the progress and accomplishments of their fellows. IES is limiting the number of predoctoral research training grants to five in FY 2014 to conserve resources. There will be no grant competition for postdoctoral training this year, although Dr. Brock emphasized the change represents a pause, not a stop to the program. Over the next year, he will be talking with university sponsors, fellows, and others about the postdoctoral program.

The third major competition for NCER is for two education R&D centers. IES will provide up to $10 million over 5 years for a National Center on Developmental Education Assessment and Instruction. It represents a response to the recognition that the assessments used to identify candidates for remediation for postsecondary education are not very good. There have been a lot of innovative proposals on reforming assessments and curriculum, said Dr. Brock, but little evaluation of these innovations. He hoped the new center would identify alternatives and also bring together researchers across the field.

Also, IES will provide up to $5 million over 5 years for an R&D Center on Knowledge Utilization, a concept that reflects conversations with the NBES. The Center will address what the research community can do to ensure that high-quality research is understood and implemented. It will develop measures to assess whether practitioners are aware of and using research, a factor that varies over time and can manifest directly or indirectly. It will also explore the conditions that support the use of evidence. For both new centers, Dr. Brock said, IES is seeking to establish national leadership by bringing together researchers, policymakers, and practitioners.

For the Statistical and Research Methodology in Education program, IES has created a new early-career award to support researchers in the first 5 years of their careers. Dr. Brock hoped the new award would bolster the pipeline of talent. The Partnerships and Collaborations Focused on Problems of Practice and Policy program encourages researchers to work with policymakers and practitioners to design, implement, and disseminate studies. Within the program, IES will offer grants of up to $2 million to support continuous improvement in research, such as efforts that promote rapid testing with quick feedback and lead to improvement in interventions. This first round of continuous improvement funding will be limited to four awards. The Partnerships and Collaborations Focused on Problems of Practice and Policy program also addresses evaluation of state and local education agency programs and policies, reinforcing the premise that researchers should work with state education agencies, Dr. Brock concluded.

National Center for Special Education Research
Deborah Speece, Ph.D., NCSER Commissioner

NCSER has funded three competitions for FY 2013, of which special education research grants is the largest, and two are new: Accelerating the Academic Achievement of Children with Learning Disabilities to promote new research and the Early Career Development and Mentoring Grant to help new assistant professors gain a foothold in research. Dr. Speece said she was pleased with the quality and number of applications for FY 2013. She noted that NCSER began funding grants in 2006 and it took some time to orient the field to the nature of the grants. She disagreed, however, with the perception that the quality of applications in the earlier years was not good. The number of applications was low at the outset of the program, and as funding has increased, so has the number of applications submitted. Throughout the existence of NCSER, the percentage of applications of sufficient quality to move on to peer review has been about the same as that for NCER, which, Dr. Speece said, indicates that the perception of poor quality is not based on reality.

FY 2012 was the high point for grant applications received, although the percentage funded remained consistent with previous years at about 11 percent, similar to the percentage funded by NCER. The FY 2013 funding rate would have been in the same range if not for the budget constraints. That situation led to the decision not to hold a new round of competition in FY 2014 but rather to use whatever grant funding is available in FY 2014 to support proposals from FY 2013 that NCSER could not fund. Dr. Speece said she believes the approach is the fiscally responsible thing to do. However, it is a blow to an organization that is supposed to be doing more to fund new knowledge for children with disabilities. Dr. Speece pointed out that the number of RCTs funded has doubled since 2006. Also, FY 2015 looks brighter, and Dr. Speece hoped that NCSER would restart its funding competitions in FY 2015. The President has proposed a $10 million increase for NCSER.

Dr. Speece noted that NCSER is looking forward to its summer Single-Case Design Institute at the University of Wisconsin. It has been well received. Recently, NCSER provided additional support for an NICHD grantee to host an annual conference on math cognition. The first conference, held in May, focused on the evolutionary origins of math cognition, and future conferences will address intervention and other influences on math development over time.

National Center for Education Statistics
Sean “Jack” Buckley, Ph.D., NCES Commissioner

Dr. Buckley focused his report on the annual report to Congress, The Condition of Education. The report has been mandated since 1875. In recent years, NCES has been moving away from paper versions of the report, opting instead to provide the information online (and print a few copies); in 2012, the report was first offered as an e-book and can be printed on demand thanks to the Government Printing Office.

For the 2013 Condition of Education, released on May 23, NCES is working to further improve the format and dissemination and make the report easier to use. For example, NCES will take advantage of current technology by updating the report continually as new data become available. The report is due in June every year, so it will represent a snapshot in time, but it will be updated throughout the year. Also, NCES is pursuing new channels for dissemination, creating YouTube videos and a mobile version for smartphones.

At its peak in 1998, The Condition of Education included 60 indicators and required two volumes, plus supplements. The 2013 version reports on 42 key indicators, categorized across 4 sections: population characteristics, participation in education, elementary and secondary education, and postsecondary education. In addition, The Condition of Education includes four spotlight reports on narrower topics, two of which received a lot of attention: “Kindergarten Entry Status: On-Time, Delayed-Entry, and Repeating Kindergartners” and “Financing Postsecondary Education in the United States.”

The next Condition of Education will reflect the experience with continual updating of data. Dr. Buckley said the report includes not just internal data but data from states and others, and it has been challenging to ensure that all the sources keep their data up to date.

Follow-Up Item

Dr. Long looked forward to a full summary of all of NCES activities at a future meeting.

Improving the Use of IES Research and Products
Edward Metz, Ph.D., Program Officer, IES SBIR Program

Dr. Metz explained that, in 1982, Congress mandated that federal agencies that provide over $100 million annually in extramural funding offer an SBIR program to infuse capital into small business R&D and spur innovation around national needs. The program was reauthorized in 2011, and 11 federal agencies have SBIR programs totaling $2.5 billion. Of that, ED has about $12 million for SBIR programs at IES and the NIDRR.

The IES SBIR funds technology products for use by students in regular or special education to improve learning and outcomes; by teachers to improve practice or efficiency; or by infants, toddlers, parents, or service providers in early intervention settings. The program aligns with IES research goals.

The SBIR structure is unique. In Phase 1, grantees receive $150,000 for 6 months for initial R&D and pilot research. In Phase 2, grantees receive $900,000 for 2 years for full-scale product development, pilot research, and demonstration of function. Phase 3 is unique to SBIR in that no funding is provided, but the goal of the program from the outset is to develop commercially viable education products that can be widely disseminated and sustained. In addition, Dr. Metz said, the IES SBIR would like to see completed products further evaluated and encourages grantees to apply for IES research grants for efficacy testing.

In February 2013, the IES SBIR received 260 proposals in response to 4 RFAs. Proposals were reviewed internally by 75 federal employee volunteers. In May, the SBIR announced 11 Phase 1 awards, 6 Phase 2 awards, and 3 Fast-Track awards (a combination of Phases 1 and 2 for proposals ready for full-scale production). Also in May, the OSTP highlighted the IES SBIR games award in its blog.

Dr. Metz presented video clips describing three products resulting from IES SBIR grants. The first, Filament Games, developed four science learning games. About 300,000 students have already played the games, and the company has won several industry innovation awards. Another company, Fluidity Software, produced FluidMath, which converts a teacher's handwritten equations into graphic forms that can be manipulated and even animated as a teaching tool. Fluidity Software partnered with manufacturers to include FluidMath as part of the standard software provided with certain products. The company has received some industry awards already, and the IES SBIR granted it another Phase 1 award to expand FluidMath for student use.

Another product, Mindset Works, applies the growth mindset theory pioneered by Dr. Carol Dweck over a multimedia platform. It includes professional development and a blended learning curriculum for the growth mindset. Dr. Metz said the Mindset Works program is exciting because it translates theory into social and behavioral strategies using technology.

Since 2003, the IES SBIR has funded 116 Phase 1 awards, 32 Phase 2 awards, and 32 Fast-Track awards. Of the 64 Phase 2 and Fast-Track awards, 41 are complete and 23 are underway. Dr. Metz categorized the 41 completed studies into three groups: failed, in process, and commercial success.

Of the 19 products in the first (failed) group, a majority were funded in the earliest years of the program (2003-2007). Sometimes, the products failed because the technology was not widely available to support the product or because of a lack of iterative research. By contrast, of the 13 products in the third (successful) group, most were funded between 2009 and 2011, suggesting that the program is homing in on the factors that affect success.

From 2012 to 2013, products developed through the IES SBIR program were used by more than 1 million users in schools in all 50 states, and the number may be increasing. Several products have won prestigious awards and many developers have published in peer-reviewed journals. Several grantees have won further R&D awards from IES, NSF, and the Bill & Melinda Gates Foundation. So far, two products have gone on to efficacy testing. Five years ago, Dr. Metz said, the IES SBIR program was unknown, but it has since been recognized by many high-profile stakeholders.

Dr. Metz identified a number of factors contributing to the program's success:

• Aligning RFP requirements with the IES goals and framework
• Bringing for-profit technology developers into the research field
• Initiating the Fast-Track award in 2006, allowing companies to bypass Phase 1 when appropriate
• Emphasizing the importance of commercial pathways from the outset (so reviewers can choose to invest in those with viable paths)
• Engaging in targeted outreach to potential grantees
• Providing substantive technical assistance
• Engaging a strong team of reviewers
• Closely monitoring research and commercialization efforts and providing substantive feedback
• Actively disseminating program results, for example, through an IES SBIR YouTube channel

In conclusion, Dr. Metz said the program has not yet reached the goal of equivalence to the research development grants. The next steps are to strengthen the research and get more products into efficacy trials.

Discussion

Susanna Loeb, Ph.D., asked in what areas the IES SBIR is particularly interested. Dr. Metz responded that the program has funded a lot of games and learned a lot about iterative development in the process. Games are just one area of interest, however. In response to Dr. Yoshikawa, Dr. Metz said the research takes place in both private and academic settings. Usually, an academic is involved either as part of the developer's company or as a consultant. Grantees can use up to $450,000 of the award to subcontract to an outside firm for additional expertise. Dr. Yoshikawa said many students have entrepreneurial interests, and he wondered whether the SBIR has a standard approach to combining research with entrepreneurship and commercial development. Dr. Metz said there is no standard approach, but he could provide guidance on request. He added that he tries to connect firms with industry people who have niche-specific expertise for technical assistance when possible.

Dr. Singer said she was surprised to learn about the program, which equates to providing venture capital to small firms and has success rates as high as or better than most venture capitalists. She asked whether SBIR funds could be used for other ED purposes, such as dissemination and communication of research. Dr. Metz said the program is based on the premise that small businesses can provide the most innovative solutions. It picks up where venture capitalists leave off, funding high-risk efforts. There is not a lot of venture capital for education because the reward is not obvious. The program could address other ED priorities, and recently, some products have focused on tools for education researchers.

Dr. Granger said one way to connect research and practice is through tools and materials. He wondered whether a developer could create a technology platform that would help principals observe teachers in class, for example. Dr. Granger asked about other lessons learned from the SBIR program, and Dr. Metz offered the following keys to success:

• Ensuring that grantees have all the components in place when they start, such as teachers to test products and mechanisms to support iterative research
• Addressing project management issues
• Including a well-versed education researcher in the development process
• Including a well-versed and well-connected commercialization expert on the team
• Paying attention to what works for entrepreneurs, which includes publishing and presenting the product, providing a great website about the product, and creating videos that connect to various audiences

Dr. Long asked what support the program provides to ensure that good products progress to the launch phase and how that support could be applied to other IES products. Dr. Metz said proposal reviewers have to see a strong pathway—and multiple pathways—for the product before approval. For example, the proposal might include a publishing company with a licensing agreement for distribution. From that point, Dr. Metz works with grantees to chisel out an approach for getting into the market. Each case is unique, he said, but there are at least 20 models for what works.

Follow-Up Item

Dr. Metz will provide to the board a summary of the successful market launch approaches that have been employed.

Dr. Long clarified that the grantee is responsible for proposing a marketing/dissemination plan and for having the connections in place to achieve it. Dr. Metz agreed, noting that the central mission of the SBIR is to support commercially viable products. As such, there may not be a lot of overlap with research funding, but there may be applicable lessons.

In response to Robert A. Underwood, Ed.D., Dr. Metz said many technology firms are funded by venture capitalists, incubators, or other small business supports to develop models. The IES SBIR is just one small program; the field is exploding, said Dr. Metz. Dr. Easton added that the SBIR is a mandatory set-aside; the individual agencies set the priorities for their programs with approval from the Small Business Administration.

Dr. Loeb said that with so many users, lessons could be learned about the use process alone—such as what activities people find difficult. Dr. Metz said that good companies collect and use data to refine their products, and the SBIR would like to help grantees get the data more easily. Dr. Long further wondered whether use could be linked to outcomes. Dr. Metz said a great next step for the program could be determining how to embed assessment of use of the products into the postaward process. Grantees are not currently required to conduct efficacy testing now, and many have limited capacity, said Dr. Metz. However, there are some requirements that grantees collect enough research data to apply for efficacy research funding later.

In response to Dr. Loeb, Dr. Metz said some IES researchers have created products through research grants and then won grants from the SBIR program to pursue commercial development.

Dissemination of IES-Funded Research by Researchers
A Framework for Thinking About Dissemination
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild framed her current thinking about dissemination in terms of the assets that programs and grantees may have:

• A clearly defined, specific audience that knows and uses the products
• A basic infrastructure for producing and distributing products
• Products appropriate in content, presentation, and media format for the intended audience

Among NCEE programs, the Regional Education Laboratories (RELs) have a built-in audience that they can identify through regional requests for information and a solid infrastructure, including an online presence. The RELs' products could be improved, but there are efforts to use multimedia and make the products more engaging. The WWC has a good infrastructure, but its audiences are not well defined, and the products are mixed as well.

Dr. Neild then described efforts to better define the WWC's audiences using the following matrix.

Products for adoption include the intervention reports, which users consult when they identify a need. The practice guides are used for instruction and organization. Various products may make the case for policy support, study design, and research topics. Dr. Neild said the products and the dissemination/engagement strategies should work together, and much could be learned from the RELs and research alliances on engagement.

In the past, the WWC developed practice guides on the basis of perceived need but did not think about the specific audiences it could reach. Now, more effort will go into identifying the audience in advance, determining how that audience typically gets its information, working within the networks that serve that audience, and consulting with representatives of the audience about needs and outreach strategies. Instead of having one dissemination strategy for all products, the strategy will be customized to the audience's needs, Dr. Neild said.

In addition, NCEE is developing a circle of “critical friends” who will provide constructive criticism. For example, members of the Education Writers Association told NCEE that they want more video products. NCEE is also seeking more input on how to improve what it already has—such as the suggestions from board members to simplify the website and improve the WWC. Dr. Neild said more effort must be made to explain the WWC and other programs to potential users, and NCEE must learn more about how people outside the research field use its products. Dr. Neild also noted that NCEE needs to better understand how others interpret the products it produces and how to clearly explain the NCEE's intent.

Finally, Dr. Neild said resources are limited, so priorities must be identified. She cautioned against rushing into technology as a Band-Aid. Investment in the RELs is 10 times higher than in the WWC, she noted.

Discussion

Dr. McLeod was among several board members who praised Dr. Neild for moving NCEE forward in its thinking about dissemination in response to board discussions. However, she pointed out, charter schools and how they use the products are rarely included, and she asked that Dr. Neild keep charter schools—both large systems and small schools—in mind.

Dr. Granger suggested reaching out to other constituent groups for constructive criticism. Teachers, for example, may find it difficult to find the answer to the question, “How do I instruct?” in WWC research products, because research may not be conducted in ways that clearly answer that question. He also suggested conserving limited resources by focusing on the source material that the NCEE already has and how to use it in new ways. Dr. Granger said RELs may know the immediate audience of those who contact the RELs but not the larger audience of those who ultimately use the products. There does not seem to be any thinking about how the RELs' products could translate across the country. Dr. Granger advised Dr. Neild to figure out what NCEE can do and limit its efforts to what it can do well.

Dr. Ford said pushing information out is one method of dissemination, but he asked how NCEE invites practitioners in. He wondered whether practitioners even know there is a multimillion-dollar government resource available to them. He advised reaching out to schools of education to target graduate students and education leaders. Dr. Ford also asked how NCEE attempts to distinguish its research-based products from competing products. Dr. Neild said some practitioners know about the WWC, although she wished more did. She agreed that NCEE should do a better job of positioning the intervention reports as the go-to source. Dr. Ford added that schools need to improve their organizational infrastructure to give people the time and motivation to access the available resources and translate them into useful classroom practices. He said that even a good dissemination plan will not help if the infrastructure is not in place in schools.

Dr. Long agreed that educators are faced with information overload. The WWC was supposed to serve as an objective resource, but that is a hard goal to accomplish, she said. Dr. Long appreciated Dr. Neild's attempt to categorize the products and audience but said we still will have to determine which are the best products and how to disseminate them. She reiterated a suggestion from previous board meetings that IES conduct a pilot study with the 10 largest school districts to determine what products they use and what works within the structures they have. There should be a mechanism for interacting with teachers and schools to get quick feedback.

Dr. Hedges agreed on the importance of working directly with users but added that it is very important to be mindful of their limited time. He urged Dr. Neild to pay close attention to understanding how users interpret what is said.

Dr. Singer compared the visual impact of the WWC website with that of two far more “glitzy” commercial education research sites, Edutopia and Pearson. Education is a huge enterprise, she said, and if the WWC tries to be all things to all people, it will fail at all of them. Teachers, parents, and others use these websites—but IES is focused on research. With budget crises forcing IES to make hard decisions, Dr. Singer suggested limiting the focus of the WWC to the educational research community. In doing so, NCEE can better pinpoint what its audience wants and needs. Moreover, the NCEE could recognize many important issues while targeting the hot topics of the day. For example, right now, the WWC could pull together and disseminate information for decisionmakers about the Common Core State Standards. Dr. Long agreed that establishing priorities is reasonable, especially when budgets are constrained.

Dr. Yoshikawa asked what products are available for policymakers, especially for planning issues, and how they can get information rapidly. Dr. Neild said products for policymakers pose an ongoing challenge. The WWC looks primarily at branded curricula. It is hard to pin down policy issues and summarize the relevant research in an engaging way. NCEE is now starting a new approach, but Dr. Neild said it may be better to put policy products on the back burner. Dr. Yoshikawa said conversations with key informants, such as IES or other agency policy specialists, may be helpful. Dr. Long said NCEE has done a good job speeding up the pace of developing reports, but there seems to be some support for focusing on hot topics as they arise.

Grantees and Dissemination
Thomas Brock, Ph.D., NCER Commissioner

Dr. Brock said grantees do a good job of disseminating their findings within the field. They receive a lot of encouragement and have lots of platforms to do so, but there is not as much emphasis on grantees communicating with people outside of research—policymakers, practitioners, parents, and students, among others. He posed the question to the board of whether this shortcoming should be tackled and how.

Dr. Brock reiterated that IES is implementing a public access policy that requires that all IES-funded studies submitted for publication at least be sent to ERIC, and those studies are probably reviewed by the WWC. The new Partnerships and Collaborations Focused on Problems of Practice and Policy grant program aims to encourage direct engagement between researchers and practitioners at the outset, so there are some mechanisms on that front.

For education research grants, the newest RFA will require a dissemination plan as part of the overall research plan. The RFA asks that applicants (1) identify the audiences (beyond other researchers) who could benefit from the research and (2) discuss how they could reach those various audiences through publications, presentations, or other products or methods. Dr. Brock pointed out that not all grant-funded activities merit the same level of dissemination, and expectations are higher for some types of research (e.g., efficacy trials) than for others (e.g., exploratory studies). The RFA also encourages the use of new technology and other mechanisms to disseminate products cheaply and efficiently.

Dr. Brock said staff have discussed whether dissemination should be a separately-scored category but decided that was too large and sudden a step. He posed three questions for the board's consideration:

• How should the importance of research dissemination be highlighted in the RFA for research grants?
• How much emphasis, if any, should dissemination plans and activities receive in the review and scoring of proposals by peer reviewers?
• What resources might IES offer to grantees who request assistance or guidance on dissemination?

Discussion

Dr. Long said the best researchers are not always the best communicators. However, with funding, our expectations should be high. In distinguishing among highly-rated proposals, dissemination should be considered. Dr. Long said HHS includes dissemination as part of its impact criteria; she suggested NCER include dissemination in the context of significance—that is, once the research is completed, will it have a significant impact?

In response to Dr. McLeod, Dr. Brock said there is room in the budget of the grant for dissemination efforts, but there is no identified percentage of funding for it.

Dr. Gamoran said researchers are often not the best people to disseminate findings to broad audiences, but that role could be assigned to others. Even within the context of dissemination of findings within the research community, said Dr. Gamoran, expectations should be differentiated according to the IES research goals.

While it's attractive to think all research needs wide dissemination, said Dr. Hedges, that is not actually a wise approach. Consider the importance of the research. In some cases, the scale is small and the findings should be directed to the research community for further testing. Even with more sophisticated research, consider whether the findings are strong enough to merit dissemination. These issues should be thought through if dissemination is going to be weighed heavily in the review process, Dr. Hedges said.

Dr. Loeb suggested that dissemination be covered by a follow-up grant, so that resources are allotted once it is clear that dissemination is needed. We are still trying to understand effective mechanisms of dissemination, she noted. Getting information out is hard, and you have to consider the payoff. We need to ask whether we want to make huge investments or whether sending little bits of information to lots of people gets the kind of use we want, Dr. Loeb noted.

Dr. Easton said he believes that researchers should learn to communicate better. The predoctoral training program suggests—but does not require—that communication be part of training. Dr. Easton said he would like to build a culture where communication is part of our jobs.

Dr. Miller said the NIH embedded a dissemination plan in some of its materials. He said NIH was nervous about the dissemination of unreplicated research, so it emphasized appropriate dissemination to potential audiences as the science warrants. Including the requirement gets researchers thinking about dissemination before they submit their applications—about the potential impact of their research on different audiences and how they would disseminate findings. Dr. Miller added that NIH had applicants submit a formal publication plan as part of their proposals, which reduces misunderstandings about dissemination.

Dr. Granger said the key message is that ED should not require all applicants to take the same approach but rather should encourage them to think about appropriate means of dissemination according to the audience and relative to the findings. His organization offers follow-up funding for dissemination if research findings merit that. It also has guidelines for dissemination but does not score proposals on the basis of dissemination plans, because it is not clear what constitutes a good plan. Dr. Granger said his organization requires applicants to at least communicate with other key players as they conduct their research to enhance the research itself. Applicants are encouraged to consider not “Who is my audience?” but rather “Who are my partners?” Dr. Granger suggested that ED emphasize communication with other researchers in the field.

Dr. Singer agreed with Dr. Easton that researchers should learn to communicate better, and anything IES can do through training or other means to create incentives for better communication would be great. To determine which research should be disseminated, Dr. Singer suggested focusing on the concept of translational research, which has caught on in medicine, and encourages researchers across fields to communicate about practical applications of their work. At present, the approach to disseminating education research is “to let 1,000 flowers bloom,” said Dr. Singer. An alternative would be to prioritize those studies that would be eligible for the next step toward translation. Making translational research a priority is different from having a dissemination plan that requires the researchers to communicate their work. It is the effective translation of work that we should be concerned about, said Dr. Singer, not the ability to disseminate it.

Dr. Gamoran noted that translational research is a special field. The more we learn about the role of evidence in decisionmaking, the better positioned we will be to establish a field of translational research in education. Having a dissemination plan is different from being a good communicator, said Dr. Gamoran, and it would be a mistake to score proposals based on those dissemination plans. Recognizing the value of interaction with practitioners on the quality of research—not findings—as Dr. Granger suggested, is another aspect of communication, said Dr. Gamoran.

Dr. Long said it is reasonable to train and expect researchers to summarize their findings into a one-page summary that is not technical, for example, but teaching them to communicate with media may be too much to ask. Requiring applicants to talk about dissemination sends a signal that they need to think about it in advance and that they will have to discuss dissemination further when findings are available. The importance of considering potential partnerships should also be emphasized early on. Applicants should understand that dissemination of findings should be appropriate to the project. They should be thinking about it all along the way, not just at the end, said Dr. Long.

Dr. Long reiterated her skepticism about including dissemination in the research plan; it is clear to reviewers how to score a research plan, but assessing dissemination is less clear. She again suggested that dissemination be included in the application section on significance, which already requires subjective judgment on the reviewers' part.

Dr. Easton pointed out that the real focus of IES is not just to bring research to practice but to bring practice into research. That concept should be included in any discussion of dissemination, he said.

Dr. Yoshikawa suggested IES consider what universal and targeted resources it might offer to applicants and grantees to assist them in thinking about dissemination before findings are known, such as grantee meetings. Once findings are known, IES can play a role in steering grantees toward the best mechanisms for dissemination on the basis of the relevance of the findings, the topics, and the target audiences. Dr. Yoshikawa also recommended targeted professional development and assessment of the most cost-effective mechanisms of dissemination.

Dr. Ford raised concerns about transparency in federal funding. That is, you may not want to disseminate bad results, but the findings of government-funded projects should be available. The public's trust in government is another component of this discussion, he noted. Dr. Granger agreed that publication bias (i.e., not publishing negative findings) is an important consideration but should be addressed at a level other than the IES grantmaking process.

Dr. Granger recommended IES consider creating some vehicles for disseminating clear findings, such as panels of key intermediaries who convene periodically and receive briefings on research. His organization funds the American Policy Forum and requires that senior researchers present their study findings to the Forum; the approach has been successful in establishing relationships.

Dr. Brock summarized the key points of the discussion as follows. Be cautious in adding dissemination to the RFA, do not break dissemination out as a separate category for scoring, and consider the ideal placement of a dissemination requirement. Dr. Brock said that as a new commissioner, he has become aware that many people do not know what IES does, including people in Congress and at the Government Accountability Office (GAO), and that is a great challenge. The livelihood of IES depends on meeting that challenge, he concluded.

Lunch
Participants adjourned for lunch at 12:26 p.m. The meeting resumed at 1:30 p.m.

Learning To Use New Technologies
Games for Learning
Constance Steinkuehler, Ph.D., Senior Policy Analyst, OSTP, Executive Office of the President; Associate Professor, University of Wisconsin—Madison

Dr. Steinkuehler said that although we may think of video games as toys, the market is varied and the industry is booming. Sales figures show that the gaming market dwarfs all other major entertainment media forms and is among the fastest growing sectors of the U.S. economy. It is estimated that 99 percent of boys and 94 percent of girls play video games, and while the gap is closing, differences in gender preferences for games remains strong.

Dr. Steinkuehler described five important affordances of video games:

• Games have broad population reach. Video games reach 97 percent of youth. Even among adult Americans, 72 percent have at least one dedicated gaming console or other apparatus.
• Games are models or simulations. Adding goals and rules to a model turns it into a game. Even when people play for fun, they can build models using, for example, real-world maps and information to evaluate the allocation of resources.
• Games are an architecture for engagement. Games tend to be provocative. Engagement matters. A recent study showed that struggling readers performed as well as good readers when they were allowed to choose text in the context of a game. We may think we are looking at academic performance, Dr. Steinkuehler stressed, but it matters whether a student is engaged in the first place.
• Games affect cognition and behavior. For example, Dr. Steinkuehler said, some neuroscience research demonstrates that games lead to improvements in senior citizens' ability to handle distractions while driving. While some in the education community believe there are no data on the effect of games on learning, Dr. Steinkuehler said a recent meta-analysis of research showed that games can yield significant improvements when compared with traditional instruction.
• Games leave behind “data exhaust.” Dr. Steinkuehler said that crafting a great game that gives the user interesting choices generates a wealth of telemetry data on choices over time that could be used to predict later performance. Researchers can also look at failure and determine whether the user is floundering or testing the system. Dr. Steinkuehler said she would like to see more attention to the potential of using games as a form of assessment.

Games can provide a lot of data that feed into continuous improvement of the digital product. For example, “A–B testing” is a standard method of randomizing consumers to one of two conditions and assessing the differences in responses. It may help identify how to tailor products to subpopulations.

The White House is interested in games as a vehicle for cognitive change and even their potential impact on policy, Dr. Steinkuehler said. For example, using smart grid technology, a power company can share usage information with homeowners that can drive individuals to change their consumption patterns dramatically. In classrooms, games can not only aid in learning but also serve as a platform for engagement.

Dr. Steinkuehler said she is assessing the current capacity of federal agencies in relation to video game technology. The IES SBIR program is a good example of product development. The federal SBIR program spreads across multiple agencies, and many awards have gone to game-based technology. Moreover, more RFAs specifically include games. HHS and the Department of Defense have even larger investments in video games than ED.

Dr. Steinkuehler said her ideal research portfolio would include investment in games as assessment systems, big data and computational social science, and data infrastructure via alliances. She pointed out that there are opportunities to leverage big data sets on what students learn across different contexts to identify patterns, make predictions, diagnose problems, and determine whether to intervene.

The characteristics of an ideal portfolio would include the following:

• Interdisciplinary teams with credentialed designers. Direct engagement with industry talent is important for creating school games.
• Ecologies of learning that connect in-school and out-of-school activities. Dr. Steinkuehler said the issue of equity is on the horizon because of the lack of political will to connect kids outside of school digitally.
• Dissemination plans. Including and funding dissemination plans is critical. There is some question whether such a requirement will shut down creativity. Dr. Steinkuehler said more deliberation is needed about research and communication, such as whether awards should be based on the researcher's ability to distribute the findings. In technology, a lot of great ideas are executed and proven, but the product sits on a shelf because the funding ran out and there is no money to “harden the market” and distribute the product.
• Attention to scale-up. In technology, researchers must consider markets, or big, broad user bases.
• More strategic relationship with the notion of risk. Dr. Steinkuehler pointed out that the IES SBIR's overall success rate of 33 percent would be considered huge by any venture capitalist. But federal agencies and program officers often do not manage risk overtly and tend to treat all investments as equal in risk. In game development, makers fail early and often. Funding should include processes that enable the risk-taking required to build technology products.

Dr. Steinkuehler concluded by pointing out the tremendous potential for education technology. Private, commercial developers are expressing a lot of interest. If IES does not fill the need and do it well, the market will, and not necessarily in ways we want it to, she noted.

New York Hall of Science
Margaret Honey, Ph.D., President and Chief Executive Officer

For example, NYSCI created SciGames, a distributed technology platform intended to support learning across formal and informal environments. It is the largest science playground in North America, said Dr. Honey, and it offers physical activities and challenges tied to curricula by taking advantage of sensors and other technologies. For example, children can ride a giant slide on mats that have a tiny gyroscope and barometer attached to capture data instantly. Teachers and their students can set up challenges or pose questions that relate to learning about potential, kinetic, and thermal energy. The students take the mats down the slide and get immediate feedback. Dr. Honey said the activity allows the students to connect the experience of the ride with the data about it. The data are stored in the cloud and can be used later in the classroom for deeper exploration of scientific principles.

The Digital Design Lab currently under construction will bring design, construction, and STEM concepts to the floor of the museum. It is intended to move away from didactic learning to more design-based input. For example, the Digital Design Lab allows users to manipulate photo and video images, using a “noticing tool” that lets them customize the experience and create something compelling for themselves. Dr. Honey described the image manipulation activity in more depth, pointing out that it engages math skills while also guiding users to create narratives and use technology.

NYSCI is supported by several funding streams that enable it to conduct some rigorous research. The initial formative research focuses on appeal, usability, and comprehensibility of the activities. The summative research includes an RCT of SciGames supported by i3 and a quasi-experimental study of the Digital Design Lab funded by the Bill & Melinda Gates Foundation.

The core idea behind NYSCI, said Dr. Honey, is to develop the capacity, using technology, to encourage young people to generate their own data through their own experiences and give them the tools to analyze the data in a thoughtful, intentional way.

Discussion

Dr. Gamoran asked how we can take advantage of the unique capabilities of video games used as learning tools to get data. A National Academy of Science report on games and learning noted many complexities in using games for assessment as we conventionally understand it, because not all the users are doing the same thing, on the same scale, or in the same framework or rubric. Dr. Steinkuehler responded that we can collect great, complicated data, but we are not sure how to mine it yet. At least initially, the games have to be tied to existing assessment approaches. There is a lot of heavy lifting yet to do, said Dr. Steinkuehler. Studies can be designed to compare users doing different things, and a good design can identify what the user did not do or where the user was confused or mistaken. Like health data, said Dr. Steinkuehler, big data provides a lot of information but also a lot of complexities to consider when looking for patterns.

Dr. Honey said NYSCI picks concepts that are central to its curriculum and can do efficacy studies as a result. She emphasized that there is a difference between learning and testing. When we develop tools for school use, we do not just want students to improve their test scores, added Dr. Honey.

Dr. Gamoran said the game environment offers an opportunity to measure more learning in more authentic ways, but we still have to solve the challenges of understanding what you are measuring. Dr. Honey noted that NYSCI holds an annual Maker Faire, where young people demonstrate a range of creative, innovative technological inventions. At NYSCI, Dr. Honey sees how students learn through problem solving, iteration, and collaboration. It is possible to build applications that capture a student's work on an iPad and that could be made available to researchers for more systematic study, said Dr. Honey. The methodology is different, but it can be potent and represents what is often lacking in traditional research approaches.

Dr. Steinkuehler emphasized that a good educational product poses a task that the student finds genuinely worthwhile, and research may be limited by the lack of such products. She pointed out that in her research on children and young adults, it is evident that having differentiated knowledge and expertise is valuable and meaningful to them, but in school, unique knowledge is not valued. She asked how we could develop rigorous assessment techniques around something that young people value—that is, knowing something different from your neighbor. The paradox of building technology for classrooms is that teachers and students want products they can tailor, but research funders want the products to be uniform to facilitate comparisons.

Dr. McLeod said she sees the potential for games to serve as strong assessment tools. If young teachers are more likely to embrace technology in the classroom, while older teachers are more reluctant to do so, Dr. McLeod wondered how we can make the transition to a technology-friendly classroom. Dr. Honey said reluctance has less to do with age than underlying pedagogical practices. If teachers are focused on test preparation and skill-building, technology does not fit so well; if they take a more project-based approach that seeks opportunities for deeper inquiry, then technology is a powerful catalyst. Unfortunately, in a culture with high pressure around standardized tests, we're seeing a resurgence of “data analytic systems” that are not much different from the technology-based curricula of the late 1980s and early 1990s. These technologies are not bad—except when they are directed at low-performing environments as a substitute for instruction, said Dr. Honey.

Dr. Long said it seems that technology was once seen as a silver bullet, but too many things failed. She asked what we have learned about where games may be useful, that is, what areas of knowledge or types of skills games could help develop. Dr. Steinkuehler said she could speak to areas that are overlooked, such as language learning and literacy. In the health care arena, most of the biggest investments are in products that essentially use games as an interface between the patient and his or her data (setting goals for behavior change, visualizing progress, etc.), and there are likely applications that can translate to education. As a vehicle for assessment and as components of a curriculum, games work best within the ecosystem of other materials, said Dr. Steinkuehler. Games are not a substitute for teachers; you still need that teacher-mentor relationship, she said, but games can complement that relationship well.

Dr. Honey emphasized that technology designed to carry the full burden of learning (e.g., training systems for military personnel) is different from technology that invites inquiry. We know that highly trained, high-quality teachers make a difference for young people, said Dr. Honey.

Dr. Buckley pointed out that NCES has assessed hundreds of thousands of young people and that it has found simulation to be very effective in some assessments, yet difficult in others. He said if you take into account novel versus traditional forms of assessment as well as novel versus traditional constructs for measurement, it becomes clear that the most effective approach is measuring traditional constructs using novel approaches. Dr. Steinkuehler said the National Academies report pointed out that most studies reviewed did not define the learning goals.

Dr. Honey questioned whether supporting learning means doing things that are measurable. We know that the ability to access information online and do projects using an iterative process are powerful ways to learn, and we do not necessarily have to measure those skills, she said. Dr. Steinkuehler said she believed that measuring such skills is possible, and she hoped researchers would not always fall back on assessing only known, desired constructs. Dr. Honey said that some constructs remain the same, only now we can look at them on a broader, richer scale.

Dr. Ford said that to transform learning in classrooms, the work has to be meaningful (and even the best schools give students a lot of busy work). Also, we need to foster a paradigm shift about what we think children should be able to do. They can come in to the classroom with the ability to ask questions, design experiments, fail, etc. Such a shift would also mean that teachers would have to assess the results of the learning process differently, said Dr. Ford. Whole new ways of teaching, including using technology in a personalized way, may not be scalable, he added. Dr. Steinkuehler agreed, saying that we are moving from being data-poor to data-rich, but little attention is paid to assessing the interaction between teachers and students.

Evaluating the IES Research Portfolio: What Is the Best Approach?
Opening Remarks by John Q. Easton, IES Director, and Bridget Terry Long, NBES Chair

Dr. Long explained that she and Dr. Easton have been discussing how to make IES a continuously improving organization. Questions have arisen about how to begin evaluating the IES research portfolio to assess whether IES is funding the right kinds of studies and what IES priorities are.

Dr. Easton said IES wants the board's input about the questions that IES should be asking about its funded research program. The Education Sciences Reform Act (ESRA) requires research around 11 broad topics, but it does not provide specific goals for IES. The portfolio has been sprawling, and now IES faces budget constraints. Even without those constraints, said Dr. Easton, it is important to ask the bigger questions. Are we building the scientific evidence base? Are we learning? Are we making a difference? Dr. Easton noted that IES is not committing to hiring an external consultant to conduct an evaluation; rather, IES wants the board's guidance on its research questions for the future, specifically for NCER and NCSER.

Discussion

In response to Dr. Loeb, Dr. Easton said the portfolio is currently categorized organizationally into five major grant programs. Dr. Gamoran said it is difficult to determine in the short-term whether a research portfolio is effective. Dr. Hedges added that the Spencer Foundation's evaluation of its predoctoral fellows program spanned a decade of funding, followed by 3-4 years to assess the impact. Dr. Easton responded that IES is 11 years old, and he expects it to remain in place for many decades, so it is appropriate to take a long-term view.

Dr. Long noted that the previous chair pressed the board to think about whether IES should invest in safe bets or high-risk projects with the potential for high reward. In addition, the board has discussed how money is spent in the NCER and NCSER portfolios, but there is still no sense of the impact those dollars invested are having, said Dr. Long. She asked for input from the board about how IES should determine its priorities.

Dr. Singer suggested better clarification of the purpose of this exercise. She noted that IES may need to justify its existence as ESRA comes up for reauthorization by demonstrating what has come from the money spent so far. Alternatively, IES may want to take advantage of the budget crisis to develop a clear decisionmaking process. One dramatic way to assess the value of investments is to cut funding to all programs except those that can craft a justification sufficient to save the program. She urged IES to take a hard look at its processes and priorities. Dr. Singer also asked where the decisionmaking for funding—such as the IES SBIR program—comes from and whether there are other factors driving decisionmaking about which the board does not yet know.

Dr. Hedges said that funding decisions should be based on IES's most important mission, which is supporting research. From there, IES should look at the categories in the portfolios and determine which ones appear to be making progress—that is, leading to other research programs, building on other work, progressing through IES's goal structure, and being validated in larger trials. The hallmark of a good portfolio is the progressive nature of research, said Dr. Hedges. Also, it is important to ask whether the work has had an impact on practice or policy—which is not easy to answer but not impossible.

Dr. Miller said NICHD went through a branch review process to map out its future. He said the process was very time-intensive, but the effort produced a rich conversation about the meaning of “impact.” To conduct the process, NICHD carved up the portfolios in various ways—by science type, population, type of research, etc.—and engaged program experts to identify gaps. It was useful to compare previous and current priorities, said Dr. Miller. A lot of the data needed for the assessment was not readily available. Dr. Miller clarified that the assessment was based on the assumption that NICHD knew what its goals were and that the goals could be revised on the basis of the findings.

Follow-Up Item

Dr. Miller will provide NICHD's final report to the directors of the NIH Institutes and Centers on its branch review process as a model for consideration for IES.

Dr. Loeb suggested that IES consider where it has capacity to influence the field. IES should identify the research gaps and target incentives at filling those holes. Taking on the whole portfolio is too big a task, she said.

Dr. Buckley noted that instead of the extreme measure of “zero-based budgeting,” in which all programs are cut unless they prove their worth, IES might consider modeling each program at three levels of budgeting. Dr. Singer agreed that evaluating the programs using a tiered approach would be better than cutting all program budgets.

Dr. Yoshikawa suggested defining the units of analysis. That is, should the research portfolio be judged on whether it is meeting IES goals, or is IES seeking to evaluate the processes of determining the funding priorities? Dr. Yoshikawa also asked whether IES has a handle on who uses the research it produces and for what purpose. In addition, IES should understand what drives researchers' decisions about whether to pursue IES funding. New kinds of analyses are emerging that are not represented in the IES portfolio, said Dr. Yoshikawa, such as public use of large data sets. He asked whether other areas or methods of research exist that are not incorporated in the body of IES research. Finally, he said, IES should seek to understand the impact of its funded research.

Dr. Granger said that it is useful to distinguish goals from strategy. The analysis of the portfolio should take into account what money will be freed up in the near future. Dr. Granger said he is frustrated by how little we know about why anything works. Without that understanding, we are just funding endless trials across endless conditions. We also need to identify and bolster the things that the research community needs to build its capacity. Until now, the enterprise has been driven by what researchers thought they needed. Dr. Granger said now is the time to assess which questions we are not answering. Finally, he suggested bringing in outside expertise to evaluate the grant scoring process. Dr. Easton pointed out that at the end of 2015, the Reading for Understanding program will end, freeing up $20 million annually.

Regarding grant scoring, Dr. Long said that even though it is recognized that the scoring process involves some subjectivity in the funding phase, we treat differences of one-tenth of a point as though they are very significant. In terms of approach, Dr. Long said IES has been providing a little bit of funding to a lot of projects, as opposed to providing a lot of funding to a few. If IES decides to switch gears, it needs to justify the change. However, said Dr. Long, the current approach is increasingly unsatisfying, especially with declining funding. She recommended considering what the federal government can do through research funding that no one else can—such as large-scale studies, forging partnerships, and basic research on fundamentals. Dr. Long said she would like to better understand how IES interacts with other ED research programs. She also suggested considering more strategic partnerships, like the IES/NSF framework project, to make dollars go further.

Dr. Yoshikawa agreed on the importance of cross-agency collaboration. He noted that children spend a lot of time outside of formal learning environments, and there may be opportunities to partner with HHS, which focuses a lot on families.

Dr. Loeb said that in the private sector, there are few opportunities for funding smaller, researcher-driven studies based on emerging concepts, and she did not want to limit opportunities even more. Instead, she suggested IES focus on how it chooses to meet its goals.

Dr. Singer said IES's biggest impact has been changing the dialogue in the larger education community and moving people into education research, and she cautioned against underestimating that impact. Dr. Hedges added that IES has also strongly affected the community's capacity to conduct research. Dr. Singer said many questions remain about the budget and also whether spending a lot of money on small projects is effective, whether building capacity within schools is cost-effective, and what the GAO report (which will come out in October) aims to achieve. She said IES needs the dispassionate assessment of outside experts to assess the situation, and Dr. Gamoran agreed.

Dr. Granger was skeptical about how much IES-funded research has done to advance the state of science around education. If building capacity is a key function of IES, he said, it may be time to think about other forms of capacity.

Regarding external review, Dr. Long said it should be focused on a needs analysis for education, but there is no need to enhance the capacity of peer reviewers outside the ED process. She asked board members for input on what is being ignored or dropped from the portfolios of other funders, such as universities.

Dr. Granger expressed skepticism about the usefulness of external evaluation. He said staff can provide a lot of insight on the gaps in the portfolio. Also, “critical friends” and the board can give their input. Dr. Easton said the conversation is increasing his sense of urgency. He said that perhaps IES does not need to reflect on its efforts and justify the past so much but rather should consider how the future should be different. Dr. Easton said he appreciates the board's willingness to push back, because ultimately it pushes IES forward. Dr. Long emphasized that IES should keep funding things that work well, and the commissioners report that many areas are improving.

Dr. Singer reiterated that the board needs a better understanding of IES's budget and more perspective from staff about where the board can help.

Follow-Up Items

At the next board meeting, IES will present an overview of the budget at the level of the RELs and programs.

At the next board meeting, IES program staff will provide their perspectives on where they need help in evaluating the portfolio.

Dr. Granger said that in discussing the budget, IES should clarify which funds are fungible. Dr. Easton pointed out that the board orientation covers budget issues. Dr. Long said she would like more detail about, for example, the breakdown of funding within NCER grants to understand the relative sizes of the grant programs.

Closing Remarks and Adjournment

Bridget Terry Long, NBES Chair

Dr. Long said that she had not yet drafted the board's annual report but will do so soon. She said the board will review and approve the report via e-mail and teleconference. Dr. Long noted that today's meeting identified several areas for the October 2013 board meeting agenda, but she welcomed more suggestions from members.

In closing, Dr. Long particularly thanked Ellie Pelaez for her assistance with the meeting. She also thanked Vice Chair Kris D. Gutierrez, Ph.D., for the work she put in to organizing the agenda and arranging for presenters for this meeting. Dr. Long adjourned the meeting at 4:05 p.m.

View, download, and print the full meeting minutes as a PDF file (1.3 MB)

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.