Location
Institute of Education Sciences (IES) Board Room
80 F Street NW
Washington, DC 20001

Participants

National Board for Education Sciences (NBES) Members Present
Bridget Terry Long, Ph.D., Chair
Deborah Loewenberg Ball, Ph.D.
Anthony S. Bryk, Ed.D. (by phone)
David Chard, Ph.D.
Adam Gamoran, Ph.D.
Robert Granger, Ed.D.
Larry V. Hedges, Ph.D.
Susanna Loeb, Ph.D.
Margaret (Peggy) R. McLeod, Ed.D.
Judith Singer, Ph.D.
Robert A. Underwood, Ed.D.
Hirokazu Yoshikawa, Ph.D.

Ex-Officio Members Present
John Q. Easton, Ph.D., Director, IES, U.S. Department of Education (ED)
Elizabeth Albro, Ph.D., Acting Commissioner, National Center for Education Research (NCER)
Ruth Curran Neild, Ph.D., Commissioner, National Center for Education Evaluation and Regional Assistance (NCEE)
Joan Ferrini-Mundy, Ph.D., Assistant Director, National Science Foundation (NSF), Directorate for Education and Human Resources
Peggy McCardle, Ph.D., Branch Chief, Child Development & Behavior Branch, Center for Research on Mothers and Children, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), National Institutes of Health (NIH)
Chuck Pierret, Bureau of Labor Statistics
Marilyn Seastrom, Ph.D., Chief Statistician and Program Director, National Center for Education Statistics (NCES) (for Sean P. "Jack" Buckley, Ph.D., Commissioner)
Deborah Speece, Ph.D., Commissioner, National Center for Special Education Research (NCSER)

NBES Staff

Ellie Pelaez, Designated Federal Official (DFO)

ED/IES Staff

Karen Akins
Corinne Alfeld
Lisa Bridges
Doris Dixon
Swati Mehta
Allen Ruby
Kim Sprague

Invited Presenters

Sue Betka, Deputy Director for Administration and Policy, IES
Steve Fleischman, M.A., Chief Executive Officer, Education Northwest
Barbara Means, Ph.D., Director, Center for Technology in Learning, SRI International
Anne Ricciuti, Ph.D., IES Deputy Director for Science
Thomas Smith, Ph.D., Director and Principal Investigator, National Center on Scaling Up Effective Schools and Associate Professor of Public Policy and Education, Department of Leadership, Policy, and Organizations, Vanderbilt University

Members of the Public

Nora Boretti, U.S. Government Accountability Office (GAO)
Megan Foster, Society for Research in Child Development
Jean Gossman, Education Daily
Stephanie Gregg, Washington Partners, LLC
Monica Herk, Ph.D.
Allie Huyghe, Strategic Education Research Partnership (SERP) Institute
Kim Hymes, Council for Exceptional Children
Sarah Grady, National Opinion Research Center at the University of Chicago
Carla Jacobs, Lewis-Burke Associates
Jim Kohlmoos, National Association of State Boards of Education
Myrna Mandlowitz, Southwest Educational Development Laboratory
Michele McLaughlin, Knowledge Alliance
Jeff Mervis, Science Magazine
Kristen Neishi, National Opinion Research Center at the University of Chicago
Juliana Paré-Blagoev, SERP Institute
Scott Spicer, GAO
Erika Thompson, American Association of State Colleges and Universities

Meeting Summary

Call to Order, Chair's Remarks
Bridget Terry Long, Ph.D., NBES Chair

Dr. Long called the meeting to order at 8:34 a.m., and Ms. Pelaez, DFO, called the roll. NBES members unanimously approved the minutes of the June 20, 2012, NBES meeting. Dr. Long welcomed two new Board members, Drs. Hedges and Loeb, noting that the Board has almost reached its full complement. Dr. Long thanked Dr. Herk, former Executive Director of NBES, for her assistance organizing the meeting and putting together the agenda. She hoped Dr. Herk would continue to be involved with the Board. Dr. Easton said that Dr. Herk will be missed, but the Board will have assistance; he said efforts are under way to assure that the Board is supported.

Dr. Long reviewed the agenda in the context of three areas on which the Board generally focuses:

• IES research funding
• Impact through disseminating and scaling up promising practices
• Advocating for support and use of research in policymaking

Swearing-In of New Members
John Q. Easton, Ph.D., IES Director

Dr. Easton swore in Drs. Hedges and Loeb. He noted that the Board now has 13 members.

Update: Recent Developments at IES
John Q. Easton, Ph.D., IES Director

Dr. Easton welcomed the Board members and Ms. Pelaez. In August, IES convened a meeting of outside experts on the use of value-added measures and student growth measures to assess teacher effectiveness. The attendees represented a wide variety of fields. Despite a huge amount of new research in the past few years, some in the field are not fully aware of what others are doing. The IES meeting sought to gather the experts, facilitate conversation, and learn where they agree or differ. That information will not only inform IES research grant-making and evaluation considerations but also serve as a basis for advising ED on policy issues. A detailed summary of the meeting, as well as the short summaries of key research prepared by the participants in advance, will be available online shortly, said Dr. Easton. No specific follow-up effort has been planned.

Commissioner Updates

National Center for Education Evaluation and Regional Assistance
Ruth Curran Neild, Ph.D., NCEE Commissioner

In September, NCEE released the State and District Receipt of Recovery Act Funds: A Report from Charting the Progress of Education Reform—An Evaluation of the Recovery Act's Role, which documents how funding was spent and includes the characteristics of funded schools and districts, amounts, etc. It is part of a larger study of major Federal funding efforts and reflects an NCEE effort to get interim reports out to the public more quickly. Dr. Neild said the What Works Clearinghouse recently released a new practice guide, Teaching Elementary School Students To Be Effective Writers, which she said is her favorite so far, because it offers a good framework and nice examples. It also demonstrates NCEE's interest in providing practice guides that are more narrowly focused and more useful to classroom teachers than previous guides. The What Works Clearinghouse also recently released the Reporting Guide for Study Authors to help researchers, who are submitting materials to the What Works Clearinghouse, better understand the components of research it includes.

Nine of the 10 Regional Education Laboratories (RELs) are up and running (one award is still being contested). The nine RELs have linked with about 70 research alliances with whom they work on a variety of topics. In the first year, the RELs and the alliances focus on building relationships through small technical assistance projects that create community and support shared knowledge. In the absence of a contract for a cross-REL coordinating task (which is under a stop-work order as a result of the protest), IES staff are doing their best to ensure coordination among the RELs and to encourage them to take the lead in areas in which they excel. REL representatives have taken part in Society for Research on Educational Effectiveness (SREE) working groups and panels, and attendees at a recent SREE conference expressed a lot of interest in the RELs' early progress. Dr. Neild noted that NCEE wants to ensure that RELs pay attention to the audiences they serve and produce products that are not only technically sound but also readable, relevant, and useful. Thus, review of products will take those factors into account, and the review criteria developed can be shared across RELs and with IES.

NCEE has held three What Works Clearinghouse reviewer training sessions in the past 3 months. More certified reviewers are needed to meet the demand to speed up evaluation of projects and publish evidence to support scaling up programs. The training sessions also serve as mechanisms to disseminate more research to the field (e.g., to those who attend the training but do not seek reviewer certification). Dr. Neild said NCEE is considering whether to hold more training sessions that would engage more broadly people representing State- and district-level organizations.

In early August, it came to light that some of the older records in the Education Resources Information Center (ERIC) included personally identifiable information. Therefore, NCEE prevented access to those records temporarily. As they are revised, access to the full text of those records will be restored, and the effort is moving along, said Dr. Neild.
Among the notable new NCEE contract awards is one for analytic technical assistance and development that allows a lot of flexibility, so that NCEE can respond to issues that raise concerns for the whole field as they arise. Among the topics to be addressed as part of this contract is a guidance document on seizing opportunities for low-cost, quick-turnaround, policy-relevant impact studies. Dr. Neild pointed out that several NCEE contract awards address postsecondary education, such as evaluations of promising strategies for Federal college access programs and experiments with Pell grant expansions. A new small business contract will focus on assessments of postsecondary research studies for the What Works Clearinghouse.

Discussion

In response to the perception that the previous RELs lacked a coherent approach and seemed to address research issues in a random manner, NCEE instituted a framework for the new RELs that requires them to develop research alliances around certain topics. NCEE emphasizes that collaboration with the research alliances should be driven by actions that practitioners want to accomplish, which can be broad at the outset, such as reducing the dropout rate. NCEE expects RELs to develop cohesive research agendas. Participants expressed interest in the evaluation contract to support professional development of math teachers, and Dr. Neild said she could provide more details about such efforts later.

National Center for Education Research (NCER)
Elizabeth Albro, Ph.D., NCER Acting Commissioner

Discussion

National Center for Special Education Research (NCSER)
Deborah Speece, Ph.D., NCSER Commissioner

Discussion

National Center for Education Statistics (NCES)
Marilyn Seastrom, Ph.D., Chief Statistician and Program Director, National Center for Education Statistics, NCES

NCES recently released results on writing and science assessments from the National Assessment of Educational Progress (NAEP) that involved interactive computerized assessments. In early December, NCES will release results from the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PISA).

NCES redesigned the National Household Education Survey from a telephone-based to a mail survey and contracted the U.S. Census Bureau to conduct the survey; results will be available in the spring. The NCES website includes a link to the 2012 revised statistical standards, and Dr. Seastrom said comments are welcome throughout the month.

The Schools and Staffing Survey (SASS) has been conducted every 4 years since 1988. In conjunction with the Office for Civil Rights, the survey is being redesigned to become a nationally representative survey that will collect data every 2 years. The results will no longer include State-level detail, but the redesign allows more flexibility to gather information on subjects of topical interest.

Once NCES puts in place its Longitudinal Study of Early Adolescence (for students in grades 6–8), it will have data on a continuous cohort of students from kindergarten through 12th grade. Efforts are underway to link NAEP and TIMSS data. NCES is working to improve data quality and increase productivity of data collection for K–12 research. Several activities focus on improving data about educational training—for example, on conditional postsecondary credentials that change in the workplace through different types of job training. The NAEP is also assessing technical and engineering literacy.

For years, said Dr. Seastrom, NCES has collected data at the school district level; to increase transparency, NCES is now extending its data collection to the established boundaries for school service areas. On the program side, it is improving access to program data and exploring ways to make data public while protecting sensitive information. NCES is revising its licensing agreements to make data available to researchers for acceptable research uses.

IES's Peer Review Process
Anne Ricciuti, Ph.D., IES Deputy Director for Science

• To enhance transparency, the names of members of standing panels who reviewed the 2012 grants (i.e., last year's grant applications) are available online, and the chairs of panels are noted. In order to preserve confidentiality, the members who served on small or specialty panels were listed in a combined list.
• To increase the prestige associated with serving on a review panel, IES is considering how to make the Standards and Review Office more prominent on the IES webpage.
• To better understand the diversity of panel members, the paperwork for reviewers now includes an optional section in which reviewers can indicate their gender and race/ethnicity. The information will be used internally to track the makeup of the panels.
• To better support panel chairs, the Standards and Review Office gathered relevant information already published by IES and others in various other formats and organized it into a supplement to the reviewers' handbook specifically for chairs. It includes IES expectations and procedures and general tips on meeting management, using input from experienced chairs. The panel chairs for the upcoming review cycle have been asked for input and feedback on the usefulness of the supplement.
• To assist panel reviewers, the reviewers' handbook has been revised to incorporate more general information (borrowed from IES procedural documents) about the types of panels and reviewers.
• To assess whether the order in which applications are reviewed has a significant effect on review outcomes, IES has randomly assigned the order of review for panels reviewing the main competitions for NCER and NCSER grants.
• To better inform applicants about progress, IES is implementing the Applicant Notification System, which allows applicants to log in online and see the status of the review process and funding. The system is modeled on those of NSF and NIH.

Discussion

Numerous Board members thanked Dr. Ricciuti for her responsiveness. She clarified that the names of reviewers for some competitions were combined into a single list because some panels are so small, so specialized, or meet so infrequently (e.g., one-time competitions for R&D centers) that it would be too easy for an applicant to identify the reviewers on a specific panel and even the reviewers who evaluated that applicant's submission. Dr. McCardle noted that the new IES reviewers' handbook indicates that reviewers are welcome to conduct a post-review debriefing; she suggested that IES stipulate in writing that debriefings should address the review process and that a DFO should be in the room during the debriefing. No further discussion of individual proposals should be permitted at debriefings. Dr. Ricciuti agreed, and indicated that only general issues should be discussed during the debriefings.

Dr. Granger asked for Dr. Ricciuti's assessment of the review procedures for reports that carry the IES imprimatur and are published (and therefore reviewed) by IES. Dr. Ricciuti responded that the report review process generally works well, although she has tried to institute improvements over the years. IES is working more collaboratively with the centers than it had in the past. The review process is modeled on that of academic journal peer review. Dr. Granger suggested collecting feedback about the process from authors who have been through it. He also suggested collecting such information from panel reviewers. Dr. Ricciuti said reviewers complete evaluation forms to identify areas for IES improvement, and the forms have recently been revised to include specific questions about training, review materials, etc. She said IES would consider taking a systematic approach to evaluating the report review process.

Dr. Easton noted that he and Dr. Ricciuti have spent a lot of time over the past 2 years on a handbook that sets expectations for centers submitting reports and for reviewers; that draft handbook is on Dr. Easton's desk for review. He suggested putting the issue of the report review process on a future Board agenda for discussion.

Dr. Ricciuti said the staff of the Standards and Review Office has been working to ensure that applicants get constructive feedback on their submissions. Ms. Pelaez has assisted in managing the logistics contract and online system to improve how well IES tracks resubmissions. It is fairly easy to distinguish a first-time submission from a resubmission, said Dr. Ricciuti, but the system must be revised to identify a second resubmission, third resubmission, etc. She emphasized that IES does not limit the number of resubmissions, but it is watching the NIH process (which does limit resubmissions) closely and considering the approach.

Communication and Dissemination
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild hoped to gather ideas and reactions from the Board about the dissemination strategy for the What Works Clearinghouse. To better assess the question, "Does IES have a good dissemination strategy for the What Works Clearinghouse?" Dr. Neild broke the question down into the following components:

• What are the intended outcomes of the strategy? What are the proximal and distal outcomes intended?
• Who is the target audience(s)? Who are the sub-audiences of the target audience?
• What information does the audience need?
  How does the audience usually get information (i.e., through what channels)?
• What gives the information credibility?
• How can information be packaged so that the audience can use it?
• What do we want the audience to do with the information?

Dr. Neild hoped these questions would spur a more sophisticated analysis of what constitutes good dissemination. She proposed answers to some of the questions but said she welcomes help from the Board in thinking through the questions.

The key audiences for the What Works Clearinghouse are education practitioners—people who make decisions about what happens in schools (teachers, policymakers, etc.)—and education researchers, who use the What Works Clearinghouse to see what others are doing in the field. Dr. Neild believes that education practitioners are the main target; IES wants them to use the What Works Clearinghouse findings to make decisions. In practice, however, the What Works Clearinghouse has more robust strategies for reaching researchers than practitioners. Now that the What Works Clearinghouse has gathered sufficient content and established its credibility, NCEE should focus more on reaching practitioners.

Next, the What Works Clearinghouse must find additional ways to increase demand while also increasing the supply of resources and products. IES has already made some improvements toward this end:

• Made Quick Reviews quick. In the past, when a study was mentioned in the press, the What Works Clearinghouse took 3–4 months to assess the quality of the study; now that process takes 10–12 days. In one situation, a study was mentioned by a small media outlet, and the What Works Clearinghouse was able to produce a Quick Review so rapidly that by the time the study was covered by a major news outlet, the story was linked to the Quick Review.
• Initiated announcements about new products on Facebook.
• Revised product templates to emphasize important information and increase clarity.
• Shortened the online news flashes and focused on using clearer wording.
• Revised the home page to make it less text-heavy and less dense and improve navigability and clarity. The site now includes a rotating header and more prominent links to the companion site, Doing What Works.

Dr. Neild said the What Works Clearinghouse is seeking ways to get more out of its existing products—for example, by developing more practice guides and using other media to leverage them. At the June NBES meeting, Dr. Singer said the What Works Clearinghouse website was too text heavy and could incorporate more video, so Dr. Neild and her colleagues are working with the website contractor to examine the feasibility of producing video "teasers," a Podcast, and other abridged products to spark interest. In addition, IES could develop more products similar to the practice guides but shorter, such as guides that focus on a single practice or policy. Even the current narrowly-focused guides have about five recommendations each, and many educators request more specifics. Shorter, more focused guides are less costly and easier to produce than full practice guides, said Dr. Neild.

The What Works Clearinghouse search function is useful but time-consuming and requires some knowledge to manage successfully. Dr. Neild said the results of the most common searches could be compiled, so that users can quickly obtain a simple, ready-made summary that responds to common questions.

Dr. Neild said most efforts have been focused on the demand side, but the What Works Clearinghouse could provide guidance to practitioners about how to use findings in the form of DVDs or print materials, for example. Products could address broad issues, such as how to interpret null findings.

Effective dissemination involves networks that convey the message. The What Works Clearinghouse and the RELs have worked together for years, and the RELs are particularly good at dissemination. The RELs could help the What Works Clearinghouse better connect with the networks that educators find credible, gather feedback on products, and develop other products that support the What Works Clearinghouse's products.

Finally, Dr. Neild asked for input on framing the What Works Clearinghouse agenda. She noted that she can track numbers of downloads and Web hits, but wondered what questions should be asked to better inform the What Works Clearinghouse products and strategies.

Observations on the What Works Clearinghouse
Steve Fleischman, Chief Executive Officer, Education Northwest

Mr. Fleischman currently directs the 2012-2017 REL Northwest at Education Northwest and has spent many years focused on the production and promotion of evidence for improving education, including some time as the communication director for the What Works Clearinghouse when it began. The What Works Clearinghouse is a work in progress, and he offered several comments intended to support and improve it.

If the What Works Clearinghouse is intended to be a decision-making tool and not only a repository of knowledge, there is a long way to go. Mr. Fleischman contended that education practitioners are the primary audience for such a tool, while researchers are secondary. Researchers produce the work that goes into the What Works Clearinghouse and "consume" the information it provides, but they are not consumers of the prime purpose of the products—that is, to make better decisions about challenges in education. Within the arena of education practitioners, said Mr. Fleischman, teachers are not likely to be among the primary audience, unless they are organized into decision-making bodies. He recommended narrowing the definition of "target audience" to focus on decision-makers and looking more closely about who they are, what they need, and how they can use the What Works Clearinghouse as a resource.

Instead of focusing on one-way dissemination of information, Mr. Fleischman suggested considering how to communicate results and foster exchange of information. The dissemination approach is not very engaging.

To increase demand for products, it may be necessary to change perceptions about the What Works Clearinghouse "brand" through new tools, resources, or mechanisms to engage people who know of it but still do not use it and to attract new users. More thought should be given to increasing the supply of products that help users figure out what to do with research findings, either by improving existing products (e.g., the practice guides) or developing new ones.

With the latest changes, the What Works Clearinghouse website is moving closer to the kind of online resource that users want, said Mr. Fleischman, but he recommended thinking about how we use other consumer reporting sites (e.g., Consumer Reports, Yelp, TripAdvisor, and CNET) to make decisions. While it is difficult for government-sponsored entities to break out of entrenched patterns for presenting information, Mr. Fleischman suggested the What Works Clearinghouse site become even bolder.

Mr. Fleischman praised the new products proposed by Dr. Neild. He suggested looking at the Institute for Healthcare Improvement (IHI) for additional ideas, because it presents information in numerous formats and various media and also because it is focused on improving practices.

The Board should consider what level of experience or knowledge constitutes expertise and whether IES would be comfortable with products that feature nonacademic experts who have developed expertise as practitioners. In some cases, such individuals can enhance the acceptance of products and recommendations.

Mr. Fleischman said the RELs can be an effective sales force to promote the What Works Clearinghouse. The RELs bring to bear their experience understanding context and relationships. For example, when REL Northwest at Education Northwest held a webinar for the Bureau of Indian Education Schools about preventing dropouts, it made sure to involve a key person in that community who was recognized in the field—but not necessarily a research expert.

Bear in mind that the What Works Clearinghouse has competition, said Mr. Fleischman. Educators typically get their research information from vendors who supply professional development along with direct support. Another source is the Association for Supervision and Curriculum Development ASCD Education Leadership series, which provides research summaries. IES should learn from the competition. Finally, said Mr. Fleischman, more work is still needed to ensure that all the relevant offices and programs across ED—not just in IES—are working together.

Discussion

Dr. McLeod appreciated that IES is attempting to improve communication, because there has traditionally been a disconnect between Federal and district efforts. She emphasized the importance of reaching policymakers not just at the national level but at the State and school district levels as well, particularly because vendors are already playing a significant role in providing them information. Dr. McLeod said it would be interesting for the Board to hear from a panel of users—whom she believes are typically central office staff, not teachers or principals—about their perceptions of the What Works Clearinghouse. She pointed out that parents need information as well, and the What Works Clearinghouse should consider how to reach them. Finally, she said she was excited about the prospect of engaging the RELs as a way to reach districts and States.

Dr. Underwood emphasized the need to better understand where decision-makers currently get their information so that we do not make false assumptions. Further, the What Works Clearinghouse should consider how it gets feedback and in particular whether users get some sort of reward for providing feedback. It may be appropriate to publicize current tracking measures, which could spur others to use the What Works Clearinghouse more.

Because the research agenda for the What Works Clearinghouse is potentially vast, said Dr. Yoshikawa, consideration should be given to the range of interventions for which the What Works Clearinghouse information could be used. The first step may be to focus on the types of interventions that you most want to know about. Because some of the interventions are targeted at a network level, testing and evaluating those interventions could set up a research agenda. Dr. Yoshikawa also recommended a State- or district-level saturation strategy. For continuous improvement efforts, Dr. Yoshikawa suggested a strategy broader than assessing a single piece of information; for example, the What Works Clearinghouse could evaluate the effects of sustained engagement efforts around a broad question over the course of a year. Finally, he called for more clarity around the measures for evaluation, particularly whether they assess immediate or long-term results.

Dr. Hedges suggested that IES take a closer look at how the intended audience uses and interprets the What Works Clearinghouse products. Dr. Singer said publishers routinely require authors to identify the intended audience, the branding needed to reach the intended audience, the means of dissemination, and the ongoing promotion of products before producing the product, while it appears that the What Works Clearinghouse was built without having detailed enough information about the intended audience. She pointed out that IES needs an overall media strategy, including one for the What Works Clearinghouse.

Asked to comment on the What Works Clearinghouse's media presence, Ms. Sparks of Education Week, confirmed that it is not easy to get information about what products have been updated or revised. She asked whether districts who are interested in becoming reviewers or acquiring training on single-case studies could do so through the What Works Clearinghouse. Dr. Neild responded that connecting individuals from districts and States to reviewer training opportunities was not a consideration until the What Works Clearinghouse received some requests about it.

Dr. Singer suggested that the What Works Clearinghouse reach out to specialized media outlets. She added that it should take advantage of social media outlets—but cautioned that if it is not able to keep its social media presence up to date, it should shut down that aspect, because an out-of-date profile is worse than none. Dr. Singer offered a number of specific suggestions for improving the reach of the What Works Clearinghouse that she hoped would spur IES to brainstorm about other potential mechanisms:

• Partner with organizations that reach the target audiences, such as the Education Writers Association (which sponsors seminars to help its members better understand research).
• Look for innovative ways to communicate, such as briefings.
• Identify a "public face"—a champion or spokesperson—for the What Works Clearinghouse.
• Build a community around the What Works Clearinghouse, perhaps with assistance from the RELs.
• Promote the journal club approach used among medical practitioners (a self-guided, group discussion of published research).
• Consider providing grants to local school systems to promote the use of the What Works Clearinghouse.

Dr. Long echoed the importance of enhancing two-way communication, not just dissemination. Researchers in general need to know how to make their research more effective—that is, how to make it useful to decision-makers. Dr. Long suggested that IES engage with some of the largest school districts in the country and ask for feedback about how those districts—from the policymakers at the top to the teachers in the classroom—use the What Works Clearinghouse products. The results could boost credibility, which could then be used to promote the What Works Clearinghouse further. Regarding competitors, such as vendors, Dr. Long stressed the importance of communicating what does not work so that practitioners and decision-makers do not waste time and resources on ineffective products.

Dr. Granger suggested that the What Works Clearinghouse would benefit from better information on its users, the decisions those users need to make in their work, the tools they need to improve their decisions, and their current sources of information about research. He suggested that research on using evidence to make decisions should be part of the overall IES mission—not just a function of the What Works Clearinghouse.

Also, Dr. Granger noted, identifying what works may not be enough. Other considerations, such as the cost and ease of implementation of interventions are also important. Potential users want to know, for example, whether effectiveness was determined by research conducted in settings like theirs. Dr. Granger pointed out that asking researchers to include such implementation, cost, and contextual information in evaluation reports might be an important way to improve the relevance of the Clearinghouse's information.

Discussion turned to the strengths and weaknesses of the practice guides. Dr. Neild said she particularly liked Teaching Elementary School Students To Be Effective Writers because the recommendations are clear, focused, and useful. The guide also give specific examples that teachers will understand and that may whet their appetites to learn more. Dr. Neild said there is consideration of developing products that elaborate on the examples in the guides. Dr. Ball said she preferred the tone of the practice guide Developing Effective Fractions Instruction for Kindergarten Through 8th Grade, because it piques readers' interests with ideas that fly in the face of typical practice. The writing guide, in contrast, lacks excitement, and the quality of evidence to support it is not high. She suggested that further discussion may be needed on the relationship of research to practice, particularly if the recommendations of a practice guide are not limited to the highest levels of evidence.

Dr. Loeb supported the idea that the What Works Clearinghouse should provide more guidance on how to interpret the evidence for effective interventions, recognizing that the research is not always clear. Dr. Neild responded that, in the spirit of transparency, there is consideration of posting more information online about how review panels reach their conclusions. She is interested in pursuing mechanisms that describe the approaches a user can take to apply the evidence to decision-making.

Dr. Loeb noted that the What Works Clearinghouse focuses on internal and not external validity. She suggested providing more details about the outcome measures and other methods of evaluation used to assess effectiveness. Dr. Neild agreed with the assessment and requested more input from Board members on how to gauge external validity and other important issues like impact on subgroups.

Dr. Chard noted that some States and districts have customized the research into approaches they can use to put theory into action, which IES should explore. He added that the What Works Clearinghouse should not overlook the possibility of partnering with the vendors of educational materials, who provide billions of dollars' worth of educational tools every year. Also, a better understanding is needed of what is happening in teacher education that drives users to the What Works Clearinghouse for information.

Dr. Gamoran said that when the What Works Clearinghouse began, there was little valid research available. Now, research is trying to catch up with the need for evidence that is useful to identify what works, for whom, and in what circumstances. Efforts are moving forward with iterative research and implementation research as well as in large-scale effectiveness studies, which can assess the heterogeneity of treatment effects. Dr. Gamoran suggested that the What Works Clearinghouse should link more closely with the Doing What Works website. Clearly, there is a need to link advice about implementation with information about programs deemed effective.

Toward that end, Dr. Singer suggested considering the user's perspective, for example, by asking, "What would I have to change to implement these recommendations?" The answer may vary depending on the user's position; thus, it may be appropriate to consider offering the same materials in different formats for different audiences. Dr. Singer pointed out that the practice guide Improving Mathematical Problem Solving in Grades 4 Through 8 appears to be written by a researcher interested in theorems and proofs; most users are more likely to be interested in learning about alternative ways of implementing the approaches. Much of the supporting information could be moved to an appendix, she said. In addition, the practice guides should be written in an engaging, active voice and include, for example, links to short videos showing a teacher applying the intervention. Dr. Singer suggested using focus groups, which could provide valuable insight that would improve the What Works Clearinghouse materials.

Dr. Gutierrez echoed Dr. Singer's concept, suggesting that products answer the question, "What do I need to carry out these recommendations?" Implementing the recommended practices may require expertise that the user lacks. Insights from practitioners who have implemented the practices is needed to ensure that the products are useful.

Dr. Long suggested the Board discuss dissemination in more depth at a future meeting. Dr. Neild thanked the Board members for their insights and hoped to return to the Board in the future with improvements to the What Works Clearinghouse as a result of the discussion.

Lunch

Participants adjourned for lunch at noon, and the meeting resumed at 1 p.m.

Design-Based Implementation Research (DBIR)
Barbara Means, Ph.D., Director, Center for Technology in Learning, SRI International
Thomas Smith, Ph.D., Director and Principal Investigator, National Center on Scaling Up Effective Schools and Associate Professor of Public Policy and Education, Department of Leadership, Policy, and Organizations, Vanderbilt University

Dr. Means pointed out two very different strategies to conducting research to improve education. One method is to identify and disseminate interventions that work—the original intent of the What Works Clearinghouse. Another is DBIR, which embeds research into the process of policymaking and practice. DBIR is not a specific methodology but rather a perspective that draws on methodologies such as implementation, design, and improvement research. Dr. Means described the underlying principles of DBIR:

• Focus on a persistent problem of practice from the point of view of multiple stakeholders. The focus of research is not defined solely by the investigators or the institution; rather, the goal of the research is negotiated among those involved.
• Commit to an iterative, collaborative design. Collaboration aims to combine insights from prior research and practice to create a design that is meaningful and useful to all. Outcomes of collaboration are more likely to be implemented.
• Pay attention to both classroom learning and improvement.
• Develop a systemic capacity for sustaining improvement. DBIR seeks to increase the capacity for systems to evaluate, research, and implement interventions across various real-world settings.

Dr. Smith said the National Center on Scaling Up Effective Schools, funded by IES, began its efforts focusing on capacity-building. Researchers aimed to identify achievement gaps, then work with school districts to build the capacity to identify effective practices (including existing efforts), adapt them to their own schools, and implement interventions in other schools to close the gaps. The Center's efforts to work within school districts brought to light the challenges of comprehensive school reform that districts face: lack of teacher buy-in, mismatch of the intervention and the context, and the lack of alignment between the goals of researchers and the needs of schools.

Dr. Smith said the first step was to identify schools that were beating the odds among low-performing subgroups. The Center partnered with districts in Florida and Texas because those States had the strongest data systems. The initial goal was to learn from schools at the higher end of the distribution for value-added gains in student performance and transfer their practices to schools at the lower end. Having completed the intensive process of in-depth, on-site review of schools and combining the findings with the results of survey data, the Center is beginning Year 3 of its effort, which looks like DBIR.

Currently, efforts are underway to facilitate partnerships between districts and the design team. The design team will share what it has learned from the first 2 years and what evidence suggests is effective, the schools will discuss their needs, and the partners will work together to design appropriate implementation strategies for interventions. Dr. Smith said there was no clear consensus on whether to focus on a single intervention or multiple interventions at the outset, and so the decision has been postponed until the initial design is proposed. However, there will be pilot implementations, and the results will feed into future iterations, eventually resulting in full implementation that has been tailored to meet the district's unique circumstances. Eventually, the Center will withdraw and, ideally, the district will have the capacity to continue the approach.

Collaboration is at the core of the approach, and each partner has equal weight in the partnership. The Center focuses on the essential components of effective schools, and from this perspective, seeks to understand not just what successful schools are doing but how they are doing it. Dr. Smith pointed out that the differences in performance across schools was not related to, for example, the lack of a rigorous, aligned curriculum (because all the districts had them and teachers followed them closely) but rather the systemic approach that individual schools used to address the issue.

To illustrate, Dr. Smith said that those schools within the Broward County Public Schools system in Florida that achieved high performance focused on personalization of social and academic learning by creating systematic structures that promote strong relationships between students and adults, an atmosphere of caring and trust, instructional activities that draw on students' experience and interest, and strong links with parents. In the design phase, instead of generalizing the findings from the literature, the partners sought examples of how the high-performing schools implemented these systems—as well as what the low-performing schools failed to do. Dr. Smith pointed out that school districts are more likely to buy in to the approach when they see the success of other schools within the same district.

The Center has two agendas: building team capacity for innovation, design, transfer, implementation, and scale-up of interventions, and deepening understanding of the practices that distinguish effective schools. In Phase 3 of the Center's efforts, the teams will assess the implementation of the designs, build capacity to implement the design innovations in more schools, monitor the outcomes, and develop tools to help others transfer effective practices across their districts. Dr. Smith emphasized that once a design is successful, the next step is to implement it in other contexts to identify the areas of overlap between effective practices and various contexts.

Discussion

A New IES Research Program
John Q. Easton, Ph.D., IES Director

Dr. Easton said that IES posted a proposal online for comment about a new IES research topic in continuous improvement research in education. It is strongly influenced by the philosophies behind DBIR, the IHI, and The Checklist Manifesto, as well as Dr. Easton's own background. He said knowing what works plays an important role in school improvement, but it is not enough. We also need to build the capacity to implement what works through measurement and adaptation. Dr. Easton posed three questions to the Board to elucidate their perceptions of the role of IES within the explosion of activity around implementation.

• How should IES refine the conceptual framework for the proposed topic so that potential grant applicants will have a clear understanding of the purpose of the topic?
• Are the three continuous improvement strategies described as examples (creating a safe, orderly, and supportive learning climate for students from preschool through high school; improving students' transition to high school; and increasing access to college and postsecondary training) sufficient?
• What standards of evidence should be built into the topic? What types of studies would provide adequate evidence of promise, efficacy, or effectiveness?

Discussion

Dr. Long asked for more elaboration on the perceived need to add such a resource topic. Dr. Easton responded that a funding partnership may be the best way to support the development of new models of continuous improvement that address systemic problems in practice and to produce proof that these models can solve problems.

In response to Dr. Chard, Dr. Easton said the proposal included the three continuous improvement strategies as a means of narrowing the field and establishing some broad criteria for evaluating applications. He added that the three strategies address pressing, almost intractable issues.

Dr. Chard wondered whether the proposal seeks to recast the IES goal structure; Dr. Easton responded that IES has been thinking about the design and development side of the goal structure and at what point the field needs sufficient evidence of impact.

Dr. Gamoran suggested that this proposal could be the first of IES's programs to implement a process for renewed or additional funding on the basis of promising initial results. Dr. McCardle said NICHD grantees receive 5-year awards that are eligible for renewal, but the proposal review process is the same (e.g., it includes peer review). In such cases, previous performance is considered. Dr. Gamoran saw the proposal as an opportunity to go beyond encouraging researchers to move their research from Goal 2 into Goal 3 by making that an explicit expectation.

Dr. Loeb said the duration of the grant should be considered carefully. The collaborations needed to pursue this research take time to build and involve some risk; in some cases, the topic would only be useful for entities that already have collaborations in place.

To narrow the range of focus of potential applications, Dr. Yoshikawa suggested the proposal provide some examples of specific research questions or guidelines appropriate for building a new field of research—for example, the researchers could look at the relationship of short-term and long-term changes and carefully define the measures used to assess such changes. The proposal should ask applicants to relate their research to the current IES goal structure in an explicit way. Dr. Hedges said that it may be necessary for either the applicant or IES to define the measures used to demonstrate improvement.

Dr. Singer felt the proposal does not clearly identify what IES seeks to encourage. If it aims to affect what happens in schools, it seems too narrow, and many researchers would decline to apply. It may be that the proposal seeks to foster the growth of DBIR and similar approaches or that it aims to encourage more collaboration. Dr. Easton clarified that IES already established another program to fund collaborations; he thought the proposal should foster the growth of DIBR and related approaches. In that case, Dr. Ball recommended, the IES should look to NSF for guidance. She added that it is not clear how DBIR builds a broad knowledge base.

Dr. Ball also said the three strategies identified are problematic, because they compartmentalize research by grade bands. She said a lot of evidence shows the relationship between elementary school and postsecondary performance, but most proposals default to preschool, secondary, and postsecondary research. The proposal lacks discussion about improving the workforce and teacher quality, she noted.

Dr. McLeod said that districts are grappling with the implementation of the Common Core Standards, which could be a topic for research within this proposal. She asked what the collaborations envisioned by this proposal would look like. Dr. Easton said many States and districts already have research partnerships in place, some of which receive IES funds for evaluation, but he would like them to focus more on problem-solving.

Dr. Granger suggested that thinking through how to conduct and learn from such DBIR efforts in the next 3 years was a worthy goal.

Dr. Long felt IES should develop some guidelines, such as the definition of an existing collaboration. Dr. Ferrini-Mundy said the NSF has been investing for a long time in partnerships. She said there seems to be a new push for real-time evidence and continuous improvement that could be introduced as a strategic approach in the areas identified by IES in the proposal. It may be necessary to support capacity-building to help schools learn how to implement this approach. Dr. Ferrini-Mundy said the proposed dollar amount seems low compared with previous partnerships, especially considering that the proposal calls for a large, systemic approach.

Dr. Gamoran noted that many partnerships have been established over the past 10 years. The IES proposal offers a model to guide those partnerships on how to proceed: identify a goal and engage one of the continuous improvement strategies outlined in the proposal to address that goal.

Dr. Loeb said that within the continuous improvement approach, two components seem to stand out: developing real-time evidence and building a framework for reflective practice at the district level. Collaborative partnerships can offer opportunities to experiment and ask questions in a low-cost, relatively easy manner, and such efforts often produce better, richer findings, said Dr. Loeb. However, she noted, the proposal initially seems to focus first on building a reflective practice but then shifts away from that concept. Dr. Easton noted that a participant in a project of the National Center on Scaling Up Effective Schools said that by working within a collaborative partnership, he and his colleagues built up a resistance to the pressure to buy the next so-called silver bullet, which seems to echo the benefits of reflective practice.

Dr. Bryk pointed out that to accelerate learning, a network of stakeholders should be aligned around a specific set of problems. Selecting the target problems, however, raises tensions and can lead to political dilemmas. These new approaches require the participants to learn as they go; as such, they do not have clear strategic goals and timelines. In real life, said Dr. Bryk, you want to be thoughtful about your first steps and have an overall goal in mind, but you want your process to evolve as you progress.

Dr. Long concluded the conversation by noting that additional comments should be sent to Dr. Easton. Dr. Easton said that IES hopes to have a well-written, clear request for applications posted by the end of February 2013.

Policy and Legislative Update
Sue Betka, Deputy Director for Administration and Policy, IES

• How does IES compare to selected research organizations (to be determined) on cost, relevance, and other outcome measures?
• What are the strengths and weaknesses of the four component IES centers?
• How have IES governance structure and standards for quality research supported or hindered the Agency's ability to fulfill its mission?
• To what extent does IES coordinate its activities internally and with other relevant entities, including ED's Comprehensive Centers programs?

Ms. Betka explained that ED has received very little specific guidance about sequestration (i.e., the requirement to cut the Federal budget across the board if Congress does not pass the Budget Control Act to reduce the deficit). The White House released a sequester report in late September; it could not be released until the 2013 appropriations were determined. The 2013 appropriation, however, is a continuing resolution that expires on March 27, 2013; it allows ED and almost all other agencies to operate at the FY 2012 level and includes a 0.6 percent increase. Using historical spending rates as a guide, Ms. Betka estimated that ED would spend about 31 percent of its total appropriation between now and March 27, 2013.

The White House determined that the ED sequester proportion would be 8.2 percent on the basis of a full year at the FY 2012 level. President Obama continues to encourage Congress to prevent the sequester. Ms. Betka said that no one really believed the sequestration would happen, but if there is no deficit reduction package by January 2, 2013, it will go into effect. She did not know whether the sequestration would apply to a half-year appropriation. Ms. Betka pointed out that the calculations are very complicated, and because there is still time to avoid sequestration, there seems to be little interest in working out the details.

The FY 2012 budget for IES was about $594 million, so the sequestration amount (8.2 percent) would be about $49 million. If sequestration comes to pass, it would be applied at the program, project, and activity levels.

Discussion

Board Leadership for 2013: Roundtable Discussion

Closing Remarks and Adjournment
John Q. Easton, Ph.D., IES Director, and Bridget Terry Long, Ph.D., NBES Chair

Dr. Long concluded that she was pleased with the meeting and with the way the IES staff is taking the Board's comments to heart. She hoped the Board would continue to build on the momentum it has created. Dr. Easton thanked the Board members for an exciting and worthwhile discussion. He reiterated his request that Board members send him additional comments on the new IES research topic. Dr. Long thanked Ms. Pelaez and Dr. Herk for their hard work. She also thanked the staff of AFYA, Inc., the meeting contractor that handled the logistics of this meeting. Dr. Long adjourned the meeting at 4:02 p.m.

View, download, and print the full meeting minutes as a PDF file (7.8 MB)

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.