Skip Navigation
A gavel National Board for Education Sciences Members | Priorities | Reports | Agendas | Minutes | Resolutions| Briefing Materials
National Board for Education Sciences
June 8, 2015 Minutes of Meeting

Location
Institute of Education Sciences (IES) Board Room
80 F Street, NW
Washington, DC 20001

Participants
National Board for Education Sciences (NBES) Members Present
David J. Chard, Ph.D. (Chair)
Susanna Loeb, Ph.D. (Vice Chair)
Adam Gamoran, Ph.D.
Larry V. Hedges, Ph.D.
Darryl J. Ford, Ph.D.
Deborah Phillips, Ph.D.
Judith Singer, Ph.D.
Robert A. Underwood, Ed.D.
Hirokazu Yoshikawa, Ph.D.

NBES Members Absent
Anthony S. Bryk, Ed.D.
Michael Feuer, Ph.D.
Kris D. Gutierrez, Ph.D.
Bridget Terry Long, Ph.D.
Robert Teranishi, Ph.D.

Ex Officio Members Present
Sue Betka, Acting Director, IES, U.S. Department of Education (ED)
Thomas Brock, Ph.D., Commissioner, National Center for Education Research (NCER)
Peggy Carr, Ph.D., Acting Commissioner, National Center for Education Statistics (NCES)
Joan Ferrini-Mundy, Ph.D., Assistant Director, National Science Foundation, Directorate for Education and Human Resources
Joan McLaughlin, Ph.D., Commissioner, National Center for Special Education Research (NCSER)
Ruth Curran Neild, Ph.D., Commissioner, National Center for Education Evaluation and Regional Assistance (NCEE)
Chuck Pierret, Bureau of Labor Statistics

Invited Presenters
Nikola Filby, Ph.D., Director, Regional Educational Laboratory (REL) West, WestEd
Neal Finkelstein, Ph.D., Associate Director, REL West, WestEd
Barbara Foorman, Ph.D., Francis Eppes Professor of Education and Director, REL Southeast, Florida State University
Tom Kane, Ph.D., Walter H. Gale Professor of Education and Economics, Harvard University
James Kemple, Ed.D., The Research Alliance for New York City Schools, New York University
George F. Koob, Ph.D., Director, National Institute on Alcohol Abuse and Alcoholism (NIAAA), National Institutes on Health (NIH)
David K. Mineta, Deputy Director, Office of Demand Reduction, Office of National Drug Control Policy (ONDP)
Susan Weiss, Ph.D., Director, Division of Extramural Research, National Institute on Drug Abuse (NIDA), NIH

NBES Staff
Kenann McKenzie-Thompson, Ph.D., NBES Executive Director
Ellie Pelaez, Management & Program Analyst, Office of the Director, IES, Designated Federal Official

IES Staff
Chris Chapman, Sample Surveys Division, NCES

Call to Order
David Chard, Ph.D., NBES Chair

Dr. Chard called the meeting to order at approximately 9:00 a.m., and Ellie Pelaez called the roll.

Dr. Chard welcomed a new member, Deborah Phillips, Ph.D. He explained that Margaret R. (Peggy) McLeod, Ed.D., resigned from the Board. Robert Teranishi, Ph.D., was newly appointed.

Dr. Chard introduced Kenann McKenzie-Thompson, Ph.D., the new NBES executive director, who thanked the Board members, IES commissioners, and staff for ushering her in to her new role. She encouraged Board members and others to contact her with feedback and suggestions. Dr. Chard said he introduced Dr. McKenzie-Thompson to key legislators on Capitol Hill as the go-to person for information about IES and education research. Dr. McKenzie-Thompson also helped complete the NBES annual report for 2014, which members will be asked to review by e-mail.

Dr. Chard said the agenda for the meeting reflects topics of interest suggested by Board members, and he encouraged Board members to continue identifying issues for discussion. Dr. Chard briefly summarized the agenda.

Opening Remarks
Sue Betka, Acting IES Director

Ms. Betka welcomed the Board members. She said there is bipartisan, bicameral support for reauthorization of the Education Sciences Reform Act (ESRA). The U.S. House of Representatives has agreed to take up the matter, but it is on hold in the Senate. Ms. Betka and others expect the Senate will move forward on ESRA following deliberations on the Elementary and Secondary Education Act (ESEA). She expressed some optimism that ESRA would be reauthorized by the current Congress.

IES Commissioners' Reports

National Center for Education Research and National Center for Special Education Research
Thomas Brock, Ph.D., NCER Commissioner, and Joan McLaughlin, Ph.D., NCSER Commissioner

Dr. Brock noted that NCER received more than 700 applications for research and training grants for fiscal year (FY) 2015 and made about 120 awards. He said NCER had more funding for awards this year because many older grants expired, freeing up funds, and there were many highly rated applications (including many resubmissions). However, NCER does not anticipate an increase in funding for FY 2016, and most of that year's funding will go to continuing grant awards rather than new awards.

Because NCER and NCSER project that available funds will be low for FY 2016, they are taking steps to limit the number of applications for the FY 2016 competition. For example, no applications will be solicited for Goal 2, Development and Innovation, in the Education Research Grants program. In addition, maximum funding for awards will be reduced slightly. In some competitions, the number of awards will be capped. Competitions for the following programs will not be held in FY 2016, although competitions are expected to resume in future years:

  • Statistical and Research Methodology (except the Early Career awards)
  • Postdoctoral Training
  • Continuous Improvement Research in Education

Dr. Brock said NCER continues to pursue some new initiatives, despite limited funding. On the basis of stakeholder input, NCER and NCSER established the Pathways to Education Science Training Program to increase diversity in education research. The program is specifically intended to increase research training opportunities for people of color, people with disabilities, veterans, and other underserved populations. Also, NCER is launching a new program called Research Networks Focused on Critical Problems of Policy and Practice. The first two networks will address the transition from pre-K to early elementary school and strategies to improve college completion.

Dr. McLaughlin said NCSER was pleased to fund two competitions in FY 2015—one for special education research grants and one for training. Training remains an important focus for NCSER, as many faculty members in the field are retiring. Dr. McLaughlin said training awards will focus on early-career investigators, postdoctoral research, and methods training in single case design.

Dr. McLaughlin added that NCER's and NCSER's research centers are using social media to share information on funding opportunities and research findings. She and Dr. Brock have started a blog to spread the word about NCER and NCSER activities, said Dr. McLaughlin.

Discussion

Adam Gamoran, Ph.D., wondered whether NCER had the capacity to withhold newly available funds to support grants next year. Ms. Betka said IES can carry over some funds. Dr. Brock noted that, given the tight funding environment in 2013 and 2014, NCER did not want to hold back any funds that could be used for applications that received excellent or outstanding ratings this year. Dr. Gamoran hoped the Board would make a formal statement to the Congress to increase funding for IES research.

Darryl J. Ford, Ph.D., asked for clarification about the kind of support NCER would provide for the Pathways to Education program and the Research and Development (R&D) Center on Virtual Learning. Dr. Brock replied that NCER hopes to give up to four new awards for the Pathways to Education program. For the R&D Center on Virtual Learning, NCER accepted applications in FY 2014, but none ranked high enough to merit funding. The competition will reopen in FY 2016. The R&D Center on Virtual Learning will fund rapid experimentation in education technology to identify and improve widely used platforms. It will also investigate how to improve teaching and student outcomes by harnessing the "big data" gathered through education technologies, such as cognitive tutors and other online tools.

National Center for Education Statistics
Chris Chapman (for Peggy Carr, Ph.D., Acting NCES Commissioner)

Mr. Chapman said the National Institute of Statistical Sciences will conduct an overview of NCES to map out the Center's future. It will determine how the Center can address policy needs as efficiently as possible over the next 10 years.

The new director of the National Assessment Governing Board, William Bushaw, will take over in July. Mr. Chapman said the National Assessment of Educational Progress (NAEP) faces budget issues and so may have to make difficult choices about the assessment schedule. Currently, making the transition from paper to tablet assessments is a priority, and it is important to study the transition carefully. Mr. Chapman also said the Achieve NAEP mapping report used a less rigorous approach than an NCES study that will be released in June.

NCES launched a new blog in April to communicate with the field more regularly. It will also begin using Twitter for outreach about topics beyond NAEP.

The Condition of Education 2015 report was released to Congress in an updated format that provides standardized indicators to help representatives focus on recent changes and key statistics. The report also includes a new "at-a-glance" section to brief policymakers. NCES created online videos to highlight key findings, which garnered some media attention.

National Center for Education Evaluation and Regional Assistance
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild said the What Works Clearinghouse (WWC) recently produced a new practice guide on algebra instruction. The practice guides are often the impetus for outreach by the RELs. NCEE is producing a video describing categories and levels of evidence (e.g., what constitutes "minimal evidence") as part of its ongoing efforts to better explain its processes and needs. A practice guide on writing at the secondary level is underway.

REL Southwest developed a toolkit that builds on the practice guide for English-language learners (ELLs). This toolkit, designed for professional learning communities, includes eight 60-minute sessions, a facilitator's guide, homework, and videos showing implementation in real-world situations. These products represent efforts to improve engagement with practitioners beyond the guides and 1-day workshops.

NCEE is working on a set of infographics to better articulate what the WWC does and how it works. Dr. Neild said it is important to clarify how the WWC makes decisions and prioritizes reports.

Through its website and social media, the WWC is addressing standards and studies that meet them. Last fall, NCEE hosted a webinar on what the WWC seeks in randomized, controlled trials (RCTs) and quasi-experimental design studies. Over the past year, NCEE has been working hard to retool the WWC's underlying database to make it easier for new users to locate WWC reviews and findings. It is redoing the "Find What Works" interface. Dr. Neild hoped to provide a demonstration by the end of October and to launch an updated site in December.

IES' public access policy requires grantees to submit their published, peer-reviewed research to the Education Resources Information Center (ERIC) to make taxpayer-funded work broadly available. NCEE is working on videos and other mechanisms to help grantees navigate the new online submission system. Dr. Neild said ERIC has become much more searchable and includes useful tags, indicating, for example, whether a study met WWC criteria.

In general, NCEE has been very productive this year and has a lot of products in the pipeline. Moreover, the products have been well received and are being promoted and disseminated through webinars and social media.

Discussion

Dr. Gamoran asked what NCEE is doing to ensure that the research community, journalists, and others are aware of the new functions of ERIC and whether the WWC includes a link to ERIC. Dr. Neild responded that NCEE is trying to ensure that investments are interrelated. For example, the references in all REL reports and WWC products include links to the resource if it is indexed in ERIC. Dr. Neild felt more could be done to let all of NCEE's constituencies know about ERIC, which has its own news flash communication (mainly aimed at librarians, the top users of ERIC), and the abstracts will have the publications listed with a link to ERIC.

Hirokazu Yoshikawa, Ph.D., praised the continued work on dissemination. He asked how practice guide topics were selected. Dr. Neild said topics come from feedback from practitioners, and the WWC often seeks out topics for which there is a need and enough evidence to provide focused, feasible, clear recommendations.

Dr. Yoshikawa also asked whether the WWC would develop policy guides for state or district administrators, who can be overwhelmed by information. Over the years, the WWC has tried to create practices guides for audiences other than practitioners but has not yet developed a model with which it is comfortable, Dr. Neild continued. However, the WWC continues to work on this issue and, at present, is working on a guide for policymakers about pre-K education.

Dr. Chard asked if there is a link between the R&D Center on Knowledge Utilization and the WWC. Dr. Brock said the new Center has not been announced, but he believes it will contribute to improving WWC products.

Dr. Chard asked whether NCEE has approached nongovernment organizations, such as Edutopia, about linking to the WWC from their websites. Dr. Neild said Edutopia sent a representative to a WWC webinar, but she did not know of direct conversations with the organization. The WWC does have a widget (an icon for a link) that can be included on any website. Dr. Chard noted that teachers seem to use Pinterest a lot to talk with each other.

Judith Singer, Ph.D., praised the progress of the WWC. She asked how the WWC is gathering feedback about how its materials are used and their impact. Dr. Neild said the WWC can track baseline metrics, but impact is difficult to measure. She noted that the RELs are one example of groups that disseminate WWC materials to other audiences.

Larry V. Hedges, Ph.D., asked whether the Bridgespan Group's recent report on research clearinghouses, including the WWC, was helpful. Dr. Neild said the report was helpful. It was nuanced and seemed to include a lot of the WWC's suggestions and ideas.

Dr. Chard said he and Susanna Loeb, Ph.D., have been discussing whether the Board should create subgroups who would talk with IES commissioners on an ongoing basis, so that the Board is engaged on a continuous basis instead of just hearing periodic reports.

Dr. Gamoran moved that the Board express to Congress in writing its support for increasing IES' budget. Dr. Hedges seconded the motion, and the Board unanimously approved. Now is the time to increase investment in research, said Dr. Gamoran. Without such investment, IES is unable to fund excellent and outstanding proposals now and in the future, he continued.

Motion Approved

On behalf of NBES, Dr. Chard and Dr. Loeb will write to the relevant Congressional committees to express support for the Administration's request to increase IES' research, development, and dissemination budget.

Improving Education: The Research Road Ahead
Opening Remarks: Susanna Loeb, Ph.D., NBES Vice Chair Presenters: Tom Kane, Ph.D., Walter H. Gale Professor of Education and Economics, Harvard University
James Kemple, Ed.D., The Research Alliance for New York City Schools, New York University

Dr. Loeb encouraged the Board to think about how research can be used in practice and what areas could use more support.

Putting the "Prove" Into Educational Improvement
Tom Kane, Ph.D., Walter H. Gale Professor of Education and Economics, Harvard University

In education—as in other fields—most new ideas will fail. Therefore, advancing the field depends on identifying the subset of what works. Currently, Dr. Kane said, "We're terrible at that." The field of education research is organized to gain consensus about what works among researchers who are funded at the federal level. At the local and state level, there is no such consensus, and progress will require more than just better translation of research findings.

The results of very small-scale trials are likely to be misleading about the magnitude of effect. By middle school, the average student gains less than 0.3 standard deviations (SDs) from one year to the next. A Boston study assessing the impact of teaching experience showed a student gain of .08 SDs with a few years of experience, which could be considered a large effect. However, detecting small but significant effects is difficult. School district leaders and principals trying to track achievement gains among a small number of classrooms will see a lot of false-positive results.

A typical evaluation takes 6 years and $12 million. For example, Race to the Top began in 2009, and there is still no consensus on whether the policies are paying off. Many schools are implementing education software, but there is no consensus on what works for whom or whether some kids are hurt by the use of software. Across the board, consensus among researchers is elusive and successful translation to the field is rare. Furthermore, almost all variation in schools happens at the teacher level, so fidelity of implementation in classroom is vital but is usually invisible.

However, Dr. Kane pointed to several assets, such as large investments in state longitudinal databases and state testing. He noted that local leaders will find their own data compelling enough to change their policies. While implementation is usually invisible, education software, video technology, and student surveys can make some aspects more visible. In addition, the Center for Education Policy Research is building a national network of research analysts.

Given these assets, Dr. Kane offered some elements of a potential solution.

  1. Start with personalized learning using educational software. Provide school district leaders with tools to evaluate their own data to determine what works. Personalized learning offers a low-cost method for providing short-term feedback on implementation on a small scale.
  2. Identify a network of districts or charter management organizations with common traits, such as the following:
    • Common interim assessments
    • Personalized learning projects underway
    • Compatible student information system
  3. Invite teachers and software providers to describe weekly implementation targets.
  4. Use log files, student surveys, and video to document implementation.
  5. Provide weekly reports by classroom so that bottlenecks are identified while there is still time to address them.
  6. Automate matching to find comparison groups. Automatically generate reports comparing treatment and comparison classrooms.
  7. Revise implementation targets.
  8. Convene the network to make sense of the findings.

This approach could be refined by adding teacher surveys, measuring the impacts on specific standards, and providing professional development interventions.

The goal should be to create a model of clinical trials that does not just mimic the pharmaceutical industry but is adapted to the specific needs of the U.S. education system. Given the distributed nature of decisionmaking, the current model is "doomed," said Dr. Kane, because it takes too long to reach consensus, and research is too disconnected from decisionmakers in the field. He proposed a model in which decisionmakers can apply the research tools to their own data, quickly identify comparison groups, and measure changes. The method is not as clean as a federally-funded RCT, but it may be quicker to identify and drive change that helps students.

Principles to Guide Research
James Kemple, Ed.D., The Research Alliance for New York City Schools, New York University

Dr. Kemple outlined four principles to guide research that IES should support:

  • Preserve scientific rigor.
  • Prioritize research relevant to policymakers and practitioners.
  • Promote the use of high-quality evidence.
  • Prepare for the future.

The focus on relevance can lead to the misperception that rigor and relevance are competing against each other and that the lack of attention to relevance is a research methods problem. However, all four of these principles build on each other. They are consistent with IES initiatives and support IES' direction.

Preserving the commitment to scientific rigor means continued funding of research to establish strong causal connections between specific policies and high-stakes student outcomes, such as school readiness, engagement, achievement in math and science, and college and career readiness. The work of IES has had the greatest impact on this research.

Leadership requires relentless communication and teaching on how evidence can make a difference, why methods and rigor matter, and how using the wrong methods will get to the wrong answer. Alternatives to RCTs should be measured against things that could confound the causal inference between relationships and outcomes. IES can do more to educate the field on feasibility and the ethics of rigorous research design, but that requires interaction with the field and also with policymakers. So-called natural RCTs and rapid-cycle trials may hold some promise, but caution is warranted.

The preservation of rigor forces hard decisions about what not to do. The field of education is good at identifying problems and proposing possible solutions, but there is a continuous struggle to determine whether and how much to invest in research on better measures of student learning, creation of new interventions, and research that examines educational processes but stops short of assessing whether they influence student development.

Prioritizing rigorous education research that is more relevant to policymakers and practitioners involves explicitly and intentionally including them in defining research questions and participating in research. Dr. Kemple suggested starting within the walls of ED by improving collaboration with other offices—such as the Office of Innovation and Improvement; the Office of Elementary and Secondary Education; Office of Career, Technical, and Adult Education; and the Office of Postsecondary Education—to source questions, disseminate information, and develop funding opportunities. Such collaboration will help IES prioritize research questions that come from those who are making difficult decisions about resource use.

Dr. Kemple recommended making greater use of opportunities to integrate rigorous evidence-building into innovative education improvement initiatives, as was the case for the Investing in Innovation Fund. Where federal requirements are involved, efforts should be made to get more input on research design and data collection, fostering more connection and buy-in from practitioners and policymakers.

Practitioner and policymakers involvement could be increased by requiring or incentivizing field-initiated studies to have local education agencies (LEAs), state education agencies (SEAs), or vendors endorse the research methods and questions. Researchers would have to demonstrate the value and relevance of their work. Communicating in advance would reveal whether policymakers and practitioners are willing to support rigorous research and whether researchers are willing to find methods that suit the implementation processes in schools.

Dr. Kemple also suggested devising creative ways to require or incentivize researchers to give more information back to the participants. Data collection can be burdensome, and participants have a right to expect some feedback.

The RELs could serve as the research operations function for IES studies. They can be advocates for rigorous, relevant research while also providing access to LEAs, SEAs, and other stakeholders. Researchers can use the RELs to gather questions and broker a two-way process for solicitations and proposals. The RELs can also be a conduit for communicating research. Dr. Kemple acknowledged that the RELs already perform a lot of these functions.

To ensure research is relevant, researchers should have explicit, well-crafted theories of action. Impact studies should be better equipped to learn from null effects. Current studies are not necessarily designed to shed light on why an intervention did not work. Researchers can also build their capacity to learn from heterogeneity effects by improving research design. Also, IES should consider encouraging researchers to include exploratory analyses, impact analyses, and discussion of implications. Such an approach builds on other data collected and creates a way to publish and disseminate those findings widely.

Dr. Kemple recommended that other research methods—such as design-based implementation research, network improvement communities, and continuous improvement research—could be incorporated into RCTs and not just studied as standalone activities. Along those lines, he said that setting up rapid-cycle RCTs or natural RCTs by using administrative records highlights the need to better understand such areas as null effects and heterogeneity. Without more research on those areas, the rapid-cycle studies will continue to look at single components in isolated situations. It may be risky to break solutions down into microcomponents, test them individually, and assume they can be reconstituted successfully in a larger context. That approach neglects the interaction of problems and solutions.

Promoting wider use and application of high-quality education research involves three elements:

  1. Thinking about dissemination and communication of findings as a continuous process, which requires ongoing relationships between researchers and practitioners, which, in turn, helps develop joint ownership of relevance and rigor.
  2. Diversifying products of research, so that findings are disseminated in multiple iterations, including practical and applied learning.
  3. Smarter use of technology to ensure access to high-quality research.

Preparing for the future refers to continued support for preparing the next generation of researchers to remain committed to scientific rigor, research relevancy, and the application of information to the field. Dr. Kemple said predoctoral and postdoctoral training should go beyond causal inference to better understand the importance of communicating with practitioners and policymakers and sourcing the research questions.

Finally, Dr. Kemple said the federal government plays an important role in building the evidence for education because it has the resources and capacity to do what state and local governments cannot. The quid pro quo for those federal resources is that states and localities use the evidence to make decisions and make a difference in education.

Discussion

Dr. Hedges acknowledged that new technology offers an opportunity for some novel research design, but he questioned whether quasi-experimental approaches would have more power to detect small effects than RCTs. Dr. Kane said the type of research he proposed would be a starting point that would enable practitioners to see what works using their own data. Dr. Hedges noted that, in medicine, it takes tens of thousands of patients to distinguish the effects of an intervention from those of a placebo; the results of small studies may amount to no more than "superstition." Dr. Kane responded that the current education system is ruled by superstition and lacks even small comparative studies. His proposal would allow school systems to pilot interventions in a limited number of classrooms and compare their results with others who are doing the same, broadening the evidence base.

Dr. Gamoran pointed out that both presenters advocate for enabling states and districts to run their own experiments, but past efforts by the RELs to engage SEAs demonstrate that states and districts lack the capacity to do research. Partnerships have succeeded, whether local or regional. Dr. Gamoran agreed with Dr. Kane that federal funding should be tied to evaluation components at the local level. Dr. Kane agreed that SEAs and districts do not yet have the capacity to conduct research, but they can build the capacity, especially when schools are already piloting programs and software. A mechanism to easily identify other classrooms implementing the same interventions would facilitate comparisons.

Dr. Singer said that identifying and understanding the target audience is key to a successful communication strategy, but experts have conflicting ideas about who the audiences are for IES research and whether IES is well-positioned to reach them. She said the Board should address where IES should be investing and how IES should determine its research priorities. Furthermore, the Board should discuss the connections between IES priorities and research funding.

Dr. Kane pointed out that federally funded interventions are well evaluated with federally funded RCTs, and that is appropriate, because federal policymakers make the ultimate decisions about whether to continue funding such research. However, if researchers do not think about how they will convince policymakers at the state and local levels to adopt their interventions, then the research is not necessarily changing what happens in schools. For most education decisions, said Dr. Kane, the target audience is superintendents, not other experts in education. Focusing on superintendents would speed progress over the current approach.

Dr. Yoshikawa said there are different models for thinking about how continuous improvement can occur in school systems at scale. The Board should consider what could be learned from continuous improvement research projects about attention to causal inference and what the findings mean for IES.

Dr. Brock clarified that IES' decision to temporarily pause funding for continuous improvement research does not mean IES is abandoning the topic. IES is using the pause to discuss with grantees and others whether continuous improvement research is an appropriate approach.

Follow-Up Item
Dr. Yoshikawa requested that the IES pause on funding for continuous improvement research be on the agenda for discussion at a future Board meeting.

Dr. Loeb wondered how much research is generalizable. That is, when local leaders decide what to implement, how can their findings reliably translate from one area to another? Dr. Kane said generalization is not happening now; interventions do not "trickle-down" from expert consensus to practice. He advocated for providing tools to decisionmakers to learn the implications of their decisions. If the effects of an intervention are heterogeneous, as Dr. Kemple suggested, then the only way to understand the effects at a local level is to implement it and evaluate it. Thus, it is important for LEAs and SEAs to build the capacity for decisionmakers to learn about the consequences of their decisions.

Dr. Kemple said that IES funding often aims to improve equity in education at the local level. Localities need to see that when they receive supplemental resources in addition to local investment, the results are worthwhile and, therefore, local investment should continue. Ideally, localities are also contemplating sustainability. Dr. Kemple added that generalizability is "more art than science," but evaluation to ascertain generalizability should continue.

Dr. Loeb agreed that there is a role for testing the decisions made at the federal level, but she hoped there is also a role for gathering knowledge about education that can apply more broadly. Even if a decision is made locally, it makes sense to evaluate at a higher level and share the findings—which is the theory that drives IES research. Dr. Loeb said both presenters seemed to be saying that for knowledge to be generalizable, a better understanding of heterogeneity and the reasons for it are needed, and she agreed.

Dr. Kane said it is important to find ways to get state and local governments into the business of evaluating performance, which would fundamentally change the rate of progress. He hopes to build a model that state and local policymakers would invest in to provide tools for school leaders to pool their resources around research.

Dr. Hedges pointed out that the causes of heterogeneity can be scrutable or inscrutable, and little is known about which aspects of heterogeneity are scrutable and which are inscrutable, posing a challenge for research design. He said more strategies may be needed to get auxiliary information that improves understanding of heterogeneity.

Dr. Gamoran noted that NCER is encouraging researchers to collect contextual data to facilitate comparisons across sites. He envisioned several schools assessing common interventions, then coming together for a meta-analysis that could explore heterogeneity. Dr. Hedges supported the idea of prospectively planning studies with an eye toward evaluating heterogeneity.

Dr. Singer agreed that the sources of heterogeneity are sometimes clear, but the "local exceptionalism" that pervades education is unhealthy. She said that personalizing education fosters the perception in every local school system that its students and context are unique, so results demonstrated elsewhere do not apply. Dr. Singer questioned why school systems refuse to buy in to an intervention unless they see it work in their own context. Other organizations and businesses readily adapt and implement ideas from around the world.

Dr. Kane said that whether local exceptionalism is real or perceived, providing local data is psychologically useful to overcome resistance and facilitate engagement in research. Dr. Singer advocated for pushing back against local exceptionalism with concrete examples that make the case for an intervention so that every method need not be tested in every population.

Dr. Loeb pointed out that most resource decisions are made at the local level. She said it may be beneficial to learn what kinds of tools, incentives, and communication strategies are needed to encourage local decisionmakers to implement effective interventions. Dr. Loeb asked what ED can do to encourage knowledge accumulation when local decisionmakers decide how to use resources.

Dr. Kemple said that local participants must recognize themselves in both the problem and potential solutions, especially when the researchers are of a different race/ethnicity then the community. It is critical to build relationships so that the local participants see their role in balancing relevance and rigorous research. Empirically, researchers can demonstrate that different schools share similar problems. Federally funded researchers should ensure that the local participants benefit from the research in some way.

Dr. Kane said that keeping stakeholders up to date on interim findings from research projects can inform decisionmaking at state and federal levels. Many researchers keep their results secret; it would be better to let the field know what is in the pipeline and when projects are likely to have results that may be useful.

Dr. Phillips pointed out that in early childhood education, researchers dig into questions more deeply through secondary research, often to learn whether an intervention deemed unsuccessful will work in some subpopulations or settings. She asked whether IES funding for large RCTs includes support for collecting data for later analysis. Dr. Phillips also asked how NCER plans to evaluate the success of its proposed research networks. That new research mechanism could foster the kind of collaboration and evaluation that is at the heart of this discussion, she noted.

Dr. Ford said the conversation has focused on how to improve the quality and dissemination of research. However, all classrooms have teachers, and the way teachers are trained has not changed. Dr. Ford proposed that IES focus on teacher improvement. He asked how all the research affects teachers and the schools of education that train them. Dr. Kane added that he believed school districts will begin creating their own professional development around Common Core State Standards, but there is no infrastructure in place to learn from those efforts.

Dr. Chard said schools in his states are "chasing exceptionalism" by constantly seeking out what is new instead of what works. The infrastructure Dr. Kane describes could be a network of state and local agencies that could force school boards to commit to an approach, which would provide longitudinal data on outcomes. Dr. Chard said there is a need to reeducate school and district leaders about how to look at what works. Dr. Kane said he hopes to build some consensus around effective learning so that schools can change the outcomes for their students.

Dr. Kemple supported Dr. Ford's notion of focusing on a single area and building a knowledge base around ideas that are likely to make a difference. He called for a unified IES culture that is visible, aggressive, and transparent to the field about what education science is and what standards IES promotes for the field. Strong partnerships can convince the field that IES standards really do matter. A clear vision of IES should come through in its funding priorities and requirements.

Lunch

The Board adjourned for lunch at 12:04 p.m. During the break, NBES members participated in ethics training, delivered by Marcia Sprague of the Ethics Division of ED's Office of the General Counsel. The public meeting resumed at 1:01 p.m.

Following lunch, Dr. Chard called for volunteers to serve on a task force to draft a letter to Congress about legislation under consideration related to the Family Educational Rights and Privacy Act (FERPA). Once drafted, the Board will hold a public meeting to discuss and approve the letter before sending it to ED. Dr. Loeb, Dr. Gamoran, and Dr. Hedges volunteered to serve on the task force.

Follow-Up Item
Dr. McKenzie-Thompson will facilitate the task force's effort to draft a letter to Congress expressing the Board's opinion about FERPA-related legislation. She will coordinate a public meeting of the Board to review and approve the draft.

Discussion of the RELs
Opening Remarks: Ruth Neild, Commissioner, NCEE
Panelists: Nikola Filby, Ph.D., Director, REL West, WestEd
Neal Finkelstein, Ph.D., Associate Director, REL West, WestEd
Barbara Foorman, Ph.D., Francis Eppes Professor of Education and Director, REL Southeast, Florida State University

Dr. Neild began the discussion by saying that the REL program is an example of a distributed approach for encouraging the production and use of research evidence in education. There are no easy answers to how to encourage states and districts to use research, and there are pros and cons to every approach. Moreover, resources are always limited, and priorities must be set. The current contracts for RELs will expire in January 2017; IES anticipates that the statement of work for the next REL contracts will be released by the end of FY 2015.

Dr. Neild provided some basic background about the RELs. She pointed out that each REL must have a governing board that includes state chiefs or their designees as well as other regional stakeholders. Because the RELs operate through contracts, not grants, IES staff are substantially involved in management of the RELs. NCEE seeks to ensure coordination across the RELs to eliminate duplication, to enforce high standards of quality, and to give a common appearance to reports.

The 2006 contracts focused on positioning the RELs as emissaries of science who primarily conducted RCTs. The 2012 contracts emphasized both research rigor and partnership. Rather than ask whether the pendulum is swinging away from rigor and toward relevance, Dr. Neild said she believes the RELs raised the bar with the 2012 contracts by attending to rigor while also building partnerships and addressing the usefulness and clarity of research.

Dr. Neild posed some questions for consideration:

What other improvements are needed in how we generate and use research in education that the REL program could help to advance?

How can IES better prepare and incent individuals who are good researchers and good collaborators to work with RELs (or on other similar projects)?

For the upcoming RELs, how can we encourage more robust competition in each region?

Facilitating Research in REL Southeast Research Alliances
Barbara Foorman, Ph.D., Francis Eppes Professor of Education and Director, REL Southeast, Florida State University

Dr. Foorman said her organization sees rigor and relevance on the same continuum. Practitioners often want answers to questions that will help them solve an immediate problem, but with limited time and resources, the challenge is prioritizing research. The REL works with practitioners to ensure that their questions are grounded in research and can be answered through studies. It also aims to build the capacity of practitioners to conduct their own research.

To address practice problems, the REL:

  • uses its governing board and research alliances to identify significant, meaningful research questions;
  • applies a logic model to clarify inputs and outputs;
  • conducts bridge events to build research knowledge and workshops on gathering and interpreting data; and
  • encourages states to share lessons learned with other states.

Dr. Foorman stressed that getting the right group of people to form a governing board means including legislative liaisons and finding SEA officials who are closely connected with programs. She said recently retired superintendents who are well respected by their peers have a lot to offer (while current superintendents rarely have time to participate). Foundation and business representatives are also important.

Dr. Foorman described some of the successful aspects of REL Southeast research alliances. All include LEAs and SEAs at the center of activities, which helps promote research across the state. Alliance members help pick topics for workshops and events and coordinate the logistics. A bridge event on language and literacy research in Florida led a school district to request a focused study on language among non-English-speaking students, which sparked a bridge event on the new practice guide on ELLs. That event prompted a teacher to document her implementation of the practice guide, which became the basis for a professional development offering. Dr. Foorman said the "circularity" of the work is healthy and exciting.

Dr. Foorman said the REL held events to promote results of the Beat the Odds study, which demonstrated better-than-expected results, but it did not get much attention in Florida until the SEA's communication director saw the infographic and distributed it to other leaders. The mode of dissemination matters, Dr. Foorman noted.

Dr. Foorman described one early literacy RCT recently completed. The participants were happy to take part because there was no "business-as-usual" arm and both of the conditions being compared were attractive. The study built capacity, because schools hired experienced staff as interventionists who stayed on after the researchers left. Dr. Foorman also outlined two other successful REL efforts to help states collect and share data on targeted issues of concern.

Research alliance members help disseminate research, facilitate bridge events, and serve as presenters and discussion leaders at REL events, Dr. Foorman pointed out. They also provide feedback on products and help distribute REL products. Dr. Foorman said the best products employ good graphic design and accessible, accurate language. They are also easy to identify and share. Dr. Foorman concluded that the REL Southeast offers several examples of how rigor becomes "contagious" when stakeholders are engaged in solving relevant and scientifically-verifiable problems in education through research.

REL West
Nikola Filby, Ph.D., Director, REL West, WestEd
Neal Finkelstein, Ph.D., Associate Director, REL West, WestEd

REL West has eight research alliances, all of which differ in their makeup. The goal of REL West is to increase the use of data and evidence in decisionmaking, representing the intersection of relevance and rigor. To achieve this goal, the REL actively partners with research alliance members; uses a range of rigorous analytic methods; provides tools and models processes (e.g., for collecting data); informs local decisionmaking processes, and disseminates research broadly.

Through its partnerships, the REL seeks to build a culture of continuous improvement. For some partners, using evidence in decisionmaking represents a different mindset, so the REL encourages them to understand their current systems, think about alternatives, act with intention, and use data to inform the next steps.

The Educator Effectiveness Alliance involves three states. All three were required by legislation to develop education evaluation systems, so they were motivated to partner with the REL. REL West assisted states in planning pilot studies and conducting psychometric analyses of measures. It enabled states to build the capacity to run their own analytics and to use the data to make changes. For each initiative, the REL created a simple, graphic logic model to help those involved think about the process as a whole and provided detail in a separate handout for the alliance partners. Research findings are disseminated through partners and other RELs.

Most importantly, said Dr. Filby, the states used the information they gathered to improve. They provided guidance and training based on the initial results and modified their evaluation models. They enacted legislative changes to fix unrealistic timelines and weightings. The states developed tools for collecting data, and they conducted more pilot programs to understand how teachers are using the data.

Dr. Finkelstein described the Silicon Valley Research Alliance (SVRA), which focuses on improving math skills, beginning in middle school, as a step toward college eligibility. The SVRA determined that algebra plays the role of "gatekeeper" for progress in math. Analysis of high school and middle school data indicated that students in area schools had a problem with the transition from basic to more advanced math. There was no standard, linear sequence of math courses to support advancement. A number of students took algebra twice, and some never took senior-level math courses.

In 2014, the REL West and SVRA published the results of their collaboration, which focused on using assessment data to guide math course placement. The study found that an inexpensive diagnostic test can predict students who will successfully complete algebra I in eighth grade and was a better predictor than a standards-based assessment.

Later this year, a follow-up publication will describe the results of the Elevate Summer Math Program on math achievement and algebra readiness. Elevate is a short-term, intensive intervention on algebra for rising eighth-graders. It incorporates extensive professional development for teachers and embeds support. Results of the program feed back to the SVRA within 1 year to inform program improvement.

Another SVRA effort provided technical assistance and professional development to teachers who were grappling with transformational geography, a new component of the Common Core State Standards. The effort uses student data to assess teachers' use of embedded assessments.

Discussion

Dr. Gamoran praised the RELs for creating networks of practitioners and policymakers to develop the topics of research that the RELs then address. He urged the RELs to go further to enhance engagement with their partners, so that the RELs are not just disseminating the information but building close relationships with practitioners and decisionmakers to understand the research.

Dr. Neild clarified that the RELs do not just disseminate information but truly partner with practitioners and policymakers to engage in research. For the RELs, the primary audience is research alliance members. Beyond that, the broader audience is the region that a REL serves and, even more broadly, the whole country.

Picking up on an earlier point about defining the audience, Dr. Foorman said that when the REL Southeast targeted principals directly and engaged them in meetings around research, the results were more effectively disseminated. Dr. Filby said more models are needed to help RELs meet the challenge of engaging partners at multiple different levels.

Regarding Dr. Neild's question about training researchers to better partner with others, Dr. Phillips said the question should be how to train researchers, practitioners, and decisionmakers to work within research partnerships. She asked what capacities school districts have that facilitate partnerships and how such capacities could help create more joint partnership models.

Dr. Finkelstein responded that the capacity to conduct research varies widely within and across school districts. Ultimately, successful efforts require a champion who has a keen interest in research and strong desire to create mechanisms for research within the district. Districts must also have the technical capacity for research and an understanding of the tradeoffs (e.g., inconvenience and burden). Dr. Filby said it is challenging to find school districts that want to partner in research.

Dr. Foorman pointed out that researchers need training on how to work in partnerships, and apprenticeships are among the few ways to gain that training. She suggested that IES' pre- and postdoctoral training involve working with the RELs. By apprenticing to a senior investigator in a research partnership, a trainee not only gains skills but also builds relationships that facilitate the continuation of the partnership over time. Dr. Chard suggested that such an approach could be incorporated into the REL contracts.

Dr. Yoshikawa said that investigators are expected to demonstrate the impact of their research but not the impact of their partnerships or training models. Incorporating partnership in pre- and postdoctoral training and integrating trainees into the RELs would be great, he stated. Dr. Yoshikawa asked whether the accountability structure for the RELs has changed over the years. He also asked whether the RELs have evaluated research alliances as a whole to understand what works.

She said the REL West seeks to understand the differences between partnering with an existing group to form an alliance or starting anew. In some cases, an existing structure helps facilitate research, while in others, it is not always clear that research fits into the mission and goals of the group.

Dr. Foorman said funding is a barrier to sustaining research alliances. In some cases, however, an alliance fails because the members are so different that they are unable to learn from one another. Some alliances work better at a small, local scale. Sometimes, a small alliance will mushroom. To improve the potential for sustainability, Dr. Foorman said the RELs should identify the pressing policy issues that stakeholders face and the extent to which they are shared within a region. When common interests are identified, members are motivated to join and continue an alliance.

Dr. Yoshikawa said alliances can be assessed from the perspective of costs, duration, or sustainability. The current contract fosters one model, but efforts should be made to identify other models for forming research alliances and to determine whether such alliances could achieve more and better results.

Dr. Loeb asked what the RELs have learned that make the process of creating and sustaining research alliances easier in the future. Dr. Filby said the tools developed through the alliances are helpful, as is sharing practical advice and finding areas of convergence around research. Dr. Neild clarified that the IES website houses all the tools and products created by the RELs, and she is working to raise awareness about them.

Dr. Finkelstein said that research alliances need to be "fed" regularly or the members lose interest. One challenge is determining the best "food." Ultimately, the research must be packaged in ways that meet the members' needs. On the basis of the REL West model, Dr. Finkelstein said, one cannot overstate the importance of maintaining a presence in the field to ensure that the partners stay engaged and contribute to the generation of important topics for research. Sustaining research alliances is a resource-intensive endeavor, he emphasized.

Dr. Foorman said another important mechanism for sustainability is pairing researchers with practitioners who are immersed in research and can act as a liaison. For example, she partners with a former practitioner and policymaker who is highly respected in the field and who is completing his doctoral work at the REL Southeast. Dr. Foorman added that the REL is figuring out ways to automate dissemination and information-sharing so that feeding the alliances is not a drain on the REL.

Dr. Singer pointed out two areas of concern. First, the research alliances described by the presenters center on topics that the presenters have pursued in their own research efforts. Dr. Singer questioned whether the REL contracts are acting as an umbrella for researchers to continue their research careers. Also, the examples given again highlight local exceptionalism by suggesting that the research findings are unique to the geographic areas in which the research was conducted. Dr. Singer wondered whether the regional RELs could partner with each other. She also said the RELs are well positioned to survey their alliance members to identify common areas of interest across states and regions.

Dr. Singer hoped that IES would think about mechanisms to expand the reach of RELs. At the end of the next round of REL contracts, she hoped that IES would be able to articulate what the RELs produced that could not have been achieved by others under another mechanism.

Dr. Chard asked for clarification of the peer review process for REL products. Dr. Filby said the RELs are still required to go through the peer review process, and the most recent contracts made the requirements clearer. Dr. Neild replied that all products are reviewed for technical quality, relevance, and readability. In general, the RELs like and want the IES "stamp of approval" that comes from the review process. The primary audience for the RELs is practitioners, but contractors are encouraged to publish peer-reviewed articles about their work for other audiences, such as their peers in education research.

Dr. Yoshikawa asked Dr. Neild to describe the range of products that RELs produce. Dr. Neild said the RELs still conduct plenty of RCTs and develop research reports. The category of "tools" encompasses a range of things that the RELs can use to expand on what works. Examples include REL Southeast's summer camp self-study guide, facilitation guides, or slide sets. REL Pacific created software for developing a customizable logic model. Dr. Neild said she would like to see more creative efforts to make tangible, practical products. Dr. Yoshikawa suggested that IES encourage innovation and development of new tools in the next REL contracts.

Dr. Finkelstein noted that over the next 16–18 months, the RELs will be working to ensure that the results of research completed are reviewed and disseminated, especially for topics that state boards of education are wrestling with right now. It is important to demonstrate that the research is relevant to current situations, he concluded.

Dr. Singer observed that, through the RELs, the federal government is playing a direct role in providing professional development regarding, for example, how ELLs are taught in Florida, and she questioned whether such a role is appropriate. As IES revises its REL contracts, Dr. Singer suggested thinking about the balance between technical assistance that verges on federal engagement in local school programming and the goal of research and dissemination.

Nexus Between Student Drug Use and Student Achievement: The Interagency Adolescent Brain Cognitive Development (ABCD) Study
Opening Remarks: Tom Brock, Ph.D., Commissioner, NCER
Panelists: George F. Koob, Ph.D., Director, NIAAA, NIH
David K. Mineta, Deputy Director, Office of Demand Reduction, ONDP
Susan Weiss, Ph.D., Director, Division of Extramural Research, NIDA, NIH

Dr. Brock said that, in recent meetings, the Board has discussed the idea of IES collaborating with other federal agencies. He explained that NIH is launching a study of the effects of drug and alcohol use on adolescent brain development. As part of the study, NIH is interested in data on students' school outcomes and school context. The study is in the early stages, and the investigators are open to input on its design.

Youth Substance Use and Academic Achievement
David K. Mineta, Deputy Director, Office of Demand Reduction, ONDP

Mr. Mineta said ONDP is alarmed and concerned about student substance abuse and its influence on academic performance, truancy, dropout rates, violence, unemployment, sexually transmitted infections, and other health indicators. As the central coordinator of drug control activities across the federal government, ONDP focuses on preventing and treating substance abuse and reducing relapses.

Mr. Mineta described several research findings that underscore the need to address substance use in the adolescent and teen years. Drug use escalates during the teen years. One study showed that as 12th-graders' perception of the risk of occasional marijuana use went down, their use of marijuana went up. Persistent marijuana use that begins early in life and extends into adulthood leads to significant drops in IQ. Students with low grades are more likely to use substances than those with better grades. As school climate improves, academic achievement improves.

Mr. Mineta referred Board members to the ONDP website and its National Drug Control Strategy for details about prevention efforts and ONDP's work with ED and education stakeholders.

The Science Behind Neuroprevention
George F. Koob, Ph.D., Director, NIAAA, NIH

Dr. Koob described recent evidence showing dramatic increases in the intensity of binge-drinking, particularly among 18–24-year-olds but extending into later adulthood. Binge drinking leads to risky behaviors, such as unprotected sex and fighting. Early substance use is tied to substance abuse and dependence later in life.

Dr. Koob explained that the frontal cortex of the brain does not fully develop until about age 25. Alcohol and drug use in adolescence disrupts the frontal cortex function and slows development. Specifically, marijuana impairs memory, and alcohol abuse impairs executive function, or decisionmaking. The combination of impaired decisionmaking and the underdeveloped frontal cortex lead to the impulsivity and compulsivity that drive addiction.

The National Consortium on Alcohol and Neurodevelopment in Adolescence is studying the effects of alcohol use on brain development in 800 subjects, ranging from 12–18 years, sampling them over time with brain scans. Early data demonstrate that brain size correlates with executive function and improvement on attentional tasks. Researchers can harmonize data across different types of brain scans, which provide information about the structure, pathways, and function of the brain. This study was the impetus for the ABCD study, a large, joint research effort around adolescent brain development.

Adolescent Brain Cognitive Development: A Longitudinal Study
Susan Weiss, Ph.D., Director, Division of Extramural Research, NIDA, NIH

The ABCD study is a national, longitudinal study of 10,000 people ages 9–20 to assess the effects of drugs on individual brain development. It involves multiple NIH Institutes, including the National Institute of Mental Health and the National Institute of Neurological Disorders and Stroke (NINDS). The latter is particularly interested in head trauma, often associated with sports injuries, and its interaction with substance abuse.

One impetus for pursuing this research now is the changing policies around marijuana use. Data show that alcohol use by 12th-grades in general is going down, but marijuana use is going up. As Dr. Koob explained, the adolescent brain is still developing and vulnerable to insults. Repeated, frequent use of marijuana in adolescence is associated with long-term changes in brain structure.

A large study in New Zealand showed a dose-response relationship between marijuana use before age 17 and adverse outcomes, including marijuana addiction, use of other drugs, and suicide attempts. Early marijuana use was also associated with lower high school graduation rates.

It is possible that substance users have different brains than nonusers or other genetic or physiological factors that affect current or later use. The ABCD study would enable investigators to gather baseline data as early as age 9. The large, nationally-representative, long-term study would allow researchers to see changes over time.

Dr. Weiss explained that NIH released a funding opportunity announcement (FOA) in which it outlined its vision for the ABCD study and asked applicants to propose how to accomplish the goals of the research. Response to the funding opportunity was robust, and awards should be announced by September. NIH hosted an expert panel workshop and put out a request for information for more input on the study's research design. It will use the input to revise the study design, then host a symposium for further discussion. Once awards are finalized, NIH will bring the researchers together for discussion about the final protocol. Dr. Weiss said NBES still has time to give its input on the research design. An external advisory board will also be convened to advise on the ABCD study.

The ABCD study will address the impact of diverse patterns of substance use on brain function as detected by imaging. Technology has advanced to the point that researchers can get good, reliable results with brain imaging from different laboratories. The study will look at the consequences of substance abuse, including academic achievement. It will assess how drug use affects the expression of psychopathology, including substance use disorders, and how the emergence of psychopathology influences drug use. Finally, the ABCD study will look at factors influencing drug use and how use of drugs contributes to other drug use.

Dr. Weiss anticipated that subjects would be tested every 2 years. NIH plans to release data for use as soon as they are available. The number of research sites is not yet known, and some sites may use a hub-and-spoke design. Dr. Weiss hoped NBES would provide insights on how the study could use the information that IES researchers are collecting.

Discussion

In response to Dr. McLaughlin, Dr. Weiss said she did not know how samples would be chosen, but the study is seeking a representative sample. Grantees will determine the sampling. Dr. Weiss anticipated that grantees would work with school systems, but the ease with which schools can collaborate with researchers varies across states.

Dr. Hedges asked whether "representative sampling" meant probability sampling, which ensures that individuals have an equal probability of being included in the sample and is the basis of claims about generalizability. Dr. Weiss said experts in sampling would address the specifics, but the goal is to have generally normative sample that includes, for example, people from rural areas and people of various race/ethnicity.

Dr. Loeb asked whether researchers already have some theories about the causes of substance use. Dr. Weiss said there is a lot of evidence about risk factors and investigators have ideas about what to look for. The study will collect biological specimens for genetic evaluation, but there is not enough funding to analyze all the samples.

Dr. Koob said NIH will have to limit the number of hypotheses in the study unless more money becomes available. Dr. Weiss said the study is not as hypothesis-driven as others. She expected that hypotheses would emerge and that other studies would be tacked on to the ABCD cohort as it develops over time. For example, NINDS is very interested in studying traumatic brain injury through the ABCD study.

Peggy Carr, Ph.D., asked whether the external advisory board or some other entity would address "the hard questions." Dr. Weiss said the advisory board, NIH staff, and grantees will handle technical issues. She recognized that the research may face ethical challenges, such as when to alert adults that drug use was detected, especially in very young children. The FOA identified the areas of expertise needed; applicants will address the requirements, and the external advisory board will tackle any areas not well enough represented in the applications.

Robert A. Underwood, Ed.D., asked what agencies are eligible for awards under the FOA. He also asked whether applicants in states that have legalized recreational marijuana use would have a better chance of being funded. Dr. Weiss replied that grantees can be individual sites or preexisting collaborations, and Washington and Colorado are well positioned. The study will have statistical controls in place for sites where marijuana is legal and will have to account for changes in policy throughout the study. Dr. Koob added that the award uses a consortium mechanism, and NIH staff have significant influence on consortia. With so many Institutes and Centers participating and making decisions about the study, it will not be "a free-for-all," said Dr. Koob.

Dr. Yoshikawa applauded the study, especially in light of the sudden demise of the National Children's Study. He noted that gathering contextual information about schools and peer groups is expensive. However, NCES has extensive data sets, and NIH could add biomarkers. Dr. Yoshikawa said the study could be a step toward developing national, longitudinal data sets that include biological and physiological markers.

Dr. Brock said IES collects a lot of data about student performance and school climate, for example. He asked whether early conversations about the ABCD study considered what kinds of questions could be answered with education research data. Mr. Chapman said NCES met with NIH a few times to discuss the administrative records data that NCES collects. He said NCES has a potentially strong set of assessment outcomes data and a good relationship with the U.S. Census Bureau, which agreed to provide district and school boundary mapping.

Dr. Brock asked how NIH is thinking about partnering with education researchers. For example, will the data become public? Can education researchers propose add-on studies? Dr. Weiss said the study is still seeking more funding, but data sets will be available, and secondary data analyses are encouraged. She thought education researchers would be an important part of the study, because education outcomes are a key issue. Dr. Koob added that the project is evolving, but it is his job to make sure it is not overloaded beyond capacity. He noted that the subjects would have imaging or neuropsychological testing every year, so any additional testing must be short.

Dr. Hedges suggested linking NCES items to NIH assessments and linking to existing data sets by state. In the future, it may be possible to get individual academic performance and contextual information. Dr. Hedges said NIH could leverage a huge amount of existing data without new testing. Dr. Weiss said concerns have been raised about the difficulty of getting individual academic data. Dr. Carr said NAEP was able to get information at the individual level through states, despite complicated privacy laws. She said NCES can provide suggestions on how to gather school data at minimal cost.

Dr. Hedges suggested NIH consider linking to international data sets, and Dr. Weiss responded that NIH is talking with several international groups. Dr. Hedges recommended that the research team include experts in causality. Dr. Weiss agreed to bring the suggestions to the advisory board for inclusion in the study design.

Dr. Phillips advised NIH to consider the lessons learned from the National Children's Study, which grappled with many of the same issues that the ABCD study is facing. Dr. Carr suggested talking with the Office of Management and Budget, because it keeps track of funding and resources across the federal government.

Closing Remarks & Adjournment
David Chard, Ph.D., NBES Chair

Dr. Chard said the next NBES meeting is scheduled for October 2. He will present a draft of the suggested letter to Congress regarding funding for education research for Board members for review. He will also take steps to convene those who volunteered to address issues related to two bills around FERPA. Both of these actions require swift movement.

Dr. Gamoran said he liked the organization of the meeting agenda, and Dr. Chard reiterated his call for feedback and suggestions from Board members for future agenda development. Dr. Chard adjourned the meeting at 4:05 p.m.

Report prepared for NBES by Dana Trevas, Shea & Trevas, Inc.

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.