Location
Institute of Education Sciences (IES) Board Room
80 F Street NW
Washington, DC 20001

Participants

National Board for Education Sciences (NBES) Members Present
David Chard, Ph.D.
Darryl J. Ford, Ph.D.
Adam Gamoran, Ph.D.
Robert Granger, Ed.D.
Kris D. Gutierrez, Ph.D.
Larry V. Hedges, Ph.D.
Susanna Loeb, Ph.D.
Bridget Terry Long, Ph.D.
Margaret R. (Peggy) McLeod, Ed.D.
Judith Singer, Ph.D.
Robert A. Underwood, Ed.D.
Hirokazu Yoshikawa, Ph.D.

NBES Members Absent
Anthony S. Bryk, Ed.D.

Ex-Officio Members Present
John Q. Easton, Ph.D., Director, IES, U.S. Department of Education (ED)
Thomas Brock, Ph.D., Commissioner, National Center for Education Research (NCER)
Joan McLaughlin, Ph.D., Commissioner, National Center for Special Education Research (NCSER)
Brett Miller, Ph.D., Health Scientist Administrator, Child Development & Behavior Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health (NIH)
Ruth Curran Neild, Ph.D., Commissioner, National Center for Education Evaluation and Regional Assistance (NCEE)
Chuck Pierret, Bureau of Labor Statistics

NBES Staff
Rebecca McGill Wilkinson, Ph.D., Associate Research Scientist, NCER
Ellie Pelaez, Designated Federal Official

ED Staff
Elizabeth Albro, Ph.D., NCER
Corinne Alfeld, Ph.D., IES
Vanessa Anderson, Ph.D., IES
Sue Betka, IES
Joelle Lastica, Ph.D., IES
Diana McCallum, Ph.D., IES
Audrey Pendleton, Ph.D., IES
Anne Ricciuti, Ph.D., IES
Elizabeth Warner, Ph.D., IES

Invited Presenters
Eileen de los Reyes, Deputy Superintendent of Academics, Boston Public Schools (BPS)
Sean Reardon, Ed.D., Stanford University
Gabriela Uro, Manager, ELL Policy and Research, Council of Great City Schools (CGCS)
Jeffrey C. Valentine, Ph.D.,, Principal Investigator, Development Services Group, Inc. (DSG); Associate Professor, College of Education and Human Development, University of Louisville

Members of the Public
Judy Ahn, National Academy of Education
Kerry Belodoff, SRI International
Craig Fisher, American Psychological Association
Jean Gossman, LRP Publications
Hannah Klein, Society for Research in Child Development (SRCD)
Jim Kohlmoos, Edge Consulting, LLC
Jonathan Magin, SRCD
Sarah Mancoll, SRCD
Josh McCrain, Consortium of Social Science Associations
Michele McLaughlin, Knowledge Alliance
Paula Skedsvold, American Educational Research Association (AERA)
Sarah Sparks, Education Week
Christy Talbot, AERA
Alison Thompson, Lewis Burke Associates, LLC
Greg White, National Academy of Education

Call to Order
Ellie Pelaez, Designated Federal Official

Ms. Pelaez called the meeting to order at 8:32 a.m. and called the roll. Board members unanimously approved the agenda for the meeting.

Election of New Board Leadership

With the consent of the Board, Bridget Terry Long, Ph.D., former chair, agreed to lead the discussion. The terms of the chair and vice chair ended in November. Because the October 2013 Board meeting was canceled (as a result of the federal government shutdown), new officers were not elected. According to the Board's charter, a chair can only serve for 2 years. IES Director John Q. Easton, Ph.D., pointed out that the Board can request that the ED Secretary amend its charter.

Several members praised Dr. Long's leadership, noting that the Board was very productive during her tenure as chair, and hoped the next chair would continue that momentum.

Adam Gamoran, Ph.D., said the Board is at a critical juncture. Dr. Easton's term as director ends in May 2015, and the National Center for Education Statistics (NCES) does not have a permanent commissioner. In addition, the Board has been weakened by the lack of an executive director to coordinate the Board's work and ensure it maintains a presence on Capitol Hill, said Dr. Gamoran. He hoped the Board would continue to make gains in advancing, promoting, and legitimizing the education research agenda.

Judith Singer, Ph.D., said Dr. Long did an excellent job of gathering perspectives from all sides, engaging with IES commissioners and others, and fostering more engagement with IES. She appreciated that Board meetings under Dr. Long had a clear purpose (e.g., understanding and informing the work of the IES commissioners).

Dr. Long said she appreciated the opportunity to build relationships with IES staff and provide helpful feedback. She said she did not anticipate how much external effort was required of the chair, and she noted that the Board suffered from the lack of a staff person located in Washington. Dr. Long announced that the Board would like to hire a consultant who will not play the same role as the former executive director but will be helpful. She said future chairs should be ready and willing to build relationships and travel to Washington, DC, as needed. She added that she would be happy to work with the next chair to provide any input or background.

Kris D. Gutierrez, Ph.D., also applauded Dr. Long's leadership. She reiterated that the work on the ground (e.g., meeting with Congressional staff) is essential to advancing the Board's agenda. She noted that the new consultant should be someone who will keep his or her finger on the pulse of federal government and keep the Board informed.

Robert Granger, Ed.D., said the next chair should insist on being involved in the process of reauthorization of the Education Sciences Reform Act (ESRA). The Board chair has more leeway for advocacy than federal employees do, so the next chair should have both an appetite and skill for it.

Dr. Granger also noted that the next chair will play a critical role not only in selecting the next IES director but also in managing the transition. Hirokazu Yoshikawa, Ph.D., suggested the Board revise its charter to extend the term of the chair so that the next chair can stay in place through the transition.

Follow-Up Item
At an upcoming meeting, Board members will discuss revisions to the charter and request that the ED Secretary amend its charter.

Darryl J. Ford, Ph.D., felt that the Board should better articulate its specific goals for each year. It should also develop orientation materials for new members that explain the Board's role and responsibilities.

Margaret R. (Peggy) McLeod, Ed.D., nominated David Chard, Ph.D., to serve as the next chair. Dr. Gamoran nominated Dr. Gutierrez, noting that the vice chair often becomes the chair. The nominees were asked to speak.

Dr. Chard said that as a dean, he has learned that an important part of leadership is the ability to listen and ensure that others' voices are heard. It is also key to see that agenda topics are thoughtfully developed and that decision-making is well informed. He hoped to stretch the limits of the Board's relationship with IES leaders and work externally to make the role of the Board as important as it should be.

Dr. Gutierrez hoped to engage Board members in informing peers about the Board's vigorous, national agenda to promote education research. As vice chair, she and Dr. Long brought in speakers to talk to the Board about new practices and pressing issues in the field. Dr. Gutierrez said she has leadership experience in academia but also as chair of AERA.

With a multidisciplinary group, Dr. Gutierrez continued, the challenge is to honor an institution's traditions but also push to ensure the institution remains relevant and current. She said she used to believe—naively—that policymakers would see the Board as the go-to source for input; she now recognizes that Board members must maintain a presence among policymakers to ensure their voices are heard. Dr. Gutierrez said she is fortunate to have many connections on which she can rely.

Dr. Granger asked the nominees to discuss what issues they believed the Board should address and how the Board's direction should be determined. Dr. Gutierrez said the vast experience and expertise of the Board members has not yet been leveraged. Additionally, she suggested getting more input from the field about pressing problems of practice. Dr. Chard echoed the concern, noting that organizations have many issues they would like to address. He suggested following up on topics from past meetings to assess progress.

Dr. Gamoran asked the nominees to address 1) how to assess the relevance of IES' work, 2) what role the Board can play in ESRA reauthorization, and 3) how to position IES more prominently as more federal agencies move toward evidence-based policymaking. Dr. Chard said he needs to better understand the limits of the Board, but hoped he could push for using the expertise of the Board in practice and policymaking to inform decisionmakers. Dr. Gutierrez suggested more policy briefings, both formal and informal, and informing Congress about the Board's capacity to provide advice. She added that she is deeply committed to the IES relevancy agenda, which demonstrates the agency's concern for practice.

Susanna Loeb, Ph.D., asked how the nominees would find time for the demands of the chair position. Dr. Gutierrez said that hiring a consultant and calling on Board members and others for assistance would help. She noted that she and Dr. Chard both attend a lot of meetings in Washington already, so there are opportunities to represent the Board outside of Board meetings. Dr. Chard said the busiest people are often ideal for leadership roles. He added that if he were elected chair, he would step down from leadership in some other organizations.

Action Item: Board members cast paper ballots, and Dr. Chard was named the new chair, effective immediately.

Dr. Long nominated Dr. Gutierrez to serve another term as vice chair. Dr. Granger nominated Dr. Loeb. Dr. Gutierrez withdrew herself from consideration and nominated Dr. Gamoran instead. Nominees were asked to address the topics raised earlier.

Dr. Gamoran said his top initiative would be taking stock of the IES relevance agenda. Dr. Loeb said she would like to focus on how Board members can do more outside of meetings to support the IES research agenda. Dr. Gamoran noted that he has a new job with fewer people to supervise and spends a lot of time in Washington already.

Dr. Singer noted that the charter does not stipulate that the Board have a vice chair. Instead, the chair could appoint an executive committee, so he could delegate work to multiple people with a range of expertise. The new consultant could coordinate such an approach.

Action Item: Board members cast paper ballots. The nominees were tied, and Dr. Gamoran withdrew. Dr. Loeb was named the new vice chair, effective immediately.

Former and New Chairs' Remarks
Dr. Long declined to make comments. Dr. Granger moved to commend Dr. Long and Dr. Gutierrez for the skill, wisdom, and time they devoted to their roles and to thank them on behalf of the Board for their service. The motion was seconded and unanimously approved.

Motion Approved
The Board commends Dr. Long and Dr. Gutierrez for the skill, wisdom, and time they devoted to their roles and thanks them for their service.

Update: Recent Developments at IES
John Q. Easton, Ph.D., IES Director

Dr. Easton welcomed Joan McLaughlin, Ph.D., as the new commissioner of NCSER. Dr. Easton also said he opted to serve as acting commissioner of NCES to facilitate greater integration between NCES and IES. The agency is actively searching for a new permanent NCES commissioner.

Dr. Easton outlined the 2014 IES budget. Assessment activities saw a significant increase, thanks to some Congressional mandates, but most other lines remained at post-sequestration levels and below the agency's 2014 funding request. While special education research funding increased, all of the funding will be used to meet continuation costs.

Dr. Granger asked how other research agencies fared. Brett Miller, Ph.D., said the NIH's budget increased slightly over the post-sequestration level. An audience member said that NIH's budget increased 3.9 percent and NSF's increased 4.2 percent over the post-sequestration levels.

At IES, funding for statewide data systems decreased, despite the request to increase funding by nearly $50 million. Dr. Easton said the request was optimistic; however, he also acknowledged that states were struggling to spend these funds, so perhaps the money could be better used elsewhere.

Dr. Gamoran expressed surprise that funding for education research did not return to pre-sequestration levels and felt that advocates should speak out about it. He asked why Congress increased the budget for assessment beyond the IES request, but Dr. Easton said he was unable to verify the rationale. Larry V. Hedges, Ph.D., said the increase may mean that Congress believes accountability is more important than research.

Dr. Singer hoped that IES would assess where education research stands in relation to medical and scientific research, as reflected by funding. The budget cuts mean that the Board and IES must get the message out about the importance of education research and work to reverse the negative funding trend, she said. In relation to the overall ED budget, Dr. Easton said ED saw increases in some programs, but its signature competitive grants did not fare very well.

Drilling down into the budget for research, development, and dissemination, Dr. Easton said the money available for new grants in current competitions stabilized at a level closer to the amounts for 2011 and 2012. Dr. Granger pointed out that the new budget does not include new funding.

Dr. Yoshikawa said the budget figures underline the importance of developing and maintaining good relationships with key Congressional committees and staffers. He noted that in his experience, Congress wants proof of the concrete impact that funded research has on major policy issues. Dr. Long agreed, saying that in her testimony before a Congressional committee, she was asked to identify studies that have had an impact. She added that there is a mismatch between the length of time that rigorous research takes and the patience of funders looking for answers.

Dr. Easton said the Reading for Understanding (RfU) initiative is in its final year. Once it ends, a large amount of money will be freed up for other IES efforts (beginning in 2015).

Funding for new research in special education was eliminated by Congress, said Dr. Easton. He described a history of cuts to special education research, partly related to the number and quality of applications received when NCSER was first established. Dr. Chard clarified that this situation was created, in part, because many special education researchers had been submitting their proposals to NCER and that the limited number and quality of applications did not reflect a lack of capacity in the field at large.

Finally, Dr. Easton said there was a lot of activity around ESRA in the fall, but a potential deal in the House fell apart over appropriations. Recent activity in the Senate has raised hopes again, he concluded.

National Center for Education Research (NCER)
Thomas Brock, Ph.D., NCER Commissioner

Dr. Brock began by noting that over 800 proposals received by NCER for research funding in Fiscal Year 2014 are currently under review, and that grant awards would be announced in the spring. As in past years, the number of awards will be based on scientific merit as determined by peer review and available funding.

In the interest of improving its grant application process, NCER surveyed last year's applicants about the structure of its Requests for Applications (RFAs) and the technical assistance NCER offers. The surveys were conducted online and were anonymous. The surveys received about a 60 percent response rate, and the feedback was mostly positive. For example, most respondents found the RFAs clear and well-organized, and said that the technical assistance was useful. Respondents also noted some areas where the language was confusing or the same information was asked for in different sections. NCER is revising the RFAs to address these issues, and the improvements will be reflected in the next set of RFAs that are released in the spring.

Following up on a Board discussion last year, NCER convened a technical working group on college and career readiness standards. This is a topic that generates many strong opinions. Proponents of new standards think they will improve instruction and learning and ensure that high school graduates are prepared to compete in the global economy. Opponents worry that the standards will reduce the flexibility of schools to tailor their curricula to their students and the needs of their community. Both sides are concerned about the heavy emphasis on standardized assessment and the possibility of negative effects on students who are struggling in school or who have special needs.

The technical working group called for good descriptive research of how standards are being implemented and longitudinal studies to track student outcomes. The group said that a rigorous impact evaluation is probably not feasible, but efforts should be made to determine how to measure the effectiveness of specific curricula or professional development programs that are aligned with the standards. The group also called for new measures to capture changes in instruction and class practices that result from new standards. NCER is reviewing the technical working group's recommendations, and is considering how these recommendations might be addressed through its various research grant programs.

In response to a question from Dr. Gamoran, Dr. Brock said NCER expects to release all of its RFAs in the spring, and that all proposals likely would be due at the same time in the summer. Dr. Gamoran said that staggering deadlines for competitions would ease the burden on research institutions, who struggle with the capacity to respond when many proposals are due at one time.

National Center for Special Education Research (NCSER)
Joan McLaughlin, Ph.D., NCSER Commissioner

Dr. McLaughlin said that as a result of the budget cuts in in 2013, NCSER could only fund 40 percent of the proposals that review panels rated as outstanding or excellent. For primary research, NCSER funded 14 of 26 such proposals. For the Early Career Development and Mentoring competition, NCSER funded all three highly rated proposals. For the Accelerating the Academic Achievement of Students with Learning Disabilities Research Initiative, Dr. McLaughlin was disappointed that NCSER could only fund one proposal.

With no money allotted for new research in 2014, NCSER will not fund any new competitions. However, NCER funded the Researcher–Practitioner Partnerships in Education Research, which includes special education researchers. NCSER plans to fund a new research competition in 2015 and hopes to continue supporting new work.

In response to Dr. Chard, Dr. McLaughlin said the subject of research is usually the deciding factor in determining eligibility for NCSER or NCER grants. However, some topics overlap, such as projects on attention-deficit hyperactivity disorders.

Dr. McLaughlin noted that NCSER has only been funding grants since 2006, so much of the research is just coming to fruition. Her staff is working to inform the field about the questions that NCSER-funded research answers. NCSER has always relied on the What Works Clearinghouse (WWC) for dissemination. Dr. McLaughlin said NCSER needs to do more to get the word out. NCSER has been synthesizing research and posting more results in the WWC, providing an annual overview of the field, and providing more research summaries to the Education Resources Information Center (ERIC). NCSER also uses various social media networks to get information out quickly.

In response to Dr. Gutierrez, Dr. McLaughlin said NCSER has funded some research on the growing number of English-language learners (ELLs) with special needs, and staff is currently working to help applicants improve the quality of proposals submitted. Dr. Brock added that NCER funds a fair amount of research on ELLs.

Dr. McLaughlin said that IES responded to the White House Office of Science Technology and Policy (OSTP) request that agencies increase public access to federally funded research findings. Once IES receives feedback from OSTP on its draft plan, IES will publicize the plan, which will have strong implications for researchers, said Dr. McLaughlin.

National Center for Education Evaluation and Regional Assistance (NCEE)
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild focused on dissemination, a topic of frequent conversation with the Board. NCEE has sought feedback and targeted efforts on the user's needs. Dr. Neild reported good progress in alerting users about new information, explaining the findings clearly, and putting the findings in context. More work is being done to help users understand how to apply the findings in their own settings.

Dr. Neild said NCEE is thinking about how to make its products more useful. For example, in addition to the full report and stand-alone executive summary, NCEE is producing study snapshots of two to four pages. The website will include videos of researchers talking about their projects. For one initiative, NCEE sponsored a live online forum to discuss research findings. NCEE is simplifying its evaluation briefs, using Twitter to push information, and improving the website's news flash function. Dr. Neild welcomed suggestions from Board members about how NCEE could help users apply research findings.

The WWC has been expanding dissemination by repackaging its previously published work, using themes likely to be of interest to a broad audience. The first effort focused on back-to-school topics; the most recent looks at college access issues. Dr. Neild said she's received good feedback about the thematic approach. Tweets about it by Sec. Duncan and others at ED have helped raised the profile of the WWC.

In November, the WWC released a short practice guide on teaching math to young children in both print and e-book format. A product feedback meeting in August brought forth a lot of good ideas for disseminating products of the WWC and the Regional Education Laboratories (RELs) and reinforced support for many products already in the pipeline.

Among the many new products from the RELs is a software package for developing logic models. Dr. Neild said more ED RFAs are requiring logic models. NCEE is also trying to help RELs improve their news blasts.

The redesigned IES website will give users more ways to search the site, revealing the broad range of information and products IES has in various formats, said Dr. Neild. Branding—that is, clearly identifying the source of each product—is posing a challenge. Dr. McLeod praised the dissemination efforts as the best thing IES has done since she has been on the Board. She was very pleased with the "amazing progress."

In response to Dr. Singer, Dr. Neild said IES has a fair amount of data about website usage that has helped shaped the design and organization of the site. It is considering creating some topical pages and already has ways to collect click-through data on various links.

Dr. Singer asked whether the IES website would ever send users to external, trusted organizations. Neild acknowledged the benefits of such links but said that more thought would be needed, since the government is careful not to advantage one group over another. To maintain the level of effort required to keep the site fresh and push information out, Dr. Neild said NCEE would probably rely on a combination of staff and contractor assistance.

National Center for Education Statistics (NCES)
John Q. Easton, Ph.D., IES Director, NCES Acting Commissioner

Dr. Easton described the reorganization of NCES into three major divisions: assessment, sample surveys, and administrative data. He also noted that to eliminate redundancy, NCES no longer has a separate business operations office.

During the reorganization, there was concern about keeping the content people together without creating a new silo. The NCES now has four cross-cutting teams, organized around topics, that meet monthly.

In response to a request from President Obama, NCES is assisting ED in developing a postsecondary institution rating system (PIRS). NCES has issued a request for information and is convening a symposium on potential data elements and metrics, methods of data collection, methods of weighting or scoring, and presentation of frameworks for a PIRS. NCES will issue a public report on this high-profile effort.

Also in response to a White House request, NCES is developing School Climate Surveys to collect data to support implementation of policies for safe, drug-free schools. The surveys will be available on a platform that enables schools to conduct the surveys on demand, generate data and reports, and keep the data locally.

A major new National Assessment of Educational Progress assessment on technology and engineering learning is now in the field. It involves scenario-based tasks completed by the students. There is agreement on the need for such an assessment, but there is still some question about what exactly the assessment measures, said Dr. Easton. Dr. Gamoran praised the effort, pointing out that schools are doing little around technology and engineering education.

Ongoing Efforts to Improve IES: Debriefing on the House Committee Hearing and Government Accountability Office (GAO) Report
Opening Remarks by Bridget Terry Long, Ph.D., and John Q. Easton, Ph.D., IES Director

Dr. Long explained that in mid-September, the GAO released a report about IES, and she testified before a U.S. House of Representatives committee about the report. In her testimony, Dr. Long described IES' strengths and opportunities for improvement. Among the areas of improvement, Dr. Long described the need to increase dissemination of research findings and improve relevance and usability of the research. She also summarized efforts underway by IES to address these issues.

Dr. Long said the committee members were clearly knowledgeable about IES programs and sought information about important contributions and research impact. The questions focused on issues of relevance, usability, and communication of findings.

The GAO report found that IES supports high-quality research and has substantially improved the quality of education research, but stakeholders have expressed concerns about IES' ability to produce timely and relevant research that meets their various needs. In addition, IES does not have a structured process for incorporating stakeholder input into its research agenda. The report also criticized IES performance measures and assessment of its own efforts. Dr. Long asked for input from IES commissioners and Board members on how to address the issues raised and respond to the report.

Dr. Easton said that in discussions with Congressional and GAO staff, he learned that IES is trusted for its objectivity, quality of work, integrity, and accuracy but also that IES is failing to show that its work matters. He stressed that the IES is doing wonderful work, but the results are not being disseminated. The research community must focus on what matters to schools and the ultimate goal of improving education, said Dr. Easton.

Discussion

Dr. Gutierrez pointed out that even those within ED do not always recognize or use IES products, although Dr. Easton countered that IES is aggressively promoting itself within the department. IES seeks to be not just as a source of information but also a resource for understanding the context of the knowledge accumulated.

Dr. Yoshikawa stressed the importance of providing concrete examples of impact. For example, IES research demonstrating the effectiveness of a math building blocks curriculum led Boston to adopt the approach for all of its pre-K programs. IES should mine its archives to find more such examples of programs that scaled up on the basis of solid research findings. Dr. Hedges added that it is useful to tell the story in a way that highlights why it takes time for research to manifest as policy changes at scale. The RfU project may provide some wonderful examples, he noted.

Dr. Gamoran asked whether IES faces challenges in finding good examples or communicating them. Dr. Easton said both contribute to the challenge, but communication is the larger issue right now. Dr. Ford suggested IES revisit its priorities and put more resources toward better communication, perhaps by hiring a consultant instead of relying on researchers and administrators to disseminate the information. Dr. Singer agreed that the GAO report may be an impetus for IES to consider what it might do differently and think about radical changes.

Dr. Hedges reminded the Board that IES has become a trusted source for education research, something that did not exist in the past. Robert A. Underwood, Ed.D., wondered what triggered the GAO evaluation and report. He believes that persistent prejudice against education research may be reflected in the GAO's findings.

Dr. Chard asked how IES could persuade more researchers to better communicate their findings. Dr. Granger said it may be appropriate to focus heavily on a few issues across centers, as NCEE did with its thematic approach to content. It is impossible to address all of the relevant issues of interest to all stakeholders, he added.

Dr. Easton noted that IES had an opportunity to respond to the GAO's findings, and its response is incorporated into the final report. In general, IES concurred with the findings, and the response describes efforts underway and specific planned steps. In addition, the Secretary is working on a response.

Board members discussed whether the Board should craft a response to the GAO report to avoid the perception that it agrees with some of the unfair criticisms. Dr. Long said a response should emphasize that IES was aware of many of the issues identified before the report was presented and had already begun addressing them. Moreover, the response should have concrete examples of relevant research that had an impact in the field.

Dr. Granger said that IES must consider whether it is doing enough to gather input from stakeholders. It must also identify the root of the enduring impression that IES is not listening to stakeholders or communicating its findings. Dr. Yoshikawa suggested establishing an ongoing mechanism for gathering input from other stakeholders and presenting that input around key issues to the Board periodically.

Dr. Long suggested the new consultant make it a priority to gather the material for a response, get contributions from staff and Board members, and circulate the response among the Board members for review. Dr. Yoshikawa said crafting a statement will take a lot of time. He suggested that before the next meeting, IES consider how the Board could function better and what role Board members should play in dissemination. Dr. Loeb said that instead of writing a direct response, IES could focus on updating its website to highlight accomplishments and demonstrate impact.

Dr. Singer also felt that a Board response would involve a lot of effort, may have little impact, and could be perceived as boosterism. She suggested that Board members and other stakeholders renew their commitment to partnering with IES to provide constructive feedback and bring in the perspectives of other stakeholders not at the table.

Dr. Long said the GAO report highlights key issues and demonstrates that IES is moving in the right direction. However, some may see the GAO report as proof that IES does not work and should not be funded. Therefore, it is important to respond. Dr. Long said the Board's annual report could describe what IES is doing right as well as identify areas that still need improvement. She said the lack of a response could lead to defunding. (The discussion was suspended without resolution.)

Lunch

During the lunch break, NBES members participated in ethics training, delivered by Marcia Sprague of the Ethics Division of ED's Office of the General Counsel.

The public meeting resumed at 1:07 p.m.

Supporting English Language Learners
Differences Among Instructional Models in English Learners' Academic and English Proficiency Trajectories: Findings from the San Francisco Unified School District (SFUSD)/Stanford Research Partnership
Sean Reardon, Ed.D., Stanford University

Dr. Reardon discussed findings from a partnership between Stanford University and SFUSD to assess differences among instructional models in ELLs' academic and English-proficiency trajectories. He summarized the various types of ELL programs offered across SFUSD (immersion, bilingual, and dual-immersion) and the theories behind each. The partnership supports numerous research projects and includes external funding to support a full-time research coordinator who has been key in bringing the organizations together.

The district is ethnically diverse, and about 37 percent of students are ELLs who primarily speak either Chinese or Spanish. As such, Dr. Reardon cautioned, the results of the research may not be generalizable. He further described the ELL programs and study design. The study outcomes were reclassification (i.e., no longer designated as ELL) on the basis of English fluency and proficiency as well as levels of math and English language arts achievement.

Dr. Reardon said there were differences in outcomes between Latino and Chinese students that his study cannot explain. The shapes of their trajectories differ, and there is a distinction between English and academic proficiency and progress toward reclassification across the two groups.

The outcomes by type of ELL program are similar, said Dr. Reardon; that is, a similar proportion of ELL students reaches English proficiency around 5th grade, regardless of the approach. Chinese students start kindergarten at a higher level of proficiency than Latino students and progress more quickly to a higher level of proficiency by 5th grade. Similarly, Chinese ELLs were more likely than Latino ELLs to be reclassified by 5th grade. The SFUSD is using the results to create ELL report cards for each school, and results will be presented to the community as well as school administrators, ELL staff, and the school board through discussion groups, conferences, and publications.

Dr. Reardon shared some lessons learned so far:

• IES research standards are hard to meet in the context of a school district research partnership.
• Results may not be generalizable and communicating that constraint may be difficult.
• It takes a lot of time and close work with the school district to ensure the data can be used for analysis, but the process yielded rich data that allow for verification.
• School districts and researchers have different agendas; the timeline of university research may be too slow for schools.
• Resources to support partnership are essential. The funding to support a data infrastructure in the district brought a lot of people to the table.
• Having a coordinator who can be a liaison has been invaluable.

ELLs and the CGCS
Gabriela Uro, Manager, ELL Policy and Research, CGCS

Ms. Uro provided a national perspective on ELL. She noted that CGCS represents less than 1 percent of all school districts, but those districts account for over one quarter of all ELLs. There is real and troubling variance in how ELLs are counted. School districts know how many ELLs they serve, but national databases are inconsistent. Notably, to determine the number of ELLs in a given area for funding decisions, organizations rely on surveys of adults that ask whether English is spoken in the home, which represents a completely different measure, said Ms. Uro.

CGCS evaluated its districts and found that most could be categorized into one of two groups: those with 1,001 to 5,000 ELLs and those with 10,001 to 50,000 ELLs. Ms. Uro pointed out that what works for a district with 50,000 ELLs may not work for one with 1,001 ELLs. Decisionmaking, priorities, and benchmarks all vary depending on the context. Thus, the context in which ELLs are being served has implications for practice.

Every state is required to measure English language proficiency, but they all use different assessment tests, said Ms. Uro. It is not clear that "proficient" means the same thing across states. Moreover, even those schools that use the same assessment tests may use different cutoffs, yielding inconsistent results. Finally, the distribution of ELLs varies greatly across school districts, and it is not clear how to determine that all ELLs are accessing rigorous content while on their way to English proficiency.

With IES assistance, CGCS funded research that found that some programs had positive effects on ELLs at mid- to advanced levels of proficiency but limited impact on those at lower levels. Districts with large percentages of ELLs at the lower levels of proficiency need more tools to help those students.

Ms. Uro said assessment remains a huge challenge, because current tests do not take into account the realities faced by districts with high numbers of ELLs. She said CGCS is hoping to capture data on achievement by ELLs, and even if the results are poor, the effort will supply a lot of good data for researchers.

Mining the Data on Language Proficiency Levels
Eileen de los Reyes, Deputy Superintendent of Academics, BPS

Ms. de los Reyes explained that in 2009, the Department of Justice (DoJ) informed BPS that it was improperly using students' listening and speaking skills to assess ELL status instead of reading and writing. As a result, the number of BPS students designated as ELLs skyrocketed overnight.

With the recognition of poor outcomes among ELLs, BPS partnered with the University of Massachusetts to study its ELLs. Initial data describing the makeup of ELLs and the languages they spoke proved shocking to the district because it revealed so many more ELLs and so many more languages than anyone expected. The university partnership identified schools with successful programs and quickly translated their practices into other schools. BPS mounted an intensive effort to expand ELL programs and train teachers, and outcomes for ELLs began to improve.

Since implementing new programs, BPS can confidently state that ELLs who arrive in the system by kindergarten outperform their peers by 5th grade. However, it cannot guarantee such success for later entrants. Analysis of state assessment results indicate that ELL students must make substantial progress each year to reach achievement goals.

Ms. de los Reyes said that BPS is redefining the narrative of ELLs, as the former ELLs are outperforming other students in their district. The ELL program is considered desirable, and professional development for ELL teachers is seen as an intervention strategy for struggling schools. Schools that did not want to accommodate ELLs in the past now do want them in their schools. Assessments at later levels show that ELLs are learning content while they are developing English proficiency.

Several concerns remain, said Ms. de los Reyes:

• Leveling off of achievement among former limited-English proficiency students
• Less successful trajectories among children who enroll after kindergarten
• Children getting stuck at the third-grade level, because those who do not progress past grade 4 are likely to become long-term ELLs
• Children who transition to 6th grade without achieving level 4 or 5 English proficiency
• Understanding the reasons behind effective professional development so that it can be translated to other programs

ELLs with Disabilities
Gabriela Uro, Manager, ELL Policy and Research, CGCS

Ms. Uro said it is very difficult to gather meaningful data on ELLs with disabilities. However, the percentage of ELLs who require special education is increasing—and outpacing the needs of non-ELLs who require special education as well as the needs of other ELLs. Some robust national research on the issue would be beneficial, she said.

Some of the issues confounding school districts are as follows:

• How is the initial identification and diagnosis of ELLs with disabilities made?
• Which factors constitute typical language acquisition barriers and which represent disability?
• Are the assessment tools trustworthy?
• Is staff adequately trained to use the assessment tools and make a diagnosis?
• Are services available? It is hard to find special education teachers who also speak Spanish, the most prominent language of ELLs; finding special teachers who speak other languages is incredibly difficult.

Ms. Uro said the first step is to gather robust data on what works and how to implement it. With the Common Core State Standards coming online, school districts are making rapid changes, but little attention is being paid to how to assess ELLs and special education students. The School Improvement Grants program has been underway for 10 years and can provide a wealth of data for researchers, said Ms. Uro, because most of the grantees have high proportions of ELLs. She advocated for a national research effort using these data, which ED has already collected.

Discussion

Dr. McLeod said the federal investment in ELLs has been "shameful." She proposed that IES establish a center within NCER to fund ELL research. Such an approach would demonstrate needed leadership at the highest level and would boost sorely-needed research efforts.

Regarding Dr. Reardon's concerns about the generalizability of findings, Dr. Easton suggesting looking more closely at whether the tools or models developed to assess programs would be useful for other schools. Dr. Reardon said that if his team had a better understanding of what produced the results, it would have a mechanism for understanding the influence of factors such as teacher capacity and family background. Ms. Uro said the research design may be useful to others because it moves the focus on to academic achievement by all students.

In response to Dr. Chard, Ms. de los Reyes said BPS has not yet created a trajectory of success for students with different disabilities and aligned those trajectories with state assessment scores. There is much work to be done to better understand the interventions for ELLs with disabilities. Dr. McLaughlin said NCSER funds a Research and Development Center that is studying trajectories for children with disabilities, and she can share information on the Center with Ms. de los Reyes.

In response to Dr. Granger, Ms. Uro said many schools seem willing to adapt programs from other parts of the country when they see enough similarity in salient features, such as the percentage of ELLs or the dominant nationalities. She said schools are building on the groundwork of others.

Dr. Yoshikawa asked the commissioners to consider how the presentations align with their perspectives on ELL research in their portfolios. In the interest of time, Dr. Chard invited commissioners to address the issue at the end of the meeting.

What Works Clearinghouse (WWC)—Postsecondary Topics
Opening Remarks
Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild said that IES sees the WWC as a large investment that is supported by several contractors. There is no one contractor who is "the WWC." Currently, four contractors handle different aspects of the WWC, and IES staff links the contractors together so they work like a team. She noted that this arrangement brings in the fresh perspective of additional contractors. It may also increase the sense of ownership among the broader research community as more people understand how to conduct WWC reviews and how to translate research findings into intervention reports.

Dr. Neild said that postsecondary education posed a new challenge for the WWC, and she was grateful for the skills of the researchers who brought the topic to the WWC. Unlike K–12 curricula, for example, the interventions studied are not necessarily branded. There appears to be a need for updates to standards or additional standards for research designs more typically used in research on postsecondary education, said Dr. Neild.

Contractor Perspective
Jeffrey C. Valentine, Ph.D., Principal Investigator, DSG, WWC—Postsecondary Topics; Associate Professor, College of Education and Human Development, University of Louisville

Dr. Valentine explained that his role on the WWC is to review effectiveness studies in postsecondary education, with an eye toward developing new WWC products in this area and expanding the reach of the WWC. The review team includes both technical and content experts. Dr. Valentine said the WWC has strong potential to affect policy and practice and plays a role in consumer protection.

Development Services Group (DSG), one of the small businesses that performs work for the WWC and the business with which Dr. Valentine works, hopes to get the postsecondary research community more involved in the WWC and to make the work of the WWC more relevant, accessible, and known. It uses various approaches to increase the visibility of the WWC among the public and the education research community.

All contractor work is governed by plans approved by IES, but the contractors propose how to conduct the work. Dr. Valentine summarized the products developed (quick reviews, single study reviews, and intervention reports). He emphasized that, as a WWC contractor, DSG follows the same WWC procedures as the other contractors for the technical aspects and relies on subject matter experts and IES staff for content issues. In the past 14 months, DSG has completed 11 quick reviews, 15 single study reviews, numerous reviews for grant competitions, and a topic area review of students in developmental education that has already generated one draft intervention report and may yield others.

Dr. Valentine said the WWC does a good job on many technical issues. The products are of high quality and represent a high standard. The WWC uses well qualified, well trained staff to code research, and the protocols for data collection are exact and specific. The IES commitment to conducting a complete literature search is laudable. The Excel-based system for review documentation of the WWC's Study Review Guides is accurate and well suited to the purpose.

Some areas of improvement are beyond the reach of the WWC alone, said Dr. Valentine. For example, reporting on implementation is universally poor, but as the previous presentation on ELLs demonstrated, more must be reported on in the context of primary studies about the context of implementation so that researchers can better assess the potential for generalization. There is also little consensus in the field about what to report regarding program implementation factors. Another area is that within much postsecondary research, regression discontinuity studies are used, which Dr. Valentine believes is related to the explosion in the number and depth of databases available. High-level work is needed on updating regression discontinuity standards and on instrumental variable approaches to the estimation of causal effects, said Dr. Valentine. The WWC is currently working on refining regression discontinuity standards.

Among the areas that WWC can control and obtain expert consultation on is the synthesis of results from multiple studies. The various WWC contractors work together on various development teams, so there is opportunity for coordination.

The overarching areas for the postsecondary team to continue to address include the following:

• Technical aspects of using multiple contractors
• How to make substantial changes that require input from multiple people
• How to revise standards and standard operating procedures across multiple contractors
• Enhancing awareness of WWC in the postsecondary field
• Engaging journalists to help with the consumer protection role
• Building trust among researchers despite the lack of interventions and outdated regression discontinuity standards

Dr. Valentine concluded that WWC can be a leader in important conversations across the social sciences on reproducibility, replication, and reporting standards.

Discussion

Asked to clarify DSG's role in reviews for grant competitions, Dr. Neild explained that NCEE supports grant making at the Department of Education by reviewing the evidence on the potential effectiveness of proposals submitted for the tiered competitions (e.g., the Investing in Innovation Fund). Such reviews must be done quickly and on a large scale, so NCEE requested help from DSG and Mathematica (another WWC contractor). Because the contractors double-code the reviews, the reviews become part of the WWC.

Dr. Granger noted that other federal departments have tiered evidence competitions, but in those cases, the contractors create the evidence model, and applicants are funded if their evidence meets the criteria. He suggested that the Department of Health and Human Services, for example, may be interested in ED's model for evaluating applications for tiered-evidence competitions. An audience member pointed out that the DoJ requires grant applicants to use only evidence cited in DoJ's research clearinghouse.

Dr. Long asked what DSG is doing to educate researchers about the standards for inclusion in the WWC (e.g., implementation reporting details) and how DSG can respond to the rapid pace of change in postsecondary research. Dr. Valentine said DSG relies on its senior content experts to identify what type of research can and should be done.

It is hard to change directions quickly, Dr. Valentine continued, but DSG can spot issues as they bubble up through the media. Personal contacts also help DSG keep in touch. In contrast to the K–12 field, which has many branded approaches, the postsecondary field is characterized by broad sets of strategies or approaches. The postsecondary approach has been to look for strategies that have been evaluated rigorously and focus the first intervention reports in these areas.

Dr. Valentine said DSG is also thinking about how to engage more researchers and affect practice. To that end, he proposed that the WWC STAT team consider developing standards for implementation reporting, for example. The standards would be promoted through professional journals and conferences, and would take effect on a specified date.

Dr. Brock said that he had not heard but was intrigued to hear the term "consumer protection" used in relation to the WWC, and he asked Dr. Valentine to elaborate on the concept. Dr. Valentine said that he sees those making decisions on the ground (about policy or practice) as the consumers, and he hoped to protect them against shoddy science that masquerades as evidence.

Dr. Gamoran asked whether postsecondary research can look at outcomes beyond the typical realm of education, such as the labor market. Dr. Valentine replied that protocols are developed with input from experts and can include the relationship of the intervention to such outcomes as achievement, degree completion, and labor.

Dr. Granger asked about the approach of evaluating strategies or practices instead of branded programs. Dr. Valentine said the WWC usually looks at branded programs because it is focused on helping decisionmakers at the program level who are not choosing a strategy but rather selecting a program. However, now that there are multiple reviews of interventions, it may be possible to identify the common factors across interventions. Postsecondary research looks at general models or practices, and there are no specific programs to evaluate. Dr. Valentine said there are opportunities for researchers to use existing data to evaluate new hypotheses.

In response to Dr. Singer, Dr. Valentine said he would measure DSG's success by the extent to which the bar has moved for WWC processes and procedures. He would also assess awareness of the WWC among postsecondary researchers, who currently know little about the WWC.

Dr. Chard postulated that there could be an explosion of branded programs for postsecondary institutions. Dr. Valentine said that if the field moves in that direction and IES wants to support decisionmaking in the context of branded programs, contractors like DSG would review the programs and also consider how to identify commonalities across branded programs.

In response to Dr. Chard, Dr. Valentine said the only requirement for reviewing research is that it target individuals in postsecondary institutions or who are candidates for such. There has been discussion with Department of Labor contractors about areas of overlap and the need to keep each other informed. Dr. Long said the postsecondary research field is wide open, and IES could take the lead. IES may want to consider its position and goals in relation to the changing nature of postsecondary education (e.g., online education).

Closing Remarks
John Q. Easton, Ph.D., IES Director, and David Chard, Ph.D., NBES Chair

Dr. Easton noted that Michele McLaughlin of the Knowledge Alliance provided some budget information in response to questions raised earlier by Board members. He distributed the information but said he had not verified the quality or accuracy of the data.

Following up on the ELL discussion, Dr. Brock said that ELL is an area of focus within NCER that has its own program officer and topic areas. Since its creation, NCER has funded about 70 ELL projects touching on all five of its research goals. In addition, NCER funded a center on ELL research, led by David Francis, Ph.D., at the University of Houston; that center recently completed its mission. Its work resulted in development of a middle school curriculum. Also, within the Reading for Understanding project, at least three teams are addressing ELLs in different grades. Dr. Brock acknowledged that IES could do more.

Dr. Chard thanked the Board members for trusting him to take on the role of chair and thanked Dr. Long for her leadership. He planned to talk with Dr. Loeb about the structure and content of future meetings, what happens between meetings, and how to further engage Board members.

Dr. Chard returned to the question of whether the Board should respond to the GAO report and, if so, how. Dr. Gamoran felt strongly that the Board should respond with a brief message of support for IES. Dr. Long said her testimony before Congress may be a good place to start in crafting a response, because it reflects the thinking of the Board. Dr. Granger suggested that Dr. Chard and Dr. Loeb talk with the IES general counsel about the mechanisms available to respond. Dr. Easton reiterated that the IES response to GAO concerns is included in the GAO report and that the ED Secretary recently completed a formal response to the final report.

Dr. Gutierrez hoped Board members would let their colleagues know about the important topics that the Board is discussing. Dr. Gamoran said an NBES executive director would be key to organizing Board efforts and ensuring follow up. Dr. Underwood noted that Congress moves very slowly, so the Board does not have to rush.

Adjournment

The meeting was adjourned at approximately 3:51 p.m.

View, download, and print the full meeting minutes as a PDF file (109 KB)

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.