September 9th   September 10th

Location:
Institute of Education Sciences Board Room
80 F Street NW
Washington, DC

September 9th

Members Present:
Duane Alexander, Ph.D., Ex Officio, National Institute of Child Health and Human Development
Jonathan (Jon) Baron, Coalition for Evidence-Based Policy
Arden L. Bement, Jr., Ph.D., Ex Officio, National Science Foundation
Elizabeth Ann Bryan, Akin Gump Strauss
Phoebe Cottingham, Ph.D., Ex Officio, Institute of Education Sciences
Carol D'Amico, Ed.D., Conexus Indiana
David C. Geary Ph.D., University of Missouri
Robert C. Granger, Ed.D., William T. Grant Foundation
Eric A. Hanushek, Ph.D., Hoover Institution, Stanford University
Caroline M. Hoxby, Ph.D., Stanford University
Gerald (Jerry) Lee, WBEB 101.1 FM Philadelphia
Lynn Okagaki, Ph.D., Ex Officio, Institute of Education Sciences
Mark Schneider, Ph.D., Ex Officio, Institute of Education Sciences
Sally E. Shaywitz, M.D., Yale University
Dixie Sommers, Ph.D., Ex Officio, (delegate) Bureau of Labor Statistics
Joseph K. Torgesen, Ph.D., Florida State University
Grover J. Whitehurst, Ph.D., Ex Officio, Institute of Education Sciences

Staff:
Norma Garza, Executive Director, NBES
Wilma Greene for Mary Grace Lucier, Designated Federal Official
Anne Ricciuti, Ph.D.

Guests:
Fred Doolittle, Ph.D.
Beth Gamse, Ph.D.
Steve Baldwin, Ph.D.

Call to Order
Robert Granger

Action: Approval of meeting agenda

Dr. Granger called the meeting to order at 10:00 a.m. Roll-call was taken and observers were introduced.

Dr. Granger began by informing participants that minutes from the previous NBES meeting had been approved by e-mail, and that they were posted on the NBES website.

Dr. Granger described the agenda and then called for a motion to approve it. A motion was made and seconded, and the Board approved the agenda unanimously.

Chair's Report
Robert Granger

Dr. Granger noted the general tenor of political change and transition as Dr. Whitehurst's term at IES comes to an end, and the nation prepares for a presidential election. He expressed his hope, however, that the federal government and the profession would continue to support and build on the substantial improvements that IES had created in the field of education research over the past six years. He said that the Board will also undergo a transition as the 4-year tenure of five Board members—himself, Dr. Hoxby, Ms. Bryan, Mr. Lee, and Dr. Torgesen—comes to an end. Dr. Granger said that Board would hold elections during a closed session at the end of Wednesday's meeting to choose a new Chair and Vice Chair given that his and Ms. Bryan's terms were ending.

Update on IES
Grover J. Whitehurst

Dr. Whitehurst thanked Dr. Granger, and said that under Dr. Granger's direction, the Board had kept IES "on its toes" while supporting the organization's important work. Dr. Whitehurst also acknowledged the ex officios for serving as valuable anchors during Board meetings.

Noting his work on his final biennial report, Dr. Whitehurst spent a few moments recounting the history of federal funding of education research. The lack of a strong research base in education in the United States prompted passage of the Education Sciences Reform Act, and led to the establishment of the Institute for Education Sciences. Dr. Whitehurst said that substantial progress has been made since 2001, which is evidenced in the draft evaluation reports.

Over the years, IES has increased its research funding, developed a strong peer review system, and made substantial progress in improving the timeliness of reports and the growth of longitudinal databases. As to the question of whether predictable and consistent effects on student outcomes have been identified, Dr. Whitehurst stated that research and data about teacher quality, assessment and accountability, and early childhood education were three critical areas where IES- funded work had created findings that were relevant to practice and policy.

Looking ahead to IES' ongoing and future work, Dr. Whitehurst said that researchers, practitioners, and policymakers should collaborate on large-scale research and development projects to solve major state and district educational challenges such as decreasing dropout rates. These projects would require significant additional investments in dollars and political will to produce rapid progress.

In terms of policy focus, IES must maintain its commitment to transformation and the rigorous pursuit of evidence-based educational science, without compromising quality for expedient results. Dr. Whitehurst challenged Board members to be aggressive in pursuit of the public interest when the federal government fails to provide quality services.

Update on IES Center Activity
IES Commissioners and Staff

After taking questions in response to Dr. Whitehurst's report, Dr. Granger informed the Board about the mission of the What Works Clearinghouse (WWC) Experts Panel. Dr. Granger said that the Appropriations Committees in both the U.S. House of Representatives and the U.S. Senate had expressed concerns about the work of the WWC, and that both committees had called for extensive reviews to determine whether it was producing scientifically valid information on its website, and if so, whether the information is being effectively disseminated and applied.

Dr. Granger said that in consultation with Dr. Whitehurst and others, he had taken the lead in July 2008 in establishing an independent panel of senior scientists—familiar not only with experimental work but with the synthesis of research information—to assess the scientific validity of evidence-based practices promoted by the WWC. He acknowledged that this was a narrower scope than envisioned in the congressional language, but it was a necessary first step that could be accomplished in a timely fashion. The six panel members (Dr. Hendricks Brown, Dr. David Card, Dr. Kay Dickersin, Dr. Joel Greenhouse, Dr. Jeff Kling, and Dr. Julia Littell) will consider three case histories, focusing on five aspects of the WWC: (1) formulation of inclusion criteria; (2) screening of studies; (3) evaluation of studies; (4) extraction of data; (5) reporting. Their report to the Board is scheduled for mid-October.

National Center for Education Evaluation and Regional Assistance (NCEE)
Phoebe Cottingham

Dr. Cottingham said that she would discuss the dissemination function of the NCEE and the main apparatus used to get dissemination evidence out to the huge field of practitioners, the Regional Education Laboratory (REL) Program. By law, products and reports that are sent out must meet IES Scientific Standards. Further, the law requires an external peer review process. This system responds to the problem that a great deal of unexamined opinion passes as evidenced-based research.

Dr. Cottingham said that one of 60 studies that successfully completed the peer review process focused on re-enrolling dropouts. The research focused on one district during a 3-year period, looking at how many dropouts were able to re-enroll and how long they remained in school. Although the study received some press, finding a way to publicize the results of these types of studies has been a challenge. The mission for the next 2 years is to schedule events and design other strategies to convey factual information to practitioners who need to understand the scientific validity of study results and how to locate relevant data that could inform their work.

Dr. Hoxby asked whether practitioners, who often ask causal questions, are satisfied with the descriptive reports available from the RELs. Dr. Cottingham said the job of the RELs is to share information that is available from the WWC, from IES reports, and research literature. The goal is to redefine the question and use literature reviews of available studies to derive an answer. Following up on Dr. Hoxby's question, Dr. Granger asked whether NCEE gathers data on customer satisfaction and use of the information to determine whether the RELs adequately respond to needs. Dr. Cottingham said that NCEE will conduct surveys to obtain this information; the problem is obtaining client lists to determine who actually received the information.

In looking at the structure of the RELs, Dr. Hanushek raised a question about the structure of the RELs, with regard to how addressing the same problems across the ten regions could take advantage of economies of scale with regard to the similar contract components. The RELS require 10 different contracts, governing boards, and organizations, and there is no requirement that they all have the same platform of potential projects. Dr. Cottingham responded that the structure is required by statute. Noting the potential for redundancy in impact evaluation studies conducted by the RELs, Dr. Geary asked whether information is shared. Dr. Cottingham said the RELs discuss products and projects during meetings twice a year, and that studies are not duplicated.

National Center for Education Research (NCER)
Lynn Okagaki

Dr. Okagaki said that as the National Center for Education Research (NCER) reviews the state of research in the field, three major issues have surfaced. One is that NCER funds very few applications with respect to commercially available curricula. She then asked whether the Centers should encourage this type of research, and if so, how. Several Board members encouraged such studies, although others were skeptical that such studies would attract top scholars.

Another issue is how to make interventions sufficiently potent and robust to truly improve student outcomes. Finally, in considering teacher professional development interventions, Dr. Okagaki said producing changes in student outcomes through teacher professional development is very difficult.

National Center for Education Statistics (NCES)
Mark Schneider

Dr. Schneider described progress in 10 different areas, including a high school longitudinal study, school and staffing survey, international benchmarking, and others.

Several Board members commended Dr. Schneider for the NCES progress under his direction.

Dr. Granger asked for an update on proposed Family Educational Rights and Privacy Act (FERPA) regulations. Dr. Whitehurst said at some point within the next month or two the regulations should be available.

Reading First
Beth C. Gamse

Dr. Gamse began her presentation by stating that the Reading First program is a substantial federal investment, with more than $6 billion spent thus far to improve the quality of literacy instruction in the early grades. The goal of the program, which focused on existing research, was to ensure that all students read on or above grade level by the time they get to third grade.

In reporting the interim results, Dr. Gamse said Reading First had an impact on highly explicit instruction, such as active teaching, modeling, or explaining of concepts, as well as the five dimensions measuring the recommended components of reading instruction, phonemic awareness, phonics, fluency, vocabulary, and comprehension. Reading First had an impact on only one grade, grade 2, on high-quality student practice. Interestingly, the program reduced the percentage of students engaged with print in grade 2. The program had no effect in grade 1. Overall, the study found no statistically significant differences in reading achievement scores in 1st, 2nd, and 3rd grade.

Board members asked additional questions about the core reading programs that schools already use and whether the study provided a true comparison between schools that use research-based instruction and schools that do not. If a study could be redesigned, Dr. Gamse said that she would design a planned variation study, based on program components rather than funding.

Impact Evaluation of Academic Instruction for After-School Programs
Fred Doolittle

Dr. Doolittle reported the first-year findings of a project that tested whether formal after-school instruction with high quality curricula in reading or math produces better academic outcomes than the usual after-school services. The study relied almost exclusively on certified teachers to work with students. Further, the teachers received initial training, on-site technical assistance, and paid preparation time. After an in-depth process of selecting curricula, the study team chose a Harcourt School Publishers math curricula for after-school called "Mathletics." The curriculum for reading was an adaptation of the reading program used in elementary schools by Success for All.

In describing the findings, Dr. Doolittle posed the initial question of whether children would attend. Researchers were concerned that students would be scared away or "burned out" by the additional study time at the end of the regular school day. Results showed that students did attend and engagement remained high. For example, Dr. Doolittle reported slightly higher attendance among the students in the enhanced instruction group than in the regular program. Both the enhanced after-school group and the regular group showed growth in math performance. The enhanced group, however, showed somewhat larger growth. In reading, the growth rates for the enhanced group and the regular group were about the same.

Dr. Doolittle said the study has added a second year of operations, data collection, and analysis. Of the 50 after-school centers involved, 27 will continue into the second year.

Dr. Shaywitz asked whether there was any information about how the chosen curricula related to the teaching taking place in the classrooms. Dr. Doolittle responded that the study ended up in sites that were not using the same curricula during the school day. Dr. Hoxby asked about the cost differences in the after-school programs, given that the enhanced group worked with certified teachers. Dr. Doolittle said that this information was not included in the report.

Trends in IES Review
Anne Ricciuti

Dr. Ricciuti addressed the allegation that top researchers are not applying to IES because the IES review process will only give high marks to proposals that conform to narrow, methodological criteria. She presented data showing that, overall, methodologists and statisticians did not score applications more harshly than non-methodologists. In addition, she presented data showing that IES funds many projects that involve the use of research designs other than randomized trials, and noted that even in areas where IES strongly prefers randomized trials, a number of projects using other designs have been funded.

Dr. Ricciuti also addressed concerns about the review of special education applications. Some researchers have expressed concern whether special education applications that are reviewed on panels that are not specifically focused on special education applications are fairly reviewed. Dr. Ricciuti presented data showing that: 1) reviewers with special education expertise on "mixed" panels do not generally score more positively than other panel members, and 2) there is no evidence that, overall, NCSER applications fare worse in review than NCER applications.

Dr. Ricciuti also provided an update to the Board on the peer-review structure. It consists of standing panels, to which members are appointed for multiple consecutive review sessions; and single-session panels, during which members are appointed for individual review sessions. The newest standing panel is the social and behavioral panel, established during the current fiscal year.

IES Evaluation Committee Report
Joe Torgesen

Dr. Torgesen briefly discussed the purposes of the IES evaluation and the Board's initial review of the contractor's findings, as well as the evaluation committee's concerns and recommendations about the draft report. He reported that the Board's Evaluation Committee had reviewed the draft report as had IES staff, and that following the meeting the Contractor would receive all comments. He reported that the contractor would then have until September 30 to consider revisions and to submit a final draft.

Summary Views
Robert Granger

Dr. Granger brought the meeting to a close by reviewing the next day's agenda. During the first session, Larry Orr, Rebecca Maynard, and Rob Hollister would give a presentation on the NCEE contracted evaluations. Afterward, the Board would discuss the draft 5-year report in some detail. Dr. Granger said that he would ask members to comment on the report's tone, structure, and content and then briefly review the document section by section.

The meeting adjourned at 4:12 p.m.

Wednesday, September 10th

Members Present:
Jonathan (Jon) Baron, Coalition for Evidence-Based Policy
Elizabeth Ann Bryan, Akin Gump Strauss
Phoebe Cottingham, Ph.D., Ex Officio, Institute of Education Sciences
Carol D'Amico, Ed.D., Conexus Indiana
David C. Geary Ph.D., University of Missouri
Robert C. Granger, Ed.D., William T. Grant Foundation
Frank Philip Handy, Strategic Industries
Eric A. Hanushek, Ph.D., Hoover Institution, Stanford University
Caroline M. Hoxby, Ph.D., Stanford University
Gerald (Jerry) Lee, WBEB 101.1 FM Philadelphia
Lynn Okagaki, Ph.D., Ex Officio, Institute of Education Sciences
Mark Schneider, Ph.D., Ex Officio, Institute of Education Sciences
Sally E. Shaywitz, M.D., Yale University
Dixie Sommers, Ph.D., Ex Officio, (delegate) Bureau of Labor Statistics
Joseph K. Torgesen, Ph.D., Florida State University
Grover J. Whitehurst, Ph.D., Ex Officio, Institute of Education Sciences

Staff:
Norma Garza, Executive Director, NBES
Wilma Greene for Mary Grace Lucier, Designated Federal Official

Guests:
Rob Hollister, Ph.D.
Rebecca Maynard, Ph.D
Larry Orr, Ph.D

Review of Previous Day's Activities and Today's Agenda
Robert Granger

After introductions, Dr. Granger announced the agenda for the second day of the NBES meeting. Dr. Granger asked Ms. Bryan to share the quote inscribed on the memento presented to Dr. Whitehurst during an informal dinner the previous night. That quote was "In grateful appreciation to Grover "Russ" Whitehurst for being in the first group 'THERE ARE BASICALLY TWO TYPES OF PEOPLE. PEOPLE WHO ACCOMPLISH THINGS AND PEOPLE WHO CLAIM TO HAVE ACCOMPLISHED THINGS. THE FIRST GROUP IS LESS CROWDED."~ Mark Twain.' " Dr. Granger also expressed his appreciation for the memento he received from the Board.

NCEE Mid-Stream Evaluation Presentation
Dr. Cottingham introduced this session by describing how she had commissioned a panel consisting of Dr(s) Rebecca Maynard, Larry Orr, and Rob Hollister to review the current NCEE portfolio of evaluations. Her purpose was to get an independent review of the current work with the hope that lessons could be learned that would inform the design and execution of future IES-funded studies.
Rebecca Maynard

Dr. Maynard gave an overview of the size, focus, and status of the evaluation portfolio. The portfolio includes 26 studies, seven of which have had reports released. The studies are quite diverse; 11 have been mandated by law. Other studies are special requests from the Office of Management and Budget or the Office of Elementary and Secondary Education. IES staff has shaped the content of another 10 of these studies under the guidance of various authorities.

Dr. Granger asked Dr. Maynard to explain the comment that a number of studies come under the guidance of various statutes or other type of accountability. Dr. Cottingham said that funding for all evaluations is either specifically allocated by Congress, or is obtained from programs that want an evaluation completed. Funds are negotiated with the Department for particular studies and the Department allocates funds to IES for that work. No independent funding is available for any of this work.

NCEE Mid-Stream Evaluation Presentation
Larry Orr

Following Dr. Maynard's presentation, Dr. Orr discussed the quality of the design of the NCEE evaluations, recognizing that most were ongoing and had not yet yielded findings or final reports describing the implementation of the projects. The evaluation team assessed the studies on the basis of seven criteria, including whether studies would provide unbiased impact estimates; the quality of measures; and the precision of the studies. In general the panel was very favorable about the design of the evaluations. One area of concern, reinforced by the Board's discussion, was the sample size in some studies. This is a problem when classrooms or schools are the unit of random assignment, and in some cases evaluations may not detect effects that are relatively small but policy-relevant.

NCEE Mid-Stream Evaluation Presentation
Robert Hollister

Dr. Hollister discussed his concern that it may be that relying on curriculum and professional development experts for advice on particular interventions and outcomes to be tested creates a distance between what is needed for theory development and validation as opposed to the policy relevant environment. He encouraged staff to think about what should be tested and how should it be tested.

Ms. Bryan asked whether evaluators have paid sufficient attention to confounds that can occur with control and treatment groups, and how they can affect study outcomes. Dr. Orr said that IES has been focusing on this issue, observing classroom practice and attempting to measure the fidelity with which an implementation was implemented.

Mr. Lee asked about studies that prepare teachers for early childhood programs. Dr. Cottingham said that none are on the docket. Funding for studies on teacher quality come out of Title II, and discussions are ongoing with Title II program leaders about their priorities.

Mr. Baron suggested broadening the circle of experts to bring in practitioners from the field. Dr. Cottingham said practitioners eventually become part of large-scale replication studies. Dr. Torgesen said that it is important to obtain preliminary data before launching large-scale studies; however, many supplemental interventions have resulted from the experiences of seasoned teachers, who have worked intensively with students.

Five-Year Report Discussion
Robert Granger

Dr. Granger described the tone and major findings contained in the draft five-year report. He characterized the storyline as "excellent progress on improving research rigor and promising progress on improving the relevance of the IES work and IES's fostering of evidence –based decision making". He then remarked on some of the key findings related to the overarching tone of the report, noting certain information that was still to come on various activities. He then opened the floor to the Board for a discussion on the draft. In general Board members concurred with the tone and substance of the draft. Several Board members underscored the importance of focusing on the executive summary, wanting it to be a document that conveyed the Board's positive view. All agreed that the Board assessment was very positive; however, several encouraged a general tone for the final document that allowed the facts about progress to speak for themselves, without any praise that might appear excessive.

Following discussion, the Board agreed to a process that would lead to a final, approved report. A subset of Board members would read the next draft; then a final draft would go to the full Board for signoff. The intent was to have this all done by mid-late October so that the report could be submitted by mid-November.

Election of NBES Chair and Vice Chair

At the end of the public meeting, the Board moved into executive session to discuss the Board chair and vice chair positions. In the executive session Erik Hanushek was selected to become Chair and Jon Baron was selected as Vice Chair. Both terms are for two years to commence on November 29, 2008 following the end of terms for Dr. Granger and Ms. Bryan. The meeting adjourned at 1:00 p.m.

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.