Location:
Institute of Education Sciences Board Room
80 F Street NW
Washington, DC

Board Members Present:
Mr. Jonathan Baron, Vice Chair of the National Board for Education Sciences
Carol D'Amico [via telephone]
David C. Geary
Eric A. Hanushek, Chairman of the National Board for Education Sciences
Philip Handy
Sally Shaywitz

Ex Officio Members Present:
Duane Alexander, National Institute of Child Health and Human Development [via telephone]
Phoebe H. Cottingham, Institute of Education Sciences
John Q. Easton, Director of the Institute of Education Sciences
Stuart Kerachsky, Institute of Education Sciences
Robert Kominski, Delegate, U.S. Census Bureau [via telephone]
Lynn Okagaki, Institute of Education Sciences
Dixie Sommers, Delegate, U.S. Bureau of Labor Statistics

Staff:
Norma Garza, Executive Director of the National Board for Education Sciences
Mary Grace Lucier, Designated Federal Official

IES Staff:
Sue Betka
Carola Conces, Intern
Tracy Dell'Angela
Ellie McCutcheon
Anne Ricciuti

ED Staff:
Carmel Martin, Assistant Secretary for Planning, Evaluation and Policy Development, U.S. Department of Education

Members of the Public:
Sarah Hutcheon, Society for Research in Child Development
Jady Johnson, Reading Recovery Council
John Kohlmoos, Knowledge Alliance
Dave Van Dyke, Washington Partners
Debbie Viadero, Education Week
Katy Vickland, SRI International

1:30 p.m.–1:45 p.m. Call to Order, Approval of Agenda, Chair Remarks, and Remarks of Executive Director

Dr. Eric Hanushek, National Board for Education Sciences (NBES) Chairman, called the meeting to order at 1:30 p.m. Ms. Mary Grace Lucier called the roll.

Dr. Hanushek welcomed participants and expressed his regrets that the meeting of the NBES previously scheduled to occur in May 2009 had to be canceled due to lack of a quorum. He introduced Dr. John Q. Easton, the newly appointed director of IES, who began his tenure on June 6, 2009. The meeting proceeded with brief introductory remarks by participants, and then by Dr. Easton.

Dr. Easton described his background as a researcher, analyst, and evaluator, both within and outside of the Chicago Public Schools system. Dr. Easton was part of the original group that founded the Consortium of Chicago School Research at the University of Chicago, which was established more than 18 years ago. He explained the Consortium's role as an external, independent, and objective partner for Chicago Public Schools. During his 12-year tenure with the Consortium, Dr. Easton provided indirect guidance to the school system by designing studies to help the school system move forward.

Dr. Easton expressed his enthusiasm for his new position as IES Director. He informed the Board that he has been spending the past several months learning about IES and developing ideas about going forward, and welcomed the Board's assistance and partnership in implementing the next generation of ideas.

Ms. Norma Garza, Executive Director of NBES, then reviewed highlights of a community-level event in Texas that she attended in July 2009, along with Mr. Juan Sepulveda, director of the White House Initiative on Educational Excellence for Hispanic Americans. Ms. Garza said she was invited to represent the Board, as part of an effort to include various offices within the U.S. Department of Education (ED). Ms. Garza reported that events were well attended and that media coverage was greater than expected. There will be ongoing community events in 16 other states and Puerto Rico, totaling 30 communities before October 2009.

Ms. Garza said that the intent of the community conversations was to provide a forum for discussion of (1) how education in Latino communities can be improved, and (2) how to structure the future White House initiative. Information gathered from these events will be used as a foundation for an expected presidential executive order, to be signed by President Barack Obama.

Ms. Garza said that the audience of educators and parents shared their concerns about high remediation in this population and the low percentage of Hispanic students who enroll in college credit courses. Additional topics related to these issues included funding, cultural considerations, and socioeconomic and legal status. Ms. Garza explained that Mr. Sepulveda assumed that funding is a primary concern. He has been inviting audiences to explore the issue at a deeper level by considering methods for attracting and retaining quality teachers, including those from alternative certification programs, as well as ensuring that accountability standards are in place for teachers who enter the field from different teacher preparation programs. Educators at the meeting also expressed some frustration with changing rules related to state standards and test alignment.

Dr. Hanushek then reviewed the following meeting agenda items:

• Initiate free-form preliminary discussions with Dr. Easton about the role of NBES in providing optimum assistance and support to his office.
• Briefly discuss procedures for completing the draft of the congressionally mandated annual NBES report.
• Hear a presentation by Dr. Carmel Martin, ED Assistant Secretary for Planning, Evaluation and Policy Development, about use of the stimulus funds allocated by the American Recovery and Reinvestment Act (ARRA) and, in particular, discuss how IES can glean pertinent information as economic stimulus initiatives are being applied.
• Hold a closed-door session for NBES members to make recommendations to Dr. Easton for new Commissioners to fill the positions of Dr. Phoebe H. Cottingham, Commissioner for the National Center for Education Evaluation and Regional Assistance (NCEE), and Dr. Lynn Okagaki, in her part-time role as Commissioner for the National Center for Special Education (NCSER).

Dr. Hanushek stressed the importance of improving the quality of education in America and IES's significant role in supporting that process, and given that the quality of the nation's workforce and economic vibrancy were at stake. He then briefly reviewed the history of educational research at the federal level, which began with the establishment of the National Institute of Education (NIE) in 1972. The agency's overly ambitious program later ended, due to conflicting political agendas between the White House and Congress. NIE was then "morphed" into the Office of Educational Research and Improvement (OERI), which remained intact until 2002. OERI was the inspiration for a National Research Council study on the role and application of science in education.

Dramatic change ensued in 2002 with the creation of IES and NBES under the Education Sciences Reform Act (ESRA) of 2002. Dr. Grover (Russ) Whitehurst was appointed as the first director of IES. NBES appointments have traditionally suffered from lag time and lapses between administrations, which accounts for the fact that only 6 of the potential 15 required Board members have been appointed to date.

Dr. Hanushek explained that IES's founding marked a new era in applying education research to policy. The agency began attracting large numbers of researchers with little previous experience in education, but who began producing a respectable body of nonpartisan education research. An effective peer review process was developed, which introduced rigorous methodologies and initiatives such as the What Works Clearinghouse (WWC) for disseminating research.

Dr. Hanushek then clarified the overall mission of the Board to (1) develop valid, reliable, and relevant research for future policymaking; (2) establish a balance between short- and long-term research objectives; (3) determine how well neutral science interacts with less neutral policies; and (4) consider approaches to expanding capacity and "evidence-based decisionmaking." Although evidence-based decisionmaking is now common parlance, the meaning of the phrase is not always clear; IES can contribute to the discussion by developing policies and programs to support this process. Dr. Hanushek asked participants to consider how the 15-member Board, comprised half of researchers and half of practitioners, can most effectively support Dr. Easton and contribute to the vision of the ESRA.

He then commented that although the Board must respond to particular operational requirements, members have significant leverage in addressing other priorities, including making appointments and advising the agency more generally. He mentioned the 5-year report that has been distributed that recounts IES's success and information about the Board's activities and recommendations. He then opened the floor for discussion of IES's successes and failures, and clarified that ultimately the Board's recommendations will be made public to guide Congress and other relevant policymaking entities.

1:45 p.m.–2:45 p.m. Overview of NBES Interaction With IES

Dr. Sally Shaywitz initiated the discussion by agreeing with Dr. Hanushek's framing of IES and NBES objectives and affirming IES's important contribution to the field of research evaluation and review. She added that she was encouraged by the development of reliable and valid measures.

Determining Components of Valid Evidence

She reflected, however, on the importance of determining what constitutes valid evidence, given that research is a living entity that is constantly in flux. Reports based on small studies have been scaled up to yield discrepant results that are not always converged from larger independent scaled-up studies. Larger studies may be conducted that earn IES support, but they tend not to be published or translated.

Dr. Shaywitz said that policy decisions are made based on research that, once scaled, does not hold up, and emphasized the need for mechanisms to align contradictory data sets of large and smaller research studies, particularly when strongly held beliefs about core findings cannot be supported.

She stressed that although the WWC represents a major step forward, the WWC does not necessarily review results of small-scale studies in comparison with the scaled-up studies. Thus, IES can play a strong role in determining which studies to disseminate, ensuring that the studies are then published and disseminated, and that they are applicable in scaled-up academic environments.

Mr. Jonathan Baron, NBES Vice Chair, responded that part of the motive behind establishing IES, as well as conditions that drive federal education legislation in the past 10 years and more, is that respected measures of long-term trends, such as the National Assessment of Educational Progress (NAEP), indicate that the United States has made only limited progress in raising reading, mathematics, and science achievement in the past 30 years. This is also true as it relates to closing the gap between minority and white students in the past 20 years. Although a number of good ideas and interventions have been proposed, tried, and backed, conclusive, scientifically valid mechanisms were not available prior to establishing IES to determine effective interventions. Thus, among the many transient interventions—such as ability grouping, effective schools, charter and magnet schools, school vouchers, bilingual education, standards and accountability, teacher performance pay, and others—very few have been rigorously evaluated.

Mr. Baron commented that a striking outcome of large IES evaluations is that a significant number of programs that appeared promising, based on preliminary studies, have been demonstrated to be ineffective when evaluated in more definitive studies, or to yield only weak effects. A similar pattern occurs in other fields, such as medicine, where rigorous evaluations are common. While null findings are important, closer collaboration between researchers and practitioners might yield a more promising group of classroom interventions for IES to evaluate, and potentially increase IES's success rate in building a body of proven-effective interventions that can be replicated and scaled up. The preponderance of null findings warrants course correction and closer collaboration between researchers and practitioners to identify the most promising candidate interventions that merit testing in IES evaluations.

Dr. Shaywitz agreed, noting that researchers have a vested interest in what constitutes their life's work, and that IES can serve as an honest broker and a neutral party in this environment. She also agreed with Dr. David Geary's comment that null findings require analysis, and emphasized it as the entire rationale for IES establishing a review process for making research determinations. She added that possibly the focus should shift from further research, to examining many core and strong beliefs that shape education research, and to build a more reliable knowledge base.

Mr. Baron also underscored IES's unique role within ED in providing a rigorous peer review process that is independent of ED program offices, and commented on congressional support of IES as the lead agency for evaluation within ED.

Dissemination of IES Research

The topic of the marketing of IES-sponsored research studies was raised by Mr. Philip Handy, who stressed the importance of outreach and dissemination of research findings and achievements—in addition to quality, relevance, and scalability of research results—which potentially could be supported by a marketing department within IES. Dr. Easton responded that Ms. Tracy Dell'Angela, IES Director of Outreach and Communications, was hired, in part, with this objective in mind.

Dr. Carol D'Amico suggested that the word "evidence" has now thoroughly permeated educators' vocabulary but may be diluted through over-use; the term "translation" may be more apt. She encouraged further discussion centered on improving communication, marketing, and messaging related to IES research outcomes and products.

Dr. Hanushek raised the issue of the possible perception of self-interest among IES personnel in promoting particular policies and programs. Mr. Baron suggested framing the issue in terms of dissemination rather than "marketing" of scientifically based research and professional development, and that the agency has garnered congressional support for promoting programs that have been successful at the national level. He suggested that as the WWC becomes more transparent and user-friendly, it has the potential to contribute more broadly to dissemination efforts.

Dr. Shaywitz underscored the critical need for access, particularly among school principals, to channels of reliable information. Dr. Geary also raised the need for marketing of null effects, without crippling teachers who need this information in the classroom.

Dr. Stuart Kerachsky, Acting Commissioner of the National Center for Education Statistics (NCES) commented on differences between NCES—a statistical agency—and NCEE, citing the inbuilt reliability of statistical agencies. NCEE is a newer type of entity within ED, and not typical in government. Although both agencies have established credibility, IES may promote results of a study that both past, present, and future political administrations may simultaneously announce as ineffective. Maintaining the balance between the IES mission to promote quality and scientific accuracy can be challenging, given political review of IES products and the need to keep in step with shifting political agendas. The agency has been insulated from this process to some extent, but the goal going forward is to protect IES's integrity while speaking with a larger voice.

Mr. Hanushek said that the Congressional Budget Office (CBO) has issued various reports that have made policymakers uncomfortable, in part, by suggesting ineffective allocation of funds. However, from its inception, the CBO determined that it would refrain from making policy recommendations, instead, issuing public documents that suggested benefit/cost ratios of one program versus another, without making final recommendations. The CBO was thus able to insulate its research from the political process. Dr. Hanushek said that in his view, IES's role is to sponsor and evaluate educational research, not necessarily to advise the administration on legislation or program implementation.

Mr. Baron responded that there are other areas of social policy that have offered definitive models of evidence-based programs—nurse visitation for low-income families and teen pregnancy prevention. While IES has developed greater rigor with regard to evaluations and produced valuable results, it needs to find at least a few interventions that are proven highly effective in scientifically-valid evaluations in order to ensure the institute's future viability. The critical question to consider is the choice of interventions for evaluation, and determining those that are the most promising.

Collaboration Between Researchers and Practitioners

Mr. Baron then cited Welfare to Work as an example of a program with a long history of large, rigorous evaluations and a body of proven research interventions. The evaluations found that the programs with the largest effects were mostly developed by innovative practitioners, primarily county welfare officials, rather than researchers. In addition, the Promise Academy, a charter school in Harlem, was also developed by practitioners. A randomized evaluation was conducted that yielded significant effects over a 3- to 4-year follow-up period. He suggested that practitioner-generated programs such as Promise Academy or Teach for America might be good candidates for IES evaluations so as to hopefully move them from the promising to the proven category. Greater collaboration between researchers and practitioners could be a potential source of good candidates for IES evaluations.

In response to Dr. Hanushek's concern about the reliability of replicating program success across a wide variety of sites, Mr. Baron mentioned Career Academies as an example of a small learning environment for students at risk of dropping out. A large, multisite evaluation study that spanned an 11- to 12-year follow-up period produced sizable increases in earnings, which is a main goal of career education.

Dr. Easton suggested that a greater focus on capacity-building rather than primarily on intervention may be warranted when considering the multiple factors determining children's academic achievement. The regimen of interventions should be considered—or the more comprehensive reality of the school as a user of information and data. How programs are selected and implemented should be considered. Dr. Hanushek commented on the challenge of adapting this perspective to standard research paradigms. Dr. Easton responded that the government is increasingly focused on data usage, which could be further supported by IES.

A general discussion ensued about the formal role of IES as "neutral bystander" vis a vis congressional mandating of research directives, as well as the legitimacy and soundness of IES-sponsored study evaluations that had demonstrated initial promise (e.g., the Early Math curricula) in spite of yielding null effects in the field. Although IES is not in a position to endorse research studies, the agency can be more strategic about how studies are disseminated when found to be applicable in larger evaluations. Commenting on scientifically based reading studies, Dr. Okagaki suggested that null effects may also be a function of the state of current practice.

Mr. Baron underscored the value of null findings, but said that IES also need to find a few interventions that are very effective when rigorously evaluated.

Dr. Handy stressed the importance of continued discussion about data quality, consistency, and scalability, and commented on NAEP's value as a tool for evaluating these parameters throughout the universe of variables that IES studies evaluate, as well as current projects aimed at compiling databases of longitudinal data. IES should also advocate that barriers be broken down between K–12, universities, and community colleges, which will level the playing field, and increase the effectiveness and marketability of IES research studies. Dr. Hanushek agreed that although verging on potentially political considerations, IES could frame recommendations on data quality parameters in the same way the CBO does when making budgetary recommendations.

2:45 p.m.–3:30 p.m. Discussion of NBES Annual Report

Dr. Hanushek introduced the topic of the draft of the NBES Annual Report, which he framed as a factual statement of Board activities over the past year, rather than an evaluation in the mode of the 5-year report on IES that evaluated the agency from its inception. He proposed that rather than reading and commenting on the draft at the meeting, a process would be developed for getting Board approval of an annual report once members have the opportunity to determine whether they are satisfied with the draft report.

Specifically, he suggested setting a deadline approximately 3 weeks from the meeting date for receiving e-mailed comments from Board members to be framed as (1) a set of brief comments to incorporate without discussion; (2) another set of e-mailed responses addressing larger issues; and (3) issues for his approval. For large issues, either a round of broader e-mailed discussions or preannounced phone interviews was proposed. Ms. Garza will forward an announcement.

The Board agreed to adopt Dr. Hanushek's proposal.

Mr. Baron asked Dr. Easton to share his perceptions of the level and type of interaction between IES and various program offices, particularly with regard to collaborating on carrying out evaluations. Dr. Easton responded that the leadership within ED is intent on learning as much as possible with regard to spending initiatives and that IES is "at the table."

3:30 p.m.–4:15 p.m. Review of the American Recovery and Reinvestment Act (ARRA)

Dr. Hanushek introduced Ms. Martin to the appointed NBES members, ex officio members, Bureau of Labor Statistics representatives, and IES commissioners at the table. He said that topics of primary interest included the following:

• Determining plans and priorities for stimulus funding and the ARRA, and
• Discussing how the Board can learn from allocation of Race to the Top funding for future use of regular funding.

Ms. Martin briefly commented on her 10 years of experience working for the U.S. Senate, and said that IES is widely supported on Capitol Hill. She explained that education is a top priority for the new Obama Administration, which is focused on using data and research to identify best practices in education and help states and localities in implementing them.

Ms. Martin explained that under ARRA 2009, ED will receive close to $100 billion (triple the discretionary budget) to (1) help create stability in the economy in the short term by saving and creating jobs; (2) avert catastrophic funding cut-backs and their impact on students; and (3) improve long-term educational outcomes, with a view toward improving economic viability and competitiveness on a global level. The priority is on funding comprehensive programs targeted to key transition points that will be "game-changing" in furthering the interests of all students, not for only one segment of the education continuum (e.g., early learning, elementary, postsecondary).

Main topics covered in Ms. Martin's PowerPoint presentation included a series of slides addressing the following policy, program, and funding priorities of the administration and ED:

• Implementing early learning initiatives and key K–12 reforms;
• Designing coherent systems of standards and assessments;
• Dramatically increasing the number of effective educators and changing the character of the profession toward greater focus on continued improvement;
• Improving data system transparency, accountability, performance management, and innovation;
• Turning around low-performing schools; and
• Allocating ARRA Formula, AARA Competitive Grants, and State Fiscal Stabilization Fund (SFSF)/Title I/Individuals with Disabilities Education Act funding.

Dr. Martin also reviewed SFSF Phase II proposed reporting requirements as well as SFSF metrics related to human capital, data, standards and assessments, and struggling schools.

She then provided an overview of a number of funding initiatives including:

• Race to the Top—a $4.35 billion competitive grant intended to encourage and reward states that make dramatic education reforms, particularly in the areas of human capital, standards and assessments, data systems, and school turnaround, as well as preliminary timelines for application for grant funds.
• Innovation Fund—provides $650 million in grant funding available to local educational agencies (LEAs) and their nonprofit partners to scale up and support proven reforms and strategies.
• School Improvement Grants—allocates more than $500,000 to the lowest performing schools with significant resources.
• Teacher Incentive Fund grant—distributes $200 million in ARRA funds to districts and states that develop innovative performance-based compensation systems; $487.3 million is requested for fiscal year 2010.
• State Longitudinal Data Systems—$250 million in ARRA funds to support states in working with state agencies and LEAs to establish seamless data systems.

Dr. Hanushek said that the Board has formally resolved that states that accept Teacher Incentive Fund grant money precommit to participating in random evaluations, and that states that accept money to improve data systems agree to make data available to researchers, under appropriate conditions.

Ms. Martin responded that Family Educational Rights and Privacy Act regulations regarding privacy issues may prohibit the sharing of data, but that the Secretary's office would make an announcement after these parameters have been determined. She again emphasized the Obama Administration's commitment to getting meaningful information into the hands of users and creating a culture of continuous improvement, beyond simply addressing the need for enhanced information technology and greater accountability. She stressed the importance of gathering good data in achieving these objectives and confirmed that ED would be working closely with IES in this regard.

Ms. Martin then reviewed several postsecondary and community college initiatives, including support for the creation of a more user-friendly Federal Student Aid application form and efforts geared toward modernizing community college facilities and online learning environments.

She ended her presentation with an explanation of future objectives regarding Elementary and Secondary Education Act and the Workforce Investment Act reauthorization. In the first case, the effort will be to ensure effective allocation of funding streams, greater accountability, and improved evaluation systems and technical assistance components. In the second case, the effort will be to better coordinate programs for those eligible for vocational rehabilitation and foster improvements in coordinating these programs; to revitalize community colleges as engines for economic development; and to improve postsecondary data systems.

Ms. Martin said that an important component going forward will be obtaining quality data from grantees to evaluate funding effectiveness, continuing collaboration with IES in conducting program evaluations, and determining how data reporting and evaluation strategies can be universally built into all ED programs.

Dr. Geary raised the issue of whether there are requirements that states collect the same core information on each children, to allow to across state comparisons and better tracking of children as they move from one state to the next. He expressed concern that states are only encouraged, and not required to do so with these funds.

4:15 p.m.–4:30 p.m. Open Board Discussion and Next Steps

Ms. Martin responded in the affirmative to Mr. Handy's question about whether charter schools will be held to higher accountability under performance contracts, given their greater degree of flexibility.

Ensuring Consistency of Data Elements

Ms. Martin also confirmed that recipients of Race to the Top and Teacher Incentive funds would also be required to precommit to participating in follow-up evaluations.

With regard to reporting requirements, she clarified that the ARRA requires grantees to report all information about the use of funds. In addition, an effort is underway through the Office of Management and Budget to develop common terms and definitions for that reporting requirement. Title I stipulates that states must report per-people expenditures. Greater consistency within SFSF and Race to the Top is also an objective, although states may diverge to some extent in the use of definitions and calculations of data elements.

The NCES Data Handbook (Handbook) will also be used to ensure common definitions of terminology in the State Longitudinal Data System grant program that IES is implementing, as well as standards for data sharing. There will also be greater state-to-state interoperability.

Ms. Martin referred to the ED Facts data collection program that had been implemented in ED prior to the current administration to develop a common core of aggregate data from state to state. The chief state school officers and state higher education authorities are also in discussions to refine and fill out definitions that are not currently accounted for in the Handbook.

Ms. Martin responded to a question from Dr. Shaywitz about standards for measuring teacher performance by explaining that although this is a matter for states and localities to resolve, ED requires that student outcome data are prioritized in determining metrics related to teacher effectiveness, in addition to peer-review processes and observations by supervisors.

Mr. Baron raised the point that numerous examples can be cited of misleading outcome metrics that do not accurately reflect a program's success or failure; instead, emphasis should be placed on grantee participation in rigorous, preferably randomized, evaluations in appropriate program areas. Ms. Martin agreed and said that ED Secretary Arne Duncan will be consulting with Dr. Easton and others at IES on this topic in the future.

4:30 p.m.–5:00 p.m. Board Recommendations for IES Commissioner Appointments

Dr. Hanushek announced that the next NBES meeting that is scheduled for September 2009 will likely be rescheduled for October 2009 to allow more time for Dr. Easton to formulate and present goals for the agency over the coming year. He then formally adjourned the meeting at 4:45 p.m. prior to going into an executive session to make recommendations for new center commissioners.

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.