Location:
Institute of Education Sciences Board Room
80 F Street NW
Washington, DC

Board Members Present:
Dr. Eric Hanushek, Chairman
Mr. Jonathan Baron, Vice Chairman
Dr. Deborah Ball
Dr. Adam Gamoran
Dr. David Geary
Mr. Philip Handy
Dr. Bridget Terry Long
Dr. Margaret (Peggy) McLeod
Dr. Sally Shaywitz

Ex Officio Members Present:
Dr. John Q. Easton, Director, Institute of Education Sciences (IES)
Dr. Robert Kominski, U.S. Census Bureau
Dr. Cora Marrett, National Science Foundation (NSF)
Dr. Lynn Okagaki, National Center for Education Research (NCER) Commissioner and Acting National Center for Special Education Research (NCSER) Commissioner
Ms. Dixie Sommers, Delegate Representing Mr. Keith Hall, Bureau of Labor Statistics
Dr. Peggy McCardle, National Institute of Child Health and Human Development (NICHD)
Ms. Val Plisko, for Dr. Stuart Kerachsky, National Center for Education Statistics (NCES) Commissioner
Dr. Rebecca Maynard, National Center for Education Evaluation and Regional Assistance (NCEE) Commissioner

NBES Staff:
Ms. Norma Garza, Executive Director

Designated Federal Official:
Ms. Mary Grace Lucier

Other Attendees
Sue Betka, IES
Ellie McCutcheon, IES
Jerry Sroufe, American Educational Research Association (AERA)
Jinfa Cai, NSF
Janice Earle, NSF
Pat Wilson, NSF
Teresa Duncan, ICF International
Joan McLaughlin, NCSER, IES
Terrie Rust, Einstein Fellow, NSF
Kim Hymes, Council for Exceptional Children
Allison Orechwa, IES
Sarah Spreitzer, Lewis-Burke Associates
Jim Kohlmoos, Knowledge Alliance
Allen Ruby, IES
Jeremy Flattau, National Academy of Sciences
Elizabeth Albro, NCER, IES
Anne Ricciuti, IES
Sarah Hutcheon, Society for Research in Child Development
Sarah Sparks, Education Week
Sheritta Cooper Porter, AFYA, Inc.
Anna Nicotera, National Alliance for Public Charter Schools
Khara Turnbull, IES
Lisa Bridges, IES
Chris Rakes, IES
Tracy Dell'Angela, IES
Matt Devine, IES
Teresa Cahalan, U.S. Department of Education
Jill Constantine, Mathematica Policy Research, Inc.
Scott Cody, Mathematica Policy Research, Inc.
Michael Ponza, Mathematica Policy Research, Inc.
Susan Sanchez, IES
Marsha Silverberg, IES
Phil Gleason, Mathematica Policy Research, Inc.
Brooks Garber, National Alliance for Public Charter Schools
Rita Zota, Office of Management and Budget
Linda Marshall, IES
Karla Lewis, SERVE Center, University of North Carolina at Greensboro
Kris Gutierrez
La Tosha Plavnik, Consortium of Social Science Associations
Kerry Bolger, AERA
Leslie B. Williams, Office of Communications and Outreach, U.S. Department of Education
Steve Fleischman, Education Northwest
Christy Talbot, AERA

8:15 a.m. – 8:30 a.m.: Call to Order, Approval of Agenda, Chair Remarks, and Remarks by the Executive Director
Dr. Eric Hanushek, National Board for Education Sciences (NBES) Chair
Ms. Norma Garza, NBES Executive Director

Dr. Hanushek called the meeting to order and welcomed four new board members. He then asked board members, meeting participants, and attendees to introduce themselves. Following these introductions, Dr. Hanushek summarized the day's schedule and noted that the first agenda item would involve review and discussion of the priorities of the Institute of Education Sciences (IES). He introduced Dr. Easton to begin the discussion.

8:30 a.m. – 9:15 a.m.: Priorities of the Institute of Education Sciences
Dr. John Q. Easton, IES Director

Dr. Easton noted that the Education Sciences Reform Act of 2002 requires that IES establish research priorities and that the IES Board approve or disapprove the priorities. He said the current priorities had been posted to the Federal Register on July 7, 2010, and that 34 comments had been received. He summarized the priorities, noting changes that include new emphasis on developing partnerships to build capacity and expanding research beyond school-age children to include outcomes for infants and education attainment for adults. He noted that the priorities also call for a sharper focus on higher order thinking; better methods to recruit, train, and retain teachers; and strategies to improve educational equity.

9:15 a.m. – 10:00 a.m. IES Priorities—Board Discussion

Mr. Baron recommended approval of the priorities and said increased partnerships between researchers, policymakers, and practitioners would increase the relevance of IES research. He noted that the Board has not been effectively utilized as a source of advice to the Director, and he encouraged the Board to expand its activity in this regard.

Dr. Gamoran also recommended approval of the IES priorities. He said IES has been successful in developing research designs and in emphasizing that judgments about cause and effect in education must be based on research and statistical methods. He suggested that there is a need for clarification on how the priorities would influence the way research is organized and funded. He noted that one of the public comments suggested an expansion of grant support for state longitudinal data systems and encouraged exploration of that linkage.

Dr. Terry Long approved the priorities, in general. She praised the increased emphasis on research that looks beyond school age to include aspects of learning in preschool age children and postsecondary vocational and adult education. She endorsed support for state longitudinal data systems and said there should be increased emphasis on rigorous causal research. She also suggested expanding investigations to include research on enhancing skills of instructors such as guidance counselors, not only teachers, and also encouraged more emphasis on gender effects, particularly adverse outcomes for boys.

Dr. Ball said IES should focus on instruction in a broad sense that goes beyond classroom teaching. She encouraged development of improved methods to measure "fine-grained" aspects of instruction to understand how programs affect outcomes. This includes how programs are implemented at the district and school levels.

Dr. Geary noted that increased emphasis on partnerships should be carefully designed to ensure that results are generalizable to a broad context, not just a school or local context. He also noted that emphasis on innovative approaches should not reduce research on ways to make steady and incremental progress based on methods already shown to be effective.

Dr. McLeod said that IES should make efforts to move from research that now seems to focus on specific small variables, and try to develop approaches that merge research into a whole that makes sense to practitioners. She said there is a greater need for useful research at the district level than at the state level, and recommended improving the way that research findings can be made available to practicing teachers, as well as to teachers in training.

Dr. McCardle said IES might want to avoid developing research agendas that overlap unnecessarily with existing basic science research into areas such as cognition and neuroscience. She said there is a need to expand students' exposure beyond traditional "hard" science to include behavioral and social sciences.

Mr. Handy recommended significant expansion of dissemination efforts to "market" research findings. He said the Board and IES were not fully meeting their responsibilities if research findings are not broadly communicated and well understood.

Dr. Hanushek said there is a need for studies that explain how best to make research findings useful and relevant. He noted that some research findings, though well-established, are not put into practice, and encouraged an increase in communication efforts to improve the way practitioners and policymakers make use of knowledge in trying to make changes in schools.

Dr. Shaywitz noted that there is little effort made to communicate findings about practices or methods that have been shown not to work.

The Eunice Kennedy Shriver National Institute of Child Health and Human Development

Dr. McCardle briefly described the structure and mission of The Eunice Kennedy Shriver NICHD and summarized recent developments relevant to IES. The new NICHD Director, Dr. Alan Guttmacher, is a pediatrician, geneticist, and worked as a middle school teacher. This experience led to his interest in learning disabilities. NICHD branches with particular relevance to IES include the Demographic and Behavioral Sciences Branch, which conducts population-level studies; the Child Development and Behavior Branch; and the Intellectual and Developmental Disabilities Branch, formerly named Mental Retardation and Developmental Disabilities Branch.

NICHD is currently engaged in a Visioning activity which will focus on nine theme areas, including development, plasticity, cognition, behavior, and developmental origins of health and disease. Much of the activity within these areas is relevant to IES.

The Opportunity Network (OppNet) is a National Institutes of Health (NIH)-wide activity that pools resources to focus on basic behavioral and social science research. There are currently nine OppNet requests for applications (RFAs) and there will be more issued in future years.

The Child Development and Behavior Branch of the NICHD held workshops on "Executive Function in Preschool Children," "Cognition, Brain Function, and Learning in Incarcerated Youth," and "Defining the Intersection of Reading and Math Disability." The Institute will develop and post summary documents from the workshops. In summer 2011, NICHD will hold a Biennial Research Training Institute on Applied Research in Child and Adolescent Development. This event is a 5-day training institute for young faculty who have not been formally trained in applied research.

NICHD has approved concepts for possible research solicitations for the Learning Disabilities Research Centers and new Learning Disability Innovation Hubs.

U.S. Census Bureau

Dr. Kominski briefly discussed changes to Census Bureau polling and research that will be relevant to IES activity. The American Community Survey (ACS) will allow more rapid analysis of data for every geopolitical unit with populations of 65,000 or greater, including all the states, all the big cities, all major metropolitan areas, and many large school districts. ACS data are available on the Census Bureau website. A question added to the ACS will elicit information on the field of study for persons with a bachelor's degree or higher. This information helps facilitate surveys conducted by the NSF and NCES on the science and technology workforce, and will be updated annually. The first 5-year data product from ACS will soon be available and will include information on many school districts. The small-scale geographic estimates will be used for funds allocation, and the National Academy of Sciences is now concluding a study that will recommend ways to use ACS data for Title III English Language Learner funds. The Census Bureau and the Bureau of Labor Statistics (BLS) are collaborating on an effort led by NCES to better understand "sub-baccalaureate education," or certifications. These have real labor market value but data about them are not captured in conventional educational surveys.

Bureau of Labor Statistics

Ms. Sommers explained the role of the BLS and described projects relevant to IES. The BLS uses Census data and other survey results to develop data on working conditions, primarily dealing with health and safety, but also what kind of benefits and compensation are available. Recently, BLS began measuring the prevalence of certifications and certificates as a component of information on careers and occupational characteristics. BLS is working with NCES to update a "crosswalk" between occupational classifications and the education program classification, both of which were revised in 2010. This effort will help us understand what happens to workers as a result of formal degree programs, workforce training programs, or secondary and postsecondary career technical programs. The data also will be useful at the state or local level for funding training programs. BLS also is revising education and training categories used in the Employment Projections Program. This effort will provide information such as level of education at entry, work experience in another occupation, licensing requirements, or whether there is typical on-the-job training for roughly 750 occupations. BLS will publish proposed revisions in the Federal Register.

Dr. Hanushek noted that the proposed categories discussed by Ms. Sommers have a single category that includes high school diplomas and GEDs. He said the two are not equivalent, particularly from a labor market standpoint, and asked how that mischaracterization could be corrected.

Ms. Sommers noted that BLS adapted U.S. Census Bureau conventions.

Dr. Kominski said that the U.S. Census Bureau is working to refine the categories to accommodate the important distinction between a GED and high school diploma.

Dr. Ball noted that teachers are the largest single labor group in the country and commended BLS for compiling valuable information on teaching and education.

National Science Foundation

Dr. Marrett described the structure of NSF, noting that one of its seven directorates is the Directorate for Education and Human Resources, which is responsible for evaluating education across all fields of science. Each of the other six directorates also conducts education, learning, and workforce activities. NSF has issued a solicitation for research proposals on transforming Science, Technology, Engineering, and Mathematics (STEM) education. The solicitation asks interdisciplinary teams to propose efforts to investigate innovative strategies already being used, as well as proposals to design prototypes for transforming education. NSF provides support to NCES for the High School Longitudinal Study. Dr. Marrett described a recent administration-level development related to the integration of education and STEM. This involves the designation of ED, NSF, and the National Institutes of Health to take the lead in this effort.

10:00 a.m. – 10:15 a.m.: Break

10:15 a.m. – 11:00 a.m.: Ex Officio Member Agency Overview

11:00 a.m. – 12:00 p.m.: What Works Clearinghouse: Overview
Jill Constantine, Deputy Director, What Works Clearinghouse Associate Director of Research and Education Area Leader, Mathematica Policy Research, Inc.
Scott Cody, Deputy Director, What Works Clearinghouse Associate Director of Research, Mathematica Policy Research, Inc.

Dr. Maynard said the What Works Clearinghouse (WWC) was developed to synthesize and disseminate research findings. To date, the WWC has released about 150 intervention reports, roughly 50 quick reviews, and practice guides that translate research evidence into language and recommendations that can be used by the field. She introduced members of the project who would describe WWC activities.

Dr. Constantine described two new WWC practice guides: "Reading Comprehension for Students in K through 3" and "Teaching Fractions in Fourth and Fifth Grade." She provided a brief description of the process involved in designing a research standard for causal inference studies less tightly controlled than randomized control trials and noted that it is necessary to develop a standard that is replicable and reliable. The development requires protocols so the standard can be applied to any study with the same design. The WWC develops study review guides for use by investigators to ensure different parts of the study meet each standard imbedded within the overall standards.

Regression Discontinuity Design

This study design involves treatment and control groups that are formed by design rather than random assignment. An example is a study of the effect of an intervention on students who perform below a certain level on an assessment. Students below the cutoff are the treatment group and those above it are the control group. The standards criteria for a Regression Discontinuity Design are integrity of the assignment variable, continuity of the relationship between the forcing variable and the outcome variable, functional form and bandwidth, and attrition.

Single Case Design

Mr. Cody explained that a single case design represents one of the only methods of studying interventions for low incidence disorders in special education research. In single case studies, the case—an individual student, a small group of students, or a classroom—is the unit of intervention and the unit of analysis. The study looks at the same case under two different situations; it provides its own control. Standards must be flexible enough to cover different designs but strong enough to ensure the same level of internal validity. Single case research does not allow reliable effect size estimates. One way to estimate the magnitude of the impact is to combine across studies and report an average typical impact of an intervention. An example of a single case design is the observation of hitting other students at a high frequency in a classroom. The researchers would take repeated observations at baseline, apply the treatment, remove the treatment, and make repeated observations to document that the frequency of hitting is reduced by the intervention. Another type of single case study might compare the effect of an intervention across time in each of four classrooms. If the outcome increases in each classroom after the intervention is applied, there is strong evidence of a causal relationship. Single case design standards require that the outcome variable be measured systematically, so it has to be measured over time across all of these different phases of the study. Inter-rater reliability must be assessed. A third standard is that the design must include at least three attempts to demonstrate an effect. The final standard requires at least five observations within each phase for a study to meet WWC standards. There is not a large enough body of research to allow generalization from single case design studies. This will require at least five studies that meet WWC standards, done by three or more research teams in three different geographic locations, with a combined total of at least 20 cases.

Discussion

Mr. Baron asked how it is possible to be confident that these methods can produce valid estimates approaching or equaling those of a well-conducted randomized trial. He noted that researchers who have evaluated quasi-experimental methods have found that many have not worked.

Mr. Cody said the standards developed for internal validity create confidence that studies have causal validity.

Dr. Maynard noted that the WWC is developing additional methodological work to assess the study designs.

Dr. Ball asked how outcomes and other variables are being measured.

Quick Reviews

Mr. Ponza explained that Quick Reviews are designed to provide objective assessments of recent high-profile education studies on the effectiveness of interventions or programs. A quick review team identifies potential studies and recommends them to IES. Two certified reviewers evaluate the study. After the study is approved, a draft is produced and reviewed within the WWC before being sent to IES for external peer review. The final publication is a one-to-two page posting on the website that includes a section describing the intervention, the population, study design, and analyses. The review also describes key features of the intervention and rates the intervention against WWC evidence standards to come up with a rating. The WWC has released about 50 reviews. Changes to the process may include increased opportunity for interaction with the study author, increased use of outside content expertise to help us define the outcome domains and specific measures, and expansion of the reports to include multiple analyses, outcomes, and subgroups. The WWC also is considering follow-up studies.

Numerous Board members discussed the relative popularity of quick reviews, concerns about selection of material, and the importance of offering authors an opportunity to respond to or expand on review criticism.

12:00 p.m. – 1:00 p.m. The Evaluation of Charter School Impacts
Phil Gleason, Senior Fellow, Project Director for the Evaluation of the Impacts of Charter School Strategies, Mathematica Policy Research, Inc.

Dr. Gleason reported on a study on charter schools using a lottery-based design.

There were three main criteria for school inclusion. The study involved middle schools with at least 2 years of experience that were oversubscribed and therefore held an admissions lottery. The treatment group was students who were offered admission. The control group was those who didn't get offered admission and attended a different school (a public school for 78 percent of these students). The study involved a sample of about 2,300 students and 36 charter middle schools across 15 states. The student population was more likely to be white than African American (53 percent versus 38 percent). The proportion of Hispanics was similar across the two sets of schools. The study found no significant treatment/control differences on measures of student achievement and proficiency; other measures include academic progress, homework completion, or measures of behavior both within and outside of school. There were significant impacts on measures of student and parent attitudes about their school. Evaluation across charter schools showed that schools with a larger percentage of students who qualified for free- and reduced-price meals showed a positive and statistically significant achievement in math scores, and better but not significant improvement in reading. Sites that served more disadvantaged students had more positive effects.

During extensive discussion, Dr. Gleason noted that the charter schools differed from control schools in the extent to which they could make autonomous decisions. He also noted that some apparent differences among charter schools may be due to school size. Larger schools, which also tended to be urban schools, tended to have better outcomes.

Board members discussed the importance of keeping in mind that many possible impacts are difficult to assess, for example, lottery losers have very different experiences and therefore different outcomes. Comments also suggested additional research to identify factors associated with different outcomes within the charter school group.

[Whereupon, at 12:56 p.m., the NBES Board Meeting recessed, to reconvene at 3:15 p.m., this same day.]

1:00 p.m. – 2:00 p.m.: Lunch Break

2:00 p.m. – 2:30 p.m.: Transport NBES Members to U.S. Department of Education— LBJ Building

2:30 p.m. – 3:00 p.m.: Swearing in of New NBES Members
U.S. Department of Education Secretary Arne Duncan

3:00 p.m. – 3:15 p.m. Transport NBES Members to Institute of Education Sciences—Capitol Place

3:15 p.m. – 4:30 p.m.: Update on IES Center Activity
IES Commissioners and Staff:
National Center for Education Statistics
National Center for Education Evaluation and Regional Assistance
National Center for Education Research
National Center for Special Education Research

National Center for Education Statistics

Ms. Plisko described the structure of the NCES, the primary federal agency for collecting and analyzing statistics on education in the United States and in other countries. Its mission includes collecting and disseminating timely, relevant, and useful information to practitioners, policymakers, parents, and the public. NCES's four major divisions are the Assessment Division; the Postsecondary and Adult and Career Education Division; the Early Childhood, International and Crosscutting Studies Division; and the Elementary/Secondary and Libraries Studies Division. She discussed career and adult education guidance in privacy and confidentiality concerns and the ED data dashboard. Board member discussion focused on complications associated with respecting privacy and obtaining data necessary to conduct meaningful research. This discussion also involved concerns that regulations and compliance vary from state to state. Members and participants also discussed studies comparing U.S. and international educational data.

National Center for Education Evaluation and Regional Assistance

Dr. Maynard explained that the mission of the NCEE includes large-scale evaluations in support of ED initiatives, the WWC, the Regional Educational Laboratories, and library services and central dissemination activities. She said there have been substantial improvements in conducting evaluations over the past 6 years, and these improvements are due in part to innovations in using new data systems. Dr. Maynard said the NCEE will continue scientific reviews of evidence through the WWC, and will work to include similar evaluation products in other areas. She noted the NCEE is committed to improving methods to disseminate and translate information. She also noted that the NCEE agenda in years ahead will include work, stimulated by the American Recovery and Reinvestment Act, on low-performing schools, teacher quality, and students with disabilities. She said the NCEE will build on the accomplishments of the WWC and strengthen the capacity of the Regional Educational Laboratories.

Mr. Baron said he was pleased with the speed and the quality of NCEE evaluations. He said the evaluation activity would benefit from a method to identify better candidate interventions to study.

Dr. Maynard noted that focus on programs and interventions for evaluation may be influenced by factors such as where ED, Congress, or some entity is investing money and energy.

Dr. Terry Long asked if NCEE could provide information about website visits in an effort to assess trends, needs, and the success of information dissemination.

Dr. Maynard said NCEE was investigating this and other methods of "product testing" and audience evaluation.

National Center for Education Research

Dr. Okagaki provided an overview of research programs in the NCER and NCSER. NCER research programs focus on school readiness, learning, and achievement in core academic content areas. Within these areas, research looks at factors such as processes of learning, instruction, practices, and organization and management. Exploratory research examines the relations between education outcomes and factors such as instructional or management practices and policies. NCER also supports the development of education interventions. As an example, she cited computer programs that recognize and use natural conversational speech to teach physics. NCER supports development and validation of screening instruments, formative assessments, assessments of teachers, and assessments of principals. Implementation research is fully integrated into NCER research programs.

National Center for Special Education Research

One new NCSER project is the National Center on Improving the Understanding of Fractions Among Students with Mathematical Learning Difficulties, which is designed to improve understanding of problems associated with fractions.

Dr. Hanushek noted that this seems to be a very specialized target for a national center.

Dr. Ball said fractions represent an important underlying mathematics process, and asked why the center focused on students with learning disabilities rather than all students.

Dr. Okagaki noted that the center is funded through NCSER, but will have an impact on understanding the instruction of fractions more generally. She said that rigorous single-case experimental research is very important for the special education research community. NCSER has funded a study of statistical methods for analyzing data from single-case experimental designs and held a technical working group meeting to discuss single-case experimental design research. This effort is an extension of NCSER's research portfolio on low-incidence disabilities, which includes research on children who are deaf or hard of hearing, have visual impairments, or have significant intellectual disabilities.

Dr. Gamoran asked if NCER had any programs to help researchers learn to submit good proposals.

Dr. Okagaki said an important role for NCER and NCSER program officers is to provide technical assistance that leads to well-designed studies. The Centers offer training that includes methodological workshops.

4:30 p.m. – 5:00 p.m.: Wrap-Up

Final Discussion

Dr. Hanushek asked Dr. Easton how to proceed on consideration of the IES priorities document.

Dr. Easton said he would solicit advice from IES commissioners about next steps and invited Board members to provide additional comments and suggestions.

Various Board members supported the idea of circulating a revised document that could be voted on at the next meeting.

Dr. Hanushek noted that Ms. Garza's 3-year term as Executive Director was ending and thanked her for her service.

5:00 p.m.: Adjournment

[Whereupon, at 4:50 p.m., the meeting of the National Board for Education Sciences was adjourned.]

The National Board for Education Sciences is a Federal advisory committee chartered by Congress, operating under the Federal Advisory Committee Act (FACA); 5 U.S.C., App. 2). The Board provides advice to the Director on the policies of the Institute of Education Sciences. The findings and recommendations of the Board do not represent the views of the Agency, and this document does not represent information approved or disseminated by the Department of Education.