Statistical and Research Methodology in Education CFDA 84.305D

Program Officer:
Dr. Allen Ruby
Allen.Ruby@ed.gov
(202) 219-1591

Purpose

Through the grant program on Statistical and Research Methodology in Education, the Institute supports research to advance education research methodologies and statistical analyses. The long-term outcome of this research program will be a wide range of methodological and statistical tools that will better enable education scientists to conduct rigorous education research.

Background

The mission of the Institute, broadly speaking, is to provide rigorous evidence on which to ground education practice and policy and to encourage its use. Critical to achieving this mission is providing education scientists with the tools they need to conduct rigorous applied research. To that end, the Institute invites applications to develop new approaches, to extend and improve existing methods and to create other tools that would enhance the ability of researchers to conduct the types of research that the Institute funds. For information on the types of research that the Institute funds, please see the Institute's research funding announcements at http://ies.ed.gov/funding. In this section, the Institute provides a few examples of areas in which research is needed to improve the statistical and methodological tools available to education scientists. However, the Institute is interested in a wide range of topics, and applicants are not limited to the examples described below.

The Institute encourages applications to develop or investigate techniques to increase the generalizability of studies. Oftentimes, evaluations of education interventions are conducted on samples that may not be truly representative of larger populations of policy interest. In some cases, a convenience sample (e.g., schools willing to participate in a study) may be used. In others cases, random samples may be taken from a small geographical area (e.g., schools within a district), and consequently the results may not generalize to larger geographical areas (e.g., all districts within a state). The Institute is interested in proposals to understand how results from these two types of samples can be generalized to broader populations. Although there has been some work in education on developing weights, based on surveys or other sources of information on the population, to make the estimate of the treatment effect more likely to reflect the effect in the general population, relatively little research has been conducted to address this problem.

The Institute is very interested in applications to identify ways to increase the power of studies to detect effects. Education evaluations can be expensive when schools are the unit of analysis. How can researchers increase statistical power without having to add additional sites? Although some work has examined the use of covariates and blocking to increase power (Bloom, Richburg-Hayes, & Black, 2007; Raudenbush, Martinez, & Spybrook, 2007), more research is needed. The Institute views this as a critical area of need for the advancement of education research. In addition, the Institute encourages applications to develop and refine tools for calculating power in complex multilevel designs. Such work would also include reference tools providing information that would enable researchers to better estimate intra-class correlations across a wide variety of measures relevant to education and special education.

Differential attrition can compromise an experimental design. Researchers need information on the causes or predictors of differential attrition, methods to reduce such attrition, guidelines to determine if such attrition has biased their estimate of the effect of an intervention, alternatives to analyzing the data (such as matched quasi-experimental comparisons) when differential attrition is high, and what data should be collected from the start of the study in case differential attrition forces them to rely alternative analysis.

Under its research grant programs to support evaluation of interventions, the Institute stresses the importance of identifying the fidelity of implementation of the intervention, as well as measurement of what occurs in the comparison condition. Research is needed on both the measurement of fidelity and the integration of fidelity data into the analysis of the intervention's impact.

Estimating treatment effects is a technical issue but interpreting the size of the effect is a judgment. Our ability to understand or provide a context for interpreting the size of an effect is limited. All too often, researchers cite Cohen's (1988) rule of thumb regarding the size of effects. Ideally, effects would be compared to other actual results or to hypothetically desired results, but much more research is needed to create a context for how we can determine if an effect is a substantial improvement or a trivial one. Recent work (Hill, Bloom, Black and Lipsey 2008) showing that the size of annual gains on nationally normed reading and mathematics tests diminishes as students enter middle and high school is an example of this type of project. The Institute strongly encourages applications to develop reference tools that provide information on typical gains across a wide variety of measures relevant to education and special education.

When random assignment is not feasible to evaluate the impact of an intervention, nonexperimental comparison group methods (e.g., instrumental variables, propensity score matching, fixed effects models) are typically employed. The Institute strongly encourages research that examines nonexperimental comparison group methods to determine which methods best reduce selection bias in estimates of the effect and the conditions that are necessary for producing such results. An example of this type of work is a study by Bloom and colleagues (2002) that utilized existing data from a large random assignment study — the National Evaluation of Welfare-to-Work Strategies — to test different approaches. The Institute has restricted use-data files from random assignment studies that could be used to conduct this type of study. Interested applicants should contact the program officer listed at the beginning of this program announcement. Information on obtaining IES restricted-use data licenses is available at http://nces.ed.gov/statprog/rudman/.

In recent years, there has been increasing interest in applying value added methods to a variety of education issues. Results of value added methods, however, are sensitive to choices regarding design and analytic models, and are subject to bias when student assignment to classroom is not random. Under the Methods research program, the Institute accepts proposals to enable the field to gain a better understanding of the strengths and weaknesses of value added methods, how they can be improved, and whether and how they can be applied to personnel and policy decision-making.

Mandated by Congress, the National Assessment of Educational Progress (NAEP) surveys the education achievement of students in the United States, and monitors their progress over time. Widely known as the "Nation's Report Card," NAEP has been collecting data to provide educators and policymakers with valid and meaningful information for more than 30 years. The state-of-the-art psychometric and sampling designs used in NAEP present an analytic challenge for many education researchers. The Institute invites proposals to develop tools or methods for making the analysis and interpretation of NAEP data easier for education leaders and decision makers or to permit advanced analytic techniques to be readily applied to NAEP data. The Institute is also interested in the development of methodological and analytic procedures relevant to NAEP. For example, applicants might propose to test alternatives to some component of the NAEP sampling or psychometric model to test analytic solutions to problems that were previously intractable in the context of NAEP.

The Institute will also accept applications to conduct methodological research that piggy-backs onto an existing study. For example, a researcher might propose to conduct systematic variation of strategies to enhance recruitment and retention of participants, to examine the influence of different consent procedures, or to test alternative data collection procedures.

As a final example, the Institute also solicits applications to improve or extend statistical analyses of single case experimental designs (e.g., alternating treatments, multiple baseline designs). Single case experimental designs pose many analytical challenges, such as violations of assumptions of traditional inferential statistics (e.g., independence between observations). Applicants may propose research that continues exploration of various approaches (e.g., hierarchical linear modeling, nonparametric tests, measurement of effect size) for analyzing results from individual single case studies as well as analyzing aggregated single case design data.

As previously noted, the Institute is interested in a wide range of topics and applicants are not limited to the examples described above.

References

Bloom, H. S., Michalopoulos, C., Hill, C. J., & Lei, Y. (2002). Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? MDRC Working Papers on Research Methodology. Downloaded from http://www.mdrc.org/publications/66/full.pdf on August 26, 2008.

Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation & Policy Analysis, 29 (1), 30–59.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives.

Raudenbush, S. W., Martinez, A., & Spybrook, J. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation & Policy Analysis, 29 (1), 5–29.