Co-Principal Investigators: Perna, Laura; Polanin, Joshua

Purpose: The research team will conduct a meta-analysis to estimate the relationships between different types of financial aid programs and student progress through postsecondary education.

Project Activities: During the first stage of the project, researchers will conduct a systematic review of the research literature on financial aid published between 2002 and 2019, to identify evaluation studies that meet a pre-established set of criteria for inclusion in the meta-analysis. During the second stage, the team will code each identified study for the type of financial aid program evaluated and its key features, the effect sizes associated with the aid program in relation to each of five postsecondary outcomes, and the main design and methodological attributes of the study. During the final stage, researchers will synthesize the findings from prior evaluation studies using advanced meta-analysis modeling techniques.

Products: The findings from this project will provide information to policymakers, researchers, college leaders, and financial aid officers about the effects of specific types of financial aid on students' progress through postsecondary education. The findings will also inform the research field regarding the extent to which research design can influence findings regarding the aid-outcomes relationship. The researchers will produce policy briefs and peer-reviewed publications that describe their findings.

Structured Abstract

Setting: The meta-analysis will include studies of financial aid programs in the United States.

Sample: The meta-analysis will focus on studies examining the effects of financial aid programs for students entering postsecondary education from high school. Studies with adult students must disaggregate the populations by age or school status in order to be included.

Intervention: This project will focus on four groupings of financial aid programs: place-based scholarship programs; broad-based state-sponsored merit-aid programs; need-based aid programs; and hybrid need- and merit-based aid programs.

Research Design and Methods: The study design couples, in sequence, a systematic review and a meta-analysis. Eligible research designs for studies to be included in the meta-analysis sample are randomized controlled trials, quasi-experimental designs, and regression discontinuity designs. To prevent sample selection bias, the design specifies an exhaustive search for references in electronic databases, publication repositories, conference paper archives, reference lists within published articles, and contacts with authors of aid program evaluations. The design specifies the use of clear inclusion/exclusion criteria implemented through a two-stage screening process with screeners using researcher-developed data entry tools that combine and adjudicate codes from at least two screeners per study. Data extraction procedures will proceed according to a pre-approved codebook consistent with What Works Clearinghouse (WWC) procedures for measuring postsecondary outcomes and calculating effect sizes.

Data Analytic Strategy: The research team will begin the data analysis by computing effect sizes for each of the postsecondary outcomes of interest: 1) college enrollment; 2) credit accumulation and persistence; 3) academic achievement, 4) degree attainment; and 5) labor market outcomes. As needed, researchers will adjust effect sizes for small-sample bias using Hedge's g, for selection bias in quasi-experimental studies according to WWC guidance, and for clustering of outcomes in cluster-randomized trials. The team will model program effects for each of the outcome domains, accounting for moderation. To estimate the meta-analytic models, the team will use a robust variance estimation model that facilitates full use of the data and makes it possible to combine multiple effect sizes both within and across studies. The analytic strategy will culminate in average effect sizes for each of the four aid program types within each of the outcome domains.