Co-Principal Investigators: Perna, Laura W.; Polanin, Joshua R.

Purpose: The research team conducted systematic literature review and meta-analysis to estimate the effects of different types of financial aid programs on college student outcomes from initial enrollment to post-college labor market outcomes.

Project Activities: The first stage of the project involved completion of a systematic review of the research literature on financial aid published between January 1, 2002 and January 15, 2020. Through this review the researchers identified evaluation studies that meet a pre-established set of criteria for inclusion in the meta-analysis. During the second stage, the researchers coded each identified study for the type of financial aid program evaluated and its key features, the effect sizes associated with the aid program in relation to each of five postsecondary outcomes, and the main design and methodological attributes of the study. Then, the researchers synthesized  the findings from prior evaluation studies using advanced meta-analysis modeling techniques. Key findings:

• Losing grant aid has negative effects on student outcomes when grant aid is reduced or eliminated. The researchers found that, although effect sizes vary, this general conclusion applies when grant aid is reduced or eliminated from programs that differ in scope (federal and state), eligibility requirements (merit and need), and award amounts (LaSota et al., 2021).

Structured Abstract

Setting: The meta-analysis included studies of financial aid programs in the United States published between January 1, 2002 and January 15, 2020.

Sample: The meta-analysis focused on studies examining the effects of financial aid programs for K–12 students meeting college aid program criteria, high school students, recent high school graduates, and adult learners.  

Intervention: This project focused on synthesizing the effects of grant aid to undergraduates that reduces college costs (does not have to be repaid). Grants may be awarded based on financial need and/or academic merit, place of residence, or other criteria. Aid includes grants, scholarships, “free tuition,” tuition waivers, and subsidies. The researchers excluded studies focused on tuition-price setting, athletic scholarships, individual tax savings accounts, work study, and aid programs requiring service. The researchers also excluded aid programs that were bundled together and did not analyze the effect of one specified aid program  The researchers included studies of the elimination or loss of grant aid meeting these intervention criteria and analyzed them separately from the studies evaluating effects of the presence of grant aid.

Studies that met inclusion criteria featured seven different types of grant aid programs: (1) federal grants, (2) national scholarships, (3) state-sponsored grants, (4) institutional grants, (5) student performance-based financial incentives, (6) emergency financial assistance, and (7) promise programs.

Research Design and Methods: The researchers conducted  a systematic review and a meta-analysis concurrently. Studies that analyzed individual student data with randomized–controlled trials, regression discontinuity designs, difference-in-differences analyses, and other quasi-experimental studies were eligible for inclusion. To prevent sample selection bias, the researchers conducted an exhaustive search for references in electronic databases, publication repositories, conference paper archives, reference lists within published articles, and contacts with authors of aid program evaluations.

The first stage of the project involved completion of a systematic review of the research literature on financial aid published between January 1, 2002 and January 15, 2020, which identified evaluation studies that meet a pre-established set of criteria for inclusion in the meta-analysis. The second stage involved coding each identified study for the type of financial aid program evaluated and its key features, the effect sizes associated with the aid program in relation to each of five postsecondary outcomes, and the main design and methodological attributes of the study. The final stage involved synthesis of the findings from prior evaluation studies using advanced meta-analysis modeling techniques.

Key measures: Following the project's pre-analysis plan and informed by What Works Clearinghouse's (WWC's) Support Postsecondary Success protocol (IES, 2019), the research team categorized outcome measures and subsequent meta-analyses into six outcome domains: enrollment, academic achievement, credit accumulation, persistence, degree completion, and post-college labor market outcomes. Enrollment includes studies of any postsecondary enrollment, enrollment in a 2-year institution, and enrollment in a 4-year institution. Academic achievement is measured by grade point average (GPA) either in an academic term or cumulative through a certain period (e.g., 2, 3, or 4 years). Credit accumulation typically is measured as an average number of college-level credits earned per term but also includes cumulative number of credits earned during specific periods (e.g., 2, 3, or 4 years). Persistence outcomes measure whether a student re-enrolls for a subsequent semester as well as total terms enrolled over multiple years (e.g., 2, 3, or 4 years). Persistence also includes measures of stopping and dropping out (effect sizes reverse coded) and transfer from two-year to four-year institutions. Completion is measured by whether a student earned any degree, an associate degree, or a bachelor's degree. Post-college labor market outcomes include average earnings through a particular period (e.g., 5, 8, or 12 years after high school graduation), as well as whether a student is employed and has year-round employment in year 10.

The research team organized the grant programs that were evaluated in studies that met their inclusion criteria into seven types: (1) federal grants, (2) national scholarships, (3) state-sponsored grants, (4) institutional grants, (5) student performance-based financial incentives, (6) emergency financial assistance, and (7) promise programs. Researchers further divided the promise program category based on the range of institutions at which the award could be used.

Data Analytic Strategy: The project team estimated a standardized mean difference effect size in the form of Hedges' g and the effect size variance for outcome measures in six outcome domains: (1) initial enrollment, (2) college academic achievement, (3) credit accumulation, (4) persistence, (5) degree completion, and (6) post-college labor market outcomes. Having baseline data that measured the difference between the intervention and comparison groups before receiving the grant was an eligibility requirement for nonrandom designs, and the eligibility requirements related to outcomes varied by the outcome type. Adjustments for baseline demographic covariates were included in effect size estimation.

To estimate the meta-analytic models, the research team used a random-effects model with robust variance estimation to produce a weighted average of the effect sizes. Effect sizes were weighted inversely according to their variances and covariances as implied by the sandwich estimator, assuming a correlated effects working model. Researchers used this specification to account for non-independent sampling errors attributable to the inclusion of multiple effect sizes from the same study. For each outcome domain, the project team estimated the heterogeneity of effects across studies using 𝜏2, which represented the absolute magnitude of effect heterogeneity, and I2, which represented the percentage of heterogeneity assumed to be from the true effects. The researchers assumed that, given the range of studies and effects included, significant heterogeneity would be found. In their pre-analysis plan, the researchers designed two sets of moderator analyses: confirmatory and exploratory. Confirmatory moderator analyses tested specific, policy-relevant hypotheses. They used an ANOVA-like moderator analysis, computing a Q-statistic. This approach allowed the researchers to evaluate whether differences between moderator levels were statistically significant.

PRODUCTS AND PUBLICATIONS

ERIC Citation: Find available citations in ERIC for this award here.

Project website: https://dsgonline.com/effects-of-college-grant-aid/

Select Publications:

LaSota, R., Polanin, J. R., Perna, L.W., Austin, M. J., Steingut, R.R., & Rodgers, M.A. (2021). The effects of losing postsecondary student grant aid: Results from a systematic review. Educational Researcher, 51(2), 160–168. doi.org/10.3102/0013189X211056868

Polanin, J.R., Zhang, Q., Taylor, J., Williams, R.T., Joshi, M., Burr, L. (2023). Evidence gap maps in education research. Journal of Research on Educational Effectiveness, 16:3, 532–552, DOI: 10.1080/19345747.2022.2139312.