Skip Navigation
Impact Evaluation of the U.S. Department of Education's Student Mentoring Program

NCEE 2009-4047
March 2009

Impacts of the Student Mentoring Program

Estimation of Overall Impacts of the Student Mentoring Program
We estimated a total of 17 impacts in three domains: (1) academic achievement and engagement; (2) interpersonal relationships and personal responsibility; and (3) high-risk or delinquent behavior.

  • The Student Mentoring Program did not lead to statistically significant impacts on students in any of the three outcome domains. The estimated impact on the Student Mentoring Program on the outcome measures for all three domains is reported in Exhibit ES.1.
  • Three of the impacts were statistically significant before accounting for multiple comparisons. However, after accounting for multiple comparisons within each of the three domains, these three impact estimates were no longer statistically significant.

Estimation of Subgroup Effects
Several subgroup analyses were statistically significant after accounting for multiple comparisons.

  • The Student Mentoring Program improved academic outcomes for girls and produced mixed academic outcomes for boys. There were several positive impacts of the program for girls. The impact on self-reported scholastic efficacy and school bonding was positive and statistically significant for girls, with treatment group girls scoring higher than control group girls. In addition, there was a statistically significant difference in impacts on the scholastic efficacy and school bonding measure by gender (effect size for girls = 0.18 versus -0.05 for boys). There was also a positive, statistically significant effect on future orientation for boys (effect size = 0.17). However, the difference in impacts between boys and girls on this measure was not statistically significant.
  • For boys, the Student Mentoring Program negatively affected self-reported prosocial behavior. Boys who were assigned to mentoring reported statistically significant lower scores on the pro-social behaviors scale compared to their control group peers. Moreover, there was a statistically significant difference in impacts on the pro-social behaviors scale by gender (effect size for girls = 0.08 versus – 0.11 for boys).
  • The Student Mentoring Program led to a decrease in truancy for younger students. Truancy (i.e., unexcused absence) showed a statistically significant improvement for younger students (below age 12) who were assigned to mentoring compared to same age peers in the control group (effect size = -0.23). However, the difference in impacts on truancy between younger and older students (aged 12 and older) was not statistically significant after accounting for multiple comparisons.

Site-Level Characteristics and Impacts
Although we did not find that the Student Mentoring Programs had statistically significant impacts on student-level outcomes for our sample as a whole, we wished to determine whether characteristics of programs and their mentors varied across sites and, if so, whether we could identify program and mentor characteristics associated with differences in impacts at the site level. Because sites were not randomly assigned to different levels of implementation—a primary potential source of impact variation—this analysis is descriptive and exploratory, not causal, in nature.

For this analysis, it was essential to develop a parsimonious model for testing for any relationship between program and mentor characteristics (and contextual factors) and site-level impacts. Therefore, in choosing the final set of site-level covariates for inclusion in our model, we considered several factors, including their theoretical importance in influencing impacts, possessing statistically significant site-level variation, and low site-level correlations among these variables to avoid problems with multicollinearity.11

The site-level covariates in our analysis included nine factors: (1) average hours of pre-match training provided to mentors; (2) amount of ongoing mentor support (average frequency of mentor-supervisor meetings); (3) use of activities in mentor/student meetings (e.g., percent of mentors reporting almost always/most of the time either working on homework and/or academic skills with students); (4) percent of mentors aged 22 or below; (5) percent of mentor/student matches of the same race/ethnicity; (6) percent of students with self-reported delinquent behaviors at baseline; (7) percent of students scoring “not proficient” in either math or reading/ELA at baseline; (8) percent of mentor/student matches lasting 6 months or longer; and (9) average total hours of mentor/student meetings per month.12

Although we did not explicitly control for multiple comparisons because this was an exploratory analysis, it is important to note that we conducted 153 individual hypothesis tests of potential associations between the 9 covariates and the 17 outcome measures, for roughly 7 or 8 of which we would expect to reject the null hypothesis at the 0.05 level by random chance alone. In fact, we found 12 statistically significant relationships.

The following associations between site-level impacts and each of these site characteristics were statistically significant at the 95 percent confidence level, holding all other characteristics constant:13

  • The frequency of mentor/supervisor meetings was negatively associated with sitelevel impacts. All other things equal, the frequency of mentor/supervisor meetings was negatively associated with site-level impacts on the Pro-social Behaviors measure from the Student Survey and on grades in math and social studies. They were also positively associated with site-level impacts on school-reported delinquency.
  • The proportion of students with self-reported delinquent behaviors at baseline had both positive and negative relationships with site-level impacts. The proportion of students with self-reported delinquent behaviors at baseline was positively associated with site-level impacts on social studies grades and negatively associated with site-level impacts on absenteeism and truancy.

However, the proportion of students with self-reported delinquent behaviors at baseline was also positively associated with site-level impacts on repeated misconduct from student records.

  • The proportion of mentors aged 22 or younger was negatively associated with sitelevel impacts on math grades.
  • The proportion of mentor/student matches of the same race/ethnicity was positively associated with site-level impacts on reading/ELA grades.
  • Average monthly hours of mentor/student meetings had both positive and negative relationships with site-level impacts. Average monthly hours of meeting were positively associated with site-level impacts on student self-reported future orientation, but negatively associated with site-level impacts on grades in math and reading/ELA.

Top

11 In general, we focused on proximal factors thought to have a direct influence on impacts rather than distal factors, which may be correlated with impacts, but whose influence may be indirect and/or mediated through more proximal causes.
12 We also included in our analyses an indicator variable for the share of the control group that received mentoring (from any source) during the outcome period to adjust for potential differential attenuation of impact estimates from site to site.
13 For the purposes of reporting associations between site-level characteristics and impacts, we refer to relationships as “positive” or “negative” in the statistical sense, reflecting the direction of the coefficient. However, in some cases a positive statistical relationship denotes a negative substantive relationship or a negative statistical relationship denotes a positive substantive relationship.