Estimation of Overall Impacts of the Student Mentoring Program
We estimated a total of 17 impacts in three domains: (1) academic achievement and
engagement; (2) interpersonal relationships and personal responsibility; and (3)
high-risk or delinquent behavior.
Estimation of Subgroup Effects
Several subgroup analyses were statistically significant after accounting for multiple
comparisons.
Site-Level Characteristics and Impacts
Although we did not find that the Student Mentoring Programs had statistically significant
impacts on student-level outcomes for our sample as a whole, we wished to determine
whether characteristics of programs and their mentors varied across sites and, if
so, whether we could identify program and mentor characteristics associated with
differences in impacts at the site level. Because sites were not randomly assigned
to different levels of implementation—a primary potential source of impact variation—this
analysis is descriptive and exploratory, not causal, in nature.
For this analysis, it was essential to develop a parsimonious model for testing for any relationship between program and mentor characteristics (and contextual factors) and site-level impacts. Therefore, in choosing the final set of site-level covariates for inclusion in our model, we considered several factors, including their theoretical importance in influencing impacts, possessing statistically significant site-level variation, and low site-level correlations among these variables to avoid problems with multicollinearity.11
The site-level covariates in our analysis included nine factors: (1) average hours of pre-match training provided to mentors; (2) amount of ongoing mentor support (average frequency of mentor-supervisor meetings); (3) use of activities in mentor/student meetings (e.g., percent of mentors reporting almost always/most of the time either working on homework and/or academic skills with students); (4) percent of mentors aged 22 or below; (5) percent of mentor/student matches of the same race/ethnicity; (6) percent of students with self-reported delinquent behaviors at baseline; (7) percent of students scoring “not proficient” in either math or reading/ELA at baseline; (8) percent of mentor/student matches lasting 6 months or longer; and (9) average total hours of mentor/student meetings per month.12
Although we did not explicitly control for multiple comparisons because this was an exploratory analysis, it is important to note that we conducted 153 individual hypothesis tests of potential associations between the 9 covariates and the 17 outcome measures, for roughly 7 or 8 of which we would expect to reject the null hypothesis at the 0.05 level by random chance alone. In fact, we found 12 statistically significant relationships.
The following associations between site-level impacts and each of these site characteristics were statistically significant at the 95 percent confidence level, holding all other characteristics constant:13
However, the proportion of students with self-reported delinquent behaviors at baseline was also positively associated with site-level impacts on repeated misconduct from student records.