Evaluation Design
This study employs a student-level random assignment design. Specifically, the current evaluation focuses on the impacts of the Student Mentoring Program on students randomly assigned to participate in the ED-funded programs compared to similar students who signed up to participate but were not assigned to participate in the programs.3 Thus, the study provides experimentally-based evidence about the efficacy of school-based mentoring programs when implemented by a variety of sponsoring organizations.
The key research questions that the evaluation addresses are:
- What is the impact of ED school-based mentoring programs on students' interpersonal relationships with adults, personal responsibility, and community involvement?
- What is the impact of ED school-based mentoring programs on students' school engagement (e.g., attendance, positive attitude towards school) and academic achievement?
- What is the impact of ED school-based mentoring programs on students' high-risk or delinquent behavior?
The sampling pool for this evaluation was based on 255 mentoring programs funded by ED in either 2004 or 2005. The study collected and aggregated data from two cohorts of students: one from the 2005–2006 school year and another from the 2006–2007 school year. The original evaluation design was based on only one cohort. The sample size calculations for this design were based on the assumption that mentoring would be provided to students for an entire school year. When it became apparent after the first program year, however, that the average amount of mentoring was much shorter (i.e., between five and six months) it was decided that in order to conduct a fair test of the program, a larger sample would be needed to detect a smaller effect size. Thus, two cohorts of students were recruited to reach the necessary sample size.
To be selected for the Impact Study, each grantee had to meet three criteria:
- Be operational so that it could recruit and match students to mentors in the Fall 2005 for the first group of grantees and Fall 2006 for the second group;
- Able to over-subscribe or identify excess demand supporting experimental study needs for an un-served control group (i.e., able to provide tangible evidence of a pool of 4th through 8th grade students referred to the mentoring program) of adequate size to support study requirements; and
- Willing and able to cooperate with the data collection and logistical needs of the national evaluation, including random assignment.