Appendix A1.1 Study characteristics: Sinclair, Christenson, Evelo, & Hurley, 1998 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Sinclair, M. F., Christenson, S. L., Evelo, D. L., & Hurley, C. M. (1998). Dropout prevention for youth with disabilities: Efficacy of a sustained school engagement procedure. Exceptional Children, 65 (1), 7–21. |
| Study design | Study used a random assignment research design and included 94 high school students—47 in the intervention group and 47 in the control group. Students were randomly assigned just before entering the ninth grade. |
| Participants | Participants were special education students enrolled in ninth grade during the 1994–95 school year who were classified with a learning, emotional, or behavioral disability. Learning disabilities were the most common classification, with 75% of participants having this diagnosis. A little more than 40% of participants were classified as having a severe disability. Most participants were African-American (59%); most were males (68%); and most participated in the free or reduced-price lunch program (71%). Students were 15-years-old, on average, when they entered ninth grade. |
| Setting | Study was conducted in Minneapolis public high schools. |
| Intervention | The intervention group received Check & Connect services in the seventh and eighth grade and, after being assigned to the intervention group, continued to receive the program in ninth grade. Students in Check & Connect had their attendance, behavior, and academic performance observed on a daily basis by their "monitor," who also functioned as a mentor and caseworker. Monitors met with students at least twice a month and more often when acute attendance, performance, or behavior problems arose. |
| Control | Control group students received Check & Connect in seventh and eighth grade but, after assignment to the control group, did not continue to receive these services when they entered high school. Control group students attended the same set of high schools attended by intervention group students. |
| Primary outcomes and measurement | Two relevant outcomes from this study are included in this summary: the percentage of students who had dropped out at the end of the first follow-up year and the number of credits earned during the first follow-up year. (See Appendix A3.) |
| Staff training | No specific information concerning staff training was provided, however, program staff noted that monitors should possess several key attributes. In particular, they indicated that " (w)hile familiarity with the schools and community resources was desirable, the essential qualifications of a monitor included patience; a belief that all students have abilities; willingness to work cooperatively with families and staff; and advocacy skills, particularly communication skills, such as the ability to negotiate, compromise, and confront conflict constructively" (p. 10). |
Appendix A1.2 Study characteristics: Sinclair, Christenson, & Thurlow, 2005 (randomized controlled trial)
| Characteristic | Description |
|---|---|
| Study citation | Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional Children, 71 (4), 465–482. |
| Study design | The study used a random assignment research design. The post-attrition sample included 144 high school students—71 in the intervention group and 73 in the control group. Students were randomly assigned at the beginning of ninth grade. |
| Participants | This replication of Check & Connect included special education students who entered ninth grade in 1996 and 1997. To be eligible for the study, participants had to be classified as having an emotional or behavioral disorder. Most students were African-American (64%); most were males (84%); and most participated in the free or reduced-price lunch program (70%). Students were 14.5-years-old, on average, when they entered ninth grade. |
| Setting | Study was conducted in Minneapolis public high schools. |
| Intervention | The intervention group participated in Check & Connect for four years, starting in ninth grade. Students had their attendance, behavior, and academic performance observed on a daily basis by their "monitor," who also functioned as a mentor and case worker. The monitor stayed with the student even if the student transferred to another school within the district. Monitors met with students at least twice a month and more often when acute attendance, performance, or behavior problems arose. |
| Control | Control group students attended the same schools as intervention students but did not receive Check & Connect . |
| Primary outcomes and measurement | Relevant outcomes included in this study are: the percentage of students who had dropped out of school at the end of the fourth year following random assignment and the percentage of students who either completed high school or their GED by the end of the fourth year. (See Appendix A3.) |
| Staff training | Monitors participated in an initial orientation workshop. They also attended weekly or biweekly staff meetings and periodic staff development sessions. Each monitor received instructions on how to complete the monitoring sheet to ensure consistency across monitors and settings. Monitors submitted printouts of attendance records with their monitoring sheets for verification purposes. |
Appendix A2.1 Outcome measures for the staying in school domain
| Outcome measure | Description |
|---|---|
| Dropped out of school at end of first year after random assignment | This measure represents whether students were not enrolled in school at the end of the first academic year after random assignment—or the end of the ninth grade, because students were randomly assigned at the beginning of high school. School enrollment was verified through a tracking system established for the study and was drawn from various sources, including project and school staff, social workers, and probation officers, as well as the school district's on-line database (as cited in Sinclair et al., 1998). |
| Dropped out of school at end of fourth year after random assignment | This measure represents students who had not completed high school or a GED and were not enrolled in school at the end of the fourth academic year after random assignment—or the senior year for those making normal progress toward graduation, because students were randomly assigned at the beginning of ninth grade. The study authors counted students as enrolled (and thus having not dropped out) if they transferred to another school district, a nonpublic school, or a state-approved education program, or if they were in a correctional institution (as cited in Sinclair, Christenson, & Thurlow, 2005). |
Appendix A2.2 Outcome measures for the progressing in school domain
| Outcome measure | Description |
|---|---|
| Credits earned during first year after random assignment | This measure was drawn from the school district's on-line database and represents the total credits earned during the first academic year after random assignment—or the ninth-grade school year, because students were randomly assigned at the beginning of high school (as cited in Sinclair et al., 1998). |
Appendix A2.3 Outcome measures for the completing school domain
| Outcome Measure | Description |
|---|---|
| Outcome measure Description Completed high school diploma or GED by end of fourth year after random assignment | This measure represents the percentage of students who completed an education program, including graduation with a standard diploma or a GED certificate, by the end of the fourth academic year after random assignment—or the senior year for those making normal progress toward graduation, because students were randomly assigned at the beginning of ninth grade. High school diploma receipt was verified through school district records. GED completion was verified independently by the State Department of Education (as cited in Sinclair, Christenson, & Thurlow, 2005). |
Appendix A3.1 Summary of study findings included in the rating for the staying in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/ students) | Check & Connect group | Control group | Mean difference3 (Check & Connect– control) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Sinclair et al., 1998 (randomized controlled trial)7 | ||||||||
| Dropped out of school (%) | Grade 9 | 94 | 9 (28) | 30 (46) | -21 | 0.88 | Statistically significant | 31 |
| Average8 for staying in school (Sinclair et al., 1998) | 0.88 | Statistically significant | 31 | |||||
| Sinclair et al., 2005 (randomized controlled trial with attrition problems)7 | ||||||||
| Dropped out of school (%) | Grade 12 | 144 | 39 (49) | 58 (50) | -18 | 0.46 | Statistically significant | 18 |
| Average8 for staying in school (Sinclair et al., 2005) | 0.46 | Statistically significant | 18 | |||||
| Domain average8 for staying in school across all studies | 0.67 | na | 25 | |||||
|
na = not applicable 1 This appendix reports findings considered for the effectiveness rating and the average improvement indices. The two Sinclair et al. studies also examined additional outcomes that do not fall within the three domains addressed by the WWC review—staying in school, progressing in school, and completing school. In the Sinclair et al. (1998) study, these additional outcomes included attendance, assignment completion, teacher's perceptions of the student's academic competence and problem behaviors, and student perceptions of the relevance of school and the likelihood of graduation. In the Sinclair et al. (2005) study, these additional outcomes included the likelihood that the student would transfer to another school and the level of the student's participation in the special education transition program. These additional results are not summarized in this report.2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For binary ("zero-one") outcomes, standard deviations were calculated using the following formula, ![sqrt{[N/ (N-1)]*sqrt[p* (1-p)]}](../images/formula_p9_a31.gif){kind=small} , where p is the percentage of the sample with a value of 1 for the outcome and N is the sample size. 3 Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of both Sinclair et al. studies, no corrections for clustering or multiple comparisons were needed. 8 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. |
||||||||
Appendix A3.2 Summary of study findings included in the rating for the progressing in school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/ students) | Check & Connect group | Control group | Mean difference3 (Check & Connect– control) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Sinclair et al., 1998 (randomized controlled trial)7 | ||||||||
| Credits earned | Grade 9 | 92 | 12.13 (6.56) | 6.63 (6.63) | 5.50 | 0.83 | Statistically significant | 30 |
| Domain average8 for progressing in school | 0.83 | Statistically significant | 30 | |||||
| 1 This appendix reports findings considered for the effectiveness rating and the average improvement index. Sinclair et al. (1998) also examined additional outcomes that do not fall within the three domains addressed by the WWC review—staying in school, progressing in school, and completing school. These additional outcomes included attendance, assignment completion, teacher's perceptions of the student's academic competence and problem behaviors, and student perceptions of the relevance of school and the likelihood of graduation. These additional results are not summarized in this report. 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Sinclair et al. (1998), no corrections for clustering or multiple comparisons were needed. 8 The WWC-computed average effect size is a simple average rounded to two decimal places. The average improvement index is calculated from the average effect size. |
||||||||
Appendix A3.3 Summary of study findings included in the rating for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome (standard deviation2) | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools/ students) | Check & Connect group | Control group | Mean difference3 (Check & Connect– control) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Sinclair et al., 2005 (randomized controlled trial with attrition problems)7 | ||||||||
| Completed high school or GED "on time" (%) | Grade 12 | 144 | 30 (46) | 29 (46) | 1 | 0.03 | ns | 1 |
| Domain average8 for completing school | 0.03 | ns | 1 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the average improvement index. Sinclair et al. (2005) also examined additional outcomes that do not fall within the three domains addressed by the WWC review—staying in school, progressing in school, and completing school. These additional outcomes included the likelihood that the student would transfer to another school and the level of the student's participation in the special education transition program. These additional results are not summarized in this report.2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For binary ("zero-one") outcomes, standard deviations were calculated using the following formula, , where p is the percentage of the sample with a value of 1 for the outcome and N is the sample size. 3 Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 4 For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Sinclair et al. (2005), no corrections for clustering or multiple comparisons were needed. 8 The WWC-computed average effect size is a simple average rounded to two decimal places. The average improvement index is calculated from the average effect size. |
||||||||
Appendix A4.1 Check & Connect rating for the staying in school domain
The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of staying in school, the WWC rated Check & Connect as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Check & Connect was assigned the highest applicable rating.
| Rating received |
|---|
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A4.2 Check & Connect rating for the progressing in school domain
The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of progressing in school, the WWC rated Check & Connect as having potentially positive effects. It did not meet the criteria for positive effects, because it only had one study that examined outcomes in this domain. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, negative effects) were not considered because Check & Connect was assigned a higher rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
Appendix A4.3 Check & Connect rating for the completing school domain
The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of completing school, the WWC rated Check & Connect as having indeterminate effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because no statistically significant or substantively important findings either positive and negative were reported on this domain.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following.
OR
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |
