Appendix A1.1 Study characteristics: Constantine, Seftor, Martin, Silva, & Myers (2006)—Texas study (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Constantine, J. M., Seftor, N. S., Martin, E. S., Silva, T., & Myers, D. (2006). A study of the effect of the Talent Search program on secondary and postsecondary outcomes in Florida, Indiana, and Texas: Final report from phase II of the national evaluation. Report prepared by Mathematica Policy Research for the U. S. Department of Education, Office of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Washington, DC: U. S. Department of Education. |
| Participants | The Texas study used a quasi-experimental research design. The sample included 4,027 students in the intervention group and 30,842 students in the comparison group. Propensity score modeling was used to match Talent Searchstudents to comparison students who attended the same high schools and who were in the ninth grade in the 1995-96 school year. Matching was based on 18 demographic and academic characteristics, including students' eighth-grade test scores, race or ethnicity, socioeconomic status, English proficiency, special education status, and enrollment in vocational education programs. There were no statistically significant differences between the intervention group and the matched comparison group on the baseline characteristics used in the matching procedures. A Talent Searchstudent could be matched to multiple comparison students. Weights were used to account for the closeness of the match—with closer matches receiving larger weights.1 In addition, the comparison sample was weighted to equal the size of the treatment group so as not to overstate statistical significance. So, the intervention and comparison groups each had an effective sample size of about 4,000. Compared with all Texas high school students, Talent Searchparticipants were more likely to be female (62% compared with 47%), economically disadvantaged (51% compared with 38%), and black or Hispanic (73% compared with 53%). Talent Searchstudents were less likely than other high school students in the state to be behind grade level (16% compared with 29%), to be receiving special education services (5% compared with 12%), or to score in the bottom quartile on standardized tests (22% compared with 27% for math and 20% compared with 27% for reading). |
| Setting | The Texas study was conducted in 10 Talent Searchprojects (each including 10-20 high schools) and included participants who entered ninth grade in 1995-96.2 |
| Intervention | Most participants received services in their junior and senior years of high school. Participants were either recruited to participate or volunteered to be in the program. |
| Comparison | Comparison group students did not participate in Talent Searchand attended the same high schools as students in the intervention group. |
| Primary outcomes and measurement | One relevant outcome from the Texas study—high school completion rates—is included in this summary. This measure represents whether students earned a high school diploma or received a GED certificate. (See Appendix A2 for a more detailed description of this outcome measure.) The study also examined the program's effects on financial aid receipt and college enrollment. However, these outcomes do not fall within the three domains (staying in school, progressing in school, and completing school) examined by the WWC's review of dropout prevention interventions. Therefore, these additional outcomes are not included in this report. |
| Teacher training | No specific information concerning staff training was provided. |
| 1 Additional information on how weights were constructed was obtained by the WWC from the study authors. 2 Information on the number of high schools per project was obtained by the WWC from the study authors. |
|
Appendix A1.2 Study characteristics: Constantine, Seftor, Martin, Silva, & Myers (2006)—Florida study (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Constantine, J. M., Seftor, N. S., Martin, E. S., Silva, T., & Myers, D. (2006). A study of the effect of the Talent Search program on secondary and postsecondary outcomes in Florida, Indiana, and Texas: Final report from phase II of the national evaluation. Report prepared by Mathematica Policy Research for the U. S. Department of Education, Office of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Washington, DC: U. S. Department of Education. |
| Participants | The Florida study used a quasi-experimental research design. The matched sample included 900 students in the intervention group and 42,514 students in the comparison group. Propensity score modeling was used to match Talent Searchparticipants to similar students who attended the same high schools and who were in the ninth grade in 1995-96. Matching was based on 13 demographic and academic characteristics, including whether students were economically disadvantaged, learning disabled, overage for grade, emotionally or physically disabled, or enrolled in a gifted or talented program. Students were also matched on gender, race or ethnicity, primary language spoken at home, and citizenship status. Intervention and comparison students were not statistically different from each other at the 0.05 level on any measures used in the matching procedures. However, there were statistically significant differences at the 0.10 level on two measures—language spoken at home and whether participated in a dropout prevention program. These differences were controlled for in regression models used to estimate program impacts. Weights were used to account for the closeness of the match, with closer matches receiving larger weights. In addition, the comparison sample was weighted to equal the size of the treatment group so as not to overstate statistical significance. So, the intervention and comparison groups each had an effective sample size of 900. Compared with all Florida high school students, participants in Talent Searchwere more likely to be black (46% compared with 25%) and economically disadvantaged (63% compared with 37%). They were less likely to be learning disabled (4% compared with 8%), overage for grade (10% compared with 26%), or male (34% compared with 53%). They were also less likely to be Hispanic (5% compared with 16%) or to speak a language other than English at home (3% compared with 14%). |
| Setting | The Florida study was conducted in five Talent Searchprojects throughout the state, each including 10-20 high schools. 1 Participants included students who entered ninth grade in 1995-96. |
| Intervention | Most participants received services in their junior and senior years of high school. School and Talent Searchstaff recruited participants to participate. |
| Comparison | Comparison group students did not participate in Talent Searchand attended the same high schools as students in the intervention group. |
| Primary outcomes and measurement | One relevant outcome from the Florida study—high school completion—is included in this summary. This measure represents whether sample members earned a high school diploma or received a GED certificate. (See Appendix A2 for a more detailed description of this outcome measure.) The study also examined the program's effects on financial aid receipt and college enrollment. However, these outcomes do not fall within the three domains (staying in school, progressing in school, and completing school) examined by the WWC's review of dropout prevention interventions. Therefore, these results are not included in this report. |
| Teacher training | No specific information concerning staff training was provided. |
|
1 Information on the number of high schools per project was obtained by the WWC from the study authors. |
|
Appendix A2 Outcome measures in the completing school domain
| Characteristic | Description |
|---|---|
| Earned a high school diploma or GED |
The percentage of students who received their diploma or GED by the end of their fifth academic year after entering the ninth grade. In both Texas and Florida, this information comes from a statewide database that tracked public high school enrollment and completion and GED receipt. The measure includes only diplomas earned from public high schools within the state and GED certificates earned within the state. Diplomas from private high schools and credentials earned outside the state are not included. |
Appendix A3 Summary of study findings included in the rating for the completing school domain1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size2 | Talent Search group | Comparison group | Mean difference3 (Talent Search- comparison) | Effect size4 | Statistical significance5 (at α= 0.05) | Improvement index6 |
| Constantine et al., 2006—Texas study (quasi-experimental design)7 | ||||||||
| Earned a high school diploma or GED | High school students | 8,054 | 86% | 77% | 9% | 0.37 | Statistically significant | +14 |
| Average8 for completing school (Constantine et al., 2006—Texas study) | 0.37 | Statistically significant | +14 | |||||
| Constantine et al., 2006—Florida study (quasi-experimental design)7 | ||||||||
| Earned a high school diploma or GED | High school students | 1,800 | 84% | 70% | 14% | 0.49 | Statistically significant | +19 |
| Average8 for completing school (Constantine et al., 2006—Florida study) | 0.49 | Statistically significant | +19 | |||||
| Domain average8 for completing school across all studies | 0.43 | na | +17 | |||||
|
na = not applicable 1 This appendix reports findings considered for the effectiveness rating and the improvement index.2 This column represents the effective sample size after weighting. Talent Searchstudents could be matched to multiple comparison students. Weights were used in the analysis that made the effective sample size for the comparison group equal to the Talent Search group. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4 Effect sizes were calculated using the Cox index for dichotomous outcomes. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 7 The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Constantine et al. (2006), no corrections for clustering or multiple comparisons were needed. 8 The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. |
||||||||
Appendix A4 Talent Searchrating for the completing school domain
The WWC rates an intervention's effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of completing school, the WWC rated Talent Searchas having potentially positive effects. It did not meet the criteria for positive effects because both Talent Searchstudies involved quasi-experimental designs. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered because Talent Searchwas assigned the highest applicable rating.
| Rating received |
|---|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. |