Appendix A1 Study characteristics: Quint, Bloom, Black, & Stephens, 2005—Houston study (quasi-experimental design)
| Characteristic | Description |
|---|---|
| Study citation | Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). Scaling up First Things First: The challenge of scaling up educational reform. New York, NY: MDRC. |
| Participants |
The study examined the performance of students attending schools in the Houston Independent School District. The main analysis sample included students from three high schools and four middle schools implementing First Things First, as well as students from a set of matched comparison schools. The middle school study did not examine outcomes relevant to WWC dropout prevention reviews. Therefore, the results in this intervention report pertain only to the high school analysis. The high school sample consists of all students enrolled in the study schools during the study period: three years prior to First Things First implementation and one year after implementation. This included a total of 7,891 high school students in the First Things First schools. The study authors did not report the number of students in comparison high schools. Comparison schools in the Houston school district were matched to each First Things First high school based on overall student performance at the schools on standardized achievement tests during the baseline period. High schools in the district whose mean combined reading and math scores on the Stanford Achievement Test, ninth edition (SAT-9), were within 0.25 standard deviations of the mean for the First Things First school were selected as comparison schools. This methodology resulted in 10 comparison schools for 2 of the First Things First high schools and 11 comparison schools for the other. A high school could be chosen as the comparison school for more than one First Things First school, and this often occurred. A total of 13 Houston high schools served as comparison schools for the First Things First Houston study. The high school study examined two cohorts of students. Cohort 1 consisted of one intervention high school that implemented First Things First in 2001 and its matched comparison schools. Cohort 2 consisted of two intervention high schools that began implementing First Things First in 2002 and their comparison schools. The WWC used results for cohorts 1 and 2 combined to rate the effectiveness of First Things First. In the years prior to the implementation of First Things First, the three First Things First schools and their comparison high schools had, on average, similar attendance and promotion rates and served students who had similar tenth-grade passing rates on math and reading tests. The study authors indicated that the First Things First high schools in Houston were heavily Hispanic, while other low-achieving high schools in Houston were heavily African-American. Therefore, it was not possible to match high schools closely on both their racial and ethnic composition and their student performance. |
| Setting | First Things First was implemented in three high schools in the Houston Independent School District: Lee High School, Sam Houston High School, and Sharpstown High School. During the study period, these three high schools served student populations that were primarily Hispanic. Lee High School began implementing First Things First in the fall of the 2001/02 school year, and Sam Houston and Sharpstown began implementing the program the following school year, 2002/03. |
| Intervention | First Things First targeted three high schools in Houston with low-achievement levels. Each of these schools implemented the key features of the three components of the whole school reform: Small learning communities. First Things First reorganizes schools into small learning communities of up to 350 students and their teachers. The small learning communities in Houston were focused around a central theme (for example, technology), and students remained in the same communities throughout high school. First Things First also recommends that schools reduce student-teacher ratios in math and language arts classes to increase the amount of individualized attention that students receive. Class sizes in the Houston study schools were reduced from an average of 26 students to 20 students between the planning year and the second implementation year. Family and student advocate system. Advocates met one-on-one with students weekly and contacted the students’ families at least monthly to discuss their academic and personal progress. In at least one Houston high school the advocates became counselors for the students, helping them to schedule the classes they needed to progress toward graduation. Consistent with the model’s guidelines, advocates in Houston high schools met weekly with their students as a group. These group sessions were held as either a daily “homeroom” period or a weekly class meeting. Instructional changes and supports. First Things First contracted with Kagan Cooperative Learning, Inc., to train teachers on cooperative learning strategies intended to complement the small learning communities. It also called for increased instructional time for math and language arts courses. |
| Comparison | Matched comparison schools were Houston high schools that did not implement First Things First. The study identified comparison schools from the districts that were similar in average performance on standardized achievement tests in the three years preceding program implementation. High schools in the district whose mean scores on the Stanford Achievement Test, ninth edition (SAT-9), for reading and math combined were within 0.25 standard deviations of the mean for the First Things First school were selected as comparison schools. |
| Primary outcomes and measurement | The outcome of interest from this study was a measure of the percentage of ninth graders who attended at least one day of school during the following year, referred to as the “one-year persistence rate.” The study also examined First Things First’s effects on attendance and standardized test scores. These outcomes do not fall within the three domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, completing school) and are not included in this report. |
| Staff training | Teachers at First Things First high schools were regular teachers employed by the Houston Independent School District. As part of the instructional changes, teachers in First Things First schools were trained in cooperative learning strategies by Kagan Cooperative Learning. Each school employed a First Things First director to serve as a liaison between the district and the First Things First schools. Staff from IRRE provided ongoing professional development for teachers in the implementation of the family advocate system. |
Appendix A2 Outcome measures in the staying in school domain
| Outcome measure | Description |
|---|---|
| One-year persistence rate | The percentage of ninth-graders who attended at least one day at a district school at any point during the following school year. Students who were recorded in district records as having transferred to another district at the end of the ninth grade are coded as having persisted in school. |
Appendix A3 Summary of study findings included in the rating for the staying in school domain 1
| Author's findings from the study | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mean outcome | WWC calculations | |||||||
| Outcome measure | Study sample | Sample size (schools) 2 | First Things First group |
Comparison group 3 | Mean difference4 | Effect size5 | Statistical significance6 (at α= 0.05) |
Improvement index7 |
| Quint, Bloom, Black, & Stephens, 2005—Houston study (quasi-experimental design)8 | ||||||||
| One-year persistence rate | Ninth graders (cohorts 1 and 2) |
16 | 76.3 | 77.3 | –1.0 | –0.03 | ns | –1 |
| Domain average for staying in school 9 | –0.03 | ns | –1 | |||||
|
ns = not statistically significant 1 This appendix reports findings considered for the effectiveness rating and the improvement index. The one-year persistence rate was measured during the second year of implementation of First Things First in Houston. |
||||||||
Appendix A4 First Things First rating for the staying in school domain
The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1
For the outcome domain of staying in school, the WWC rated First Things First as having no discernible effects. It did not meet the criteria for positive effects, potentially positive effects, mixed effects, potentially negative effects, or negative effects because it had only one study, and that study showed no statistically significant or substantively important outcomes, either positive or negative, in this domain.
For explanations of how First Things First fared on the criteria for these ratings, see below.
| Rating received |
|---|
|
No discernible effects: No affirmative evidence of effects.
|
| Other ratings considered |
|
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.
AND
|
|
Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.
AND
|
|
Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria.
OR
|
|
Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence.
AND
|
|
Negative effects: Strong evidence of a negative effect with no overriding contrary evidence.
AND
|
|
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme. |
Appendix A5 Extent of evidence by domain
| Sample size | ||||
|---|---|---|---|---|
| Outcome domain | Number of studies | Schools | Students | Extent of evidence1 |
| Staying in school | 1 | 16 | nr | small |
| Progressing in school | 0 | na | na | na |
| Completing school | 0 | na | na | na |
|
na = not applicable or not studied |
||||
|Institute of Education Sciences
