Appendix A1.1 Study characteristics: Flay, Acock, Vuchinich, & Beets, 2006 (randomized controlled trial)

Characteristic	Description
Study citation	Flay, B., Acock, A., Vuchinich, S., & Beets, M. (2006). Progress report of the randomized trial of Positive Action in Hawaii: End of third year of intervention. Available from Positive Action, Inc. 264 4th Avenue South, Twin Falls, ID 83301.
Participants	The study tracked students who started grades K–1 in the 2001–02 academic year. The report reviewed by the WWC examined the outcomes of 2,666 third- and fourth-grade students at the end of the third year of program implementation. These outcomes included students who had remained in the school since the beginning of the study as well as new students in the school. About 25% of the students surveyed each year were new to the school. The students were enrolled in 20 elementary schools that were randomly assigned into conditions. A small percentage of the students were white (13.40% in the intervention school and 17.94% in the comparison). Among the remaining students the following ethnic groups were represented: Hawaiian, Filipino, Asian (other), Japanese, Samoan, Hispanic, Chinese, Black, and Portuguese. About 60% of the schools in the sample were Title I schools.
Setting	The study was conducted in 20 K–5 or K–6 schools in Hawaii. Five pairs of matched participating schools were from Oahu, three pairs were from Maui, and two pairs were from Molokai.
Intervention	The program consisted of the Positive Action curriculum and additional components that involved school principals, counselors, parents, and community members. The components addressed school and classroom management, school climate, family and community involvement, and skills and knowledge related to core values.1 The report reviewed by the WWC presented findings for the end of the third year of program implementation. The authors report that by the third year, two schools were still implementing at a very low level, three at a moderate-to-high level, and five at a high level. But even the high-implementation schools were still not implementing at the level the program developer would expect for high-implementation schools. For example, few schools implemented the family- or community-involvement programs.
Comparison	The comparison schools were drawn from the same school districts as the intervention schools and were matched on demographic characteristics, student behavior, and academic achievement. Comparison schools did not implement the Positive Action program. The comparison schools had other types of character education activities they regarded as business-as-usual, which were also practiced (although to a lesser extent) in the intervention schools.1
Primary outcomes and measurement	The study examined students' outcomes in the academic achievement and behavior domains.2 Outcomes in the academic achievement domain included percent proficient on the reading and math sections of a state standardized test and daily absences. Outcomes in the behavior domain included students' reports of use of alcohol, tobacco, and illegal drugs. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.)
Teacher training	Prior to each school year, the program developer, Dr. Allred, provided teacher/staff training to each intervention school. The training lasted 3–4 hours the first year and 1–2 hours each of the subsequent years. In addition, Dr. Allred visited each school at least once each year to provide an in-service session (usually 30-50 minutes). Finally, 5–6 representatives from each intervention school participated each winter in a mini-conference to obtain further training on the schoolwide components of the program and to share experiences.
1 Schools in the intervention and comparison groups implemented additional character and behavior programs at the time of the study, which they regarded as business-as-usual. These programs include the Hawai'i State Department of Education (HIDOE) mandated General Learner Outcomes (GLO), TRIBES, Lions Quest, and additional programs related to school management and organization and drug and violence prevention, such as Positive Behavioral Support, D. A. R. E., Red Ribbon Week, Peace Builders, and Peace Week. The study authors informed the WWC that these schools had some of the additional programs in place before the time of pretest. Because the same practices took place in both study conditions, and because greater time dedicated to those additional practices was reported for the comparison group than for the intervention group, this study was reviewed. The WWC cautions that interaction effects between Positive Action and the additional programs may have enhanced the program effects. 2 The study also examined students' attitudes and values and additional teacher-reported outcomes in the behavior domain. But, since complete statistical information was not available, these outcomes were not reviewed.

Top

Appendix A1.2 Study Characteristics: Flay & Allred, 2003 (quasi-experimental design)

Characteristic	Description
Study citation	Flay, B. R., & Allred, C. G. (2003). Long-term effects of the Positive Action program. American Journal of Healthy Behavior, 27 (1), 6–21.
Participants	The study included 36 elementary schools. About 62% of the students in the intervention group participated in the free or reduced-price lunch program compared with 67% in the comparison group. About half of the students were white (50.59% in the intervention group and 44.66% in the comparison group). About one-fourth of the students were African-American (24.61% in the intervention group and 28.48% in the comparison group). About one-fifth of the students were Hispanic (20.71% in the intervention group and 23.23% in the comparison group). Because schools in the intervention and comparison conditions were matched, there were no statistically significant differences in demographic characteristics between the two groups.
Setting	The participating schools were from one large Southeastern school district. The school district was characterized by a large number of school sites (about 65) that have implemented Positive Action for four or more years.
Intervention	The program consisted of the Positive Action curriculum and additional activities that involved school principals, parents, and community members. The components addressed school and classroom management, learning climate, and skills and knowledge related to core values.3
Comparison	The comparison schools were drawn from the same school district as the intervention schools and were matched on demographic characteristics. Comparison schools did not implement the Positive Action program.
Primary outcomes and measurement4	The study measures in the behavior domain included violence and suspensions rates. The study measures in the academic achievement domain included the Florida Comprehensive Aptitude Test (FCAT) and grade retention rates. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.)
Teacher training	No information on teacher training was provided.
3 Half of the schools in the intervention group implemented additional character and behavior programs at the time of the study. The study authors informed the WWC that these schools had the additional programs in place before the time of pretest. That is, the Positive Action program was the only new addition between the pretest and the posttest. The WWC cautions that interaction effects between Positive Action and the additional programs may have enhanced the program effects. 4 Flay and Allred (2003) also investigated the long-term impact of Positive Action on students in middle and high schools. The study conducted school-level analyses comparing three categories of schools: less than 60%, between 60% and 79%, and between 80% and 100% of the student population with exposure to the elementary school Positive Action program. But, because those analyses did not use a comparison group, they were not reviewed in this WWC report.

Top

Appendix A2.1 Outcome measures in the behavior domain

Outcome measure	Description
Violence rates	School-level archival data that consist of disciplinary referrals for incidents of violence per 100 students (as cited in Flay & Allred, 2003).
Suspensions rates	School-level archival data that consist of percent of students who received out-of-school suspensions (as cited in Flay & Allred, 2003).
Tobacco use	One item rated on a three-point scale on which students indicate if they have ever smoked cigarettes (as cited in Flay et al., 2006).
Alcohol use	One item rated on a three-point scale on which students indicate if they have ever drunk alcoholic beverages (as cited in Flay et al., 2006).
Being drunk	One item rated on a three-point scale on which students indicate if they have ever been drunk (as cited in Flay et al., 2006).
Illegal drug use	One item rated on a three-point scale on which students indicate if they have ever used illegal drugs (as cited in Flay et al., 2006).
Violent behavior	A self-report survey that measures level of engagement in five serious behaviors: carried a knife or razor to use to hurt someone, threatened to cut or stab someone, cut or stabbed someone on purpose to hurt them, carried a gun, and shot at someone (as cited in Flay et al., 2006).
Suspensions	School-level archival data that consist of the rates at which students were suspended during the school year (as cited in Flay et al., 2006).

Top

Appendix A2.2 Outcome measures in the academic achievement domain

Outcome measure	Description
The Florida Comprehensive Aptitude Test (FCAT)	This standardized test contains two basic components: criterion-referenced tests (CRT), measuring selected benchmarks in mathematics, reading, science, and writing from the Sunshine State Standards (SSS); and norm-referenced tests (NRT) in reading and mathematics, measuring individual student performance against national norms. Total score for the grade 4 FCAT was used in the reviewed study (as cited in Flay & Allred, 2003).
The Hawaii Content and Performance Standards test (HCPS)	A state standardized academic achievement test. Fifth-grade test scores on the reading and math subtests were used in this study (as cited in Flay et al., 2006).
Absentee rates	School-level archival data that consist of percent of students absent for 21 or more days during the school year (as cited in Flay & Allred, 2003).
Retention in grade	School-level archival data that consist of the number of students retained in the same grade level (as cited in Flay et al., 2006).

Top

Appendix A3.1 Summary of study findings included in the rating for the behavior domain¹

			Author's findings from the study
			Mean outcome (standard deviation2)		WWC calculations
Outcome measure	Study sample3	Sample size (students/schools)	PositiveAction group	Comparison group	Mean difference4 (PositiveAction – comparison)	Effect size5	Statistical significance6 (at α= 0.05)	Improvement index7
Flay et al., 2006 (randomized controlled trial)
Suspensions	Grade 5	2,660/20	1.63 to 1,000 students	6.2 to 1,000 students	0.01	0.81	Statistically significant	+29
Tobacco use	Grade 5	2,660/20	3.33 (na)	4.78 (na)	1.45	0.23	Statistically significant	+9
Alcohol use	Grade 5	2,660/20	9.63 (na)	14.83 (na)	5.20	0.30	Statistically significant	+12
Being drunk	Grade 5	2,660/20	0.74 (na)	3.35 (na)	2.61	0.93	Statistically significant	+32
Illegal drug use	Grade 5	2,660/20	0 (na)	2.39 (na)	2.39	1.08	Statistically significant	+36
Serious violence (boys)	Grade 5	1,330/20	10.8 (na)	23.5 (na)	12.70	0.56	Statistically significant	+21
Serious violence (girls)	Grade 5	1,330/20	5.8 (na)	3.6 (na)	-2.20	-0.30	ns	-12
Average8 for behavior (Flay et al., 2006)						0.52	Statistically significant	+20
Flay & Allred, 2003 (quasi-experimental design)
Violence rates	Grades 1–6	36 schools	3.83 (na)	12.11 (na)	8.28	0.75	Statistically significant	+27
Suspensions rates	Grades 1–6	36 schools	2.72 (na)	4.09 (na)	1.37	0.25	Statistically significant	+10
Average8 for behavior (Flay & Allred, 2003)						0.50	Statistically significant	+19
Domain average8 across studies						0.51	na	+19
ns = not statistically significant na = not applicable 1 This appendix reports findings considered for the effectiveness rating and the improvement index. 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 Flay and Allred (2003) also investigated the impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because it did not use a comparison group. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for outcomes indicating problem behavior was reversed, to indicate a positive effect that is associated with a small rate of problem behavior in the intervention schools. 5 Effect sizes for all outcomes in the behavior domain were calculated using the odds ratio formula. For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The WWC-computed average effect sizes are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes.

Top

Appendix A3.2 Summary of study findings included in the rating for the academic achievement domain¹

			Author's findings from the study
			Mean outcome (standard deviation2)		WWC calculations
Outcome measure	Study sample3	Sample size (students/schools)	PositiveAction group	Comparison group	Mean difference4 (PositiveAction – comparison)	Effect size5	Statistical significance6 (at α= 0.05)	Improvement index7
Flay et al., 2006 (randomized controlled trial)
The Hawaii Content and Performance Standards test (HCPS)–math (percentage reaching proficiency)	Grade 5	20/2,660	26.00 (na)	21.00 (na)	5.00	0.17	ns	+7
The Hawaii Content and Performance Standards test (HCPS)–reading (percentage reaching proficiency)	Grade 5	20/2,660	52.00 (na)	44.00 (na)	8.00	0.19	ns	+8
Grade retention	Grade 5	20/2,660	1 to 1,000 students	6 to 1,000 students	0.01	1.09	Statistically significant	+36
Average8 for academic achievement (Flay et al., 2006)						0.48	Statistically significant	+19
Flay & Allred, 2003 (quasi-experimental design)
The Florida Comprehensive Aptitude Test (FCAT)	Grade 4	36 schools	290.90 (nr)	278.40 (nr)	12.50	na 9	Statistically significant	na 9
Absentee rates	Grades 1–6	36 schools	10.79 (na)	12.36 (na)	1.57	0.25	ns	+10
Average8 for academic achievement (Flay & Allred, 2003)						0.25	Statistically significant	+10
Domain average8 across studies						0.37	na	+14
ns = not statistically significant nr = not reported na = not applicable 1 This appendix reports findings considered for the effectiveness rating and the improvement index. One additional finding based on a subtest of the Florida Comprehensive Aptitude Test (FCAT) was not used for rating purposes because of an overlap with total FCAT score, but is presented in Appendix A4. 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 3 Flay and Allred (2003) also investigated the impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because it did not use a comparison group. 4 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for absentee rates was reversed, to indicate a positive effect that is associated with smaller absentee rates in the intervention schools. 5 Effect sizes for all outcomes reviewed for Flay and Allred (2003) and Flay et al. (2006), except for students scores on the FCAT in Flay and Allred (2003), were calculated using the odds ratio formula. For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations. 6 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 7 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 8 The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. 9 Student-level standard deviations were not available in Flay and Allred (2003) for the FCAT results. The study provided school-level pooled standard deviations but no standard devations by condition. Because the student-level effect size and improvement index could not be computed, the magnitude of the effect size for this outcome measure was not considered for rating purposes. However, the statistical significance for this outcome is comparable to other studies and is included in the intervention rating. For further details, please see Technical Details of WWC-Conducted Computations.

Top

Appendix A4 Summary of additional study findings for the academic achievement domain¹

			Author's findings from the study
			Mean outcome (standard deviation2)		WWC calculations
Outcome measure	Study sample	Sample size (students/schools)	PositiveAction group	Comparison group	Mean difference3 (PositiveAction – comparison)	Effect size4	Statistical significance5 (at α= 0.05)	Improvement index6
Flay & Allred, 2003 (quasi-experimental design)
Florida Reading Test	Grade 4	36 schools	105.90 (nr)	73.10 (nr)	32.80	1.29	Statistically significant	+40
nr = not reported 1 This appendix presents an additional finding that falls in the academic achievement domain. This is a subtest of the Florida Comprehensive Aptitude Test (FCAT). Findings for total score on the FCAT are presented in Appendix A3.2. 2 The standard deviation across all students in each group shows how dispersed the participants' outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For the calculation of an effect size pooled standard deviations, which were converted to student-level standard deviations, were used. 3 Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4 For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations. 5 Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, a correction for clustering was needed. 6 The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results.

Top

Appendix A5.1 Positive Action rating for the behavior domain

The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.¹

For the outcome domain of behavior, the WWC rated Positive Action as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest applicable rating.

Rating received
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant positive effects. Criterion 2: No studies showing statistically significant or substantively important negative effects. Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain.
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for rati, ngs of potentially positive effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A5.2 Positive Action rating for the academic achievement domain

The WWC rates interventions as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.¹

For the outcome domain of academic achievement, the WWC rated Positive Action as having positive effects. The remaining ratings (potnetially postivie effects, mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest applicable rating.

Rating received
Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant positive effects. Criterion 2: No studies showing statistically significant or substantively important negative effects. Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain.
1 For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC Intervention Rating Scheme for a complete description.

Top

Appendix A6 Extent of evidence by domain

	Sample size
Outcome domain	Number of studies	Schools	Students	Extent of evidence1
Behavior	2	56	over 2,660	Medium to large
Knowledge, attitudes, and values	0	0	0	na
Academic achievement	2	56	over 2,660	Medium to large
na = not applicable/not studied 1 A rating of "medium to large" requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the rating is "small."

Top