A Pilot Evaluation of Small Group Challenging Horizons Program (CHP): A Randomized Trial
Langberg, Joshua M.; Smith, Bradley H.; Bogle, Kristin E.; Schmidt, Jonathan D.; Cole, Wesley R.; Pender, Carolyn A. S. (2007). Journal of Applied School Psychology, v23 n1 p31-58. Retrieved from: https://eric.ed.gov/?id=EJ755984
-
examining48Students, grades6-7
Grant Competition
Review Details
Reviewed: October 2016
- Grant Competition (findings for Challenging Horizons Program)
- Randomized Controlled Trial
- Meets WWC standards without reservations because it is a randomized controlled trial with low attrition.
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Children's Impairment Rating Scale: Academic Progress |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
2.05 |
3.70 |
Yes |
|
|
|
|
Children's Impairment Rating Scale: Clinical Significance of Academic Progress Problems |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
0.43 |
0.74 |
Yes |
|
|
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Conners Abbreviated Symptom Questionnaire: Organization |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
1.00 |
2.33 |
Yes |
|
|
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Children's Impairment Rating Scale: Clinical Significance of Problems Overall, parent report |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
0.43 |
0.67 |
Yes |
|
|
|
|
Children's Impairment Rating Scale: Behavioral Functioning |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
4.86 |
8.78 |
Yes |
|
|
|
|
Children's Impairment Rating Scale: Overall Problem Severity |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
1.86 |
2.74 |
Yes |
|
|
|
|
Children's Impairment Rating Scale: Overall Problem Severity, teacher report |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
2.73 |
3.39 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Conners Abbreviated Symptom Questionnaire: Homework Completion |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
1.05 |
1.78 |
No |
-- |
|
Outcome measure |
Comparison | Period | Sample |
Intervention mean |
Comparison mean |
Significant? |
Improvement index |
Evidence tier |
|---|---|---|---|---|---|---|---|---|
|
Children's Impairment Rating Scale: Self-esteem |
Challenging Horizons Program vs. District-run after-school program |
Full sample;
|
1.86 |
2.96 |
Yes |
|
|
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 33%
Male: 67% -
Urban
-
- B
- A
- C
- D
- E
- F
- G
- I
- H
- J
- K
- L
- P
- M
- N
- O
- Q
- R
- S
- V
- U
- T
- W
- X
- Z
- Y
- a
- h
- i
- b
- d
- e
- f
- c
- g
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- x
- w
- y
South Carolina
Study Details
Setting
The small group Challenging Horizons Program (CHP) was implemented in two racially and economically diverse urban schools in Columbia, South Carolina. The intervention occurred after school in regular classrooms within the school.
Study sample
The students examined are from two racially and economically diverse urban schools in South Carolina. All students had scored as "below basic" on the recent PACT test. There were 32 boys and 16 girls in the study. Intervention group: 62% African American and 38% Caucasian. Comparison group: 70% African American and 30% Caucasian. Intervention group: socioeconomic status 3.45 Comparison group: socioeconomic status 3.22
Intervention Group
The small group Challenging Horizons Program (CHP) operated 4 days per week for 2 hours each day after school in regular classrooms from September through December. The school district paid for all program and transportation costs. The intervention had a behavioral level system with four levels where students were placed weekly depending on progress toward targeted behaviors such as binder, bookbag, and locker organization; percentage of homework assignments accurately recorded; and behavior during the program. Students on higher levels were allowed certain privileges (e.g., choice of snacks, choice of where they sat in the classroom, eligibility to be team captains during recreation time) while students on lower levels had to work on homework during recreation time, be in sight of a counselor at all times, and did not get choice of snack. Classrooms were assigned by grade level (6th or 7th) with no more than 12 students per room. Upon arrival, there was a check-in, snack, and homework time. Then all students participated in two 35-minute groups. All students participated in an academic remediation group plus one of four manualized skills groups (1) individual goals group targeting specific academic and behavioral problem areas , 2) organization group where students were taught an organization system for bookbags, binders, and locker organization, 3) homework/time management group designed to teach students to plan ahead for tests and long-term assignments, and 4) academic interventions group designed to teach note-taking, written language, and study skills. The final 35 minutes of the program were devoted to a recreational activity (kickball, board games, computer time), which was determined given the individual students' behavioral level. All parents received weekly or daily report cards, depending on the behavioral level of the student.
Comparison Group
85% of the students in the comparison group participated in the district-run after-school program, which was run by certified middle-school teachers after school 3 days per week for 2 hours each day. The curriculum was the same curriculum delivered in the CHP academic remediation group and was designed to prepare students for PACT testing. Students in the comparison group attended an average of 2 days per week receiving a total of 4 hours of the curriculum weekly. Classes, which were taught in group format, included 15 students or fewer per classroom. Students attended subject sessions according to the subject areas they scored “below basic” on the previous year’s PACT test (e.g., Language Arts, Science, or Math). The program was free and transportation was provided. In the limitations section of the article, the authors stated, "CHP was compared with a community control rather than a multifaceted intervention of equal duration and intensity...we can make no inferences about the relative efficacy of the small group CHP as compared with similar evidence-based interventions." The authors note the difference in respect to service hours provided.
Support for implementation
The program was implemented primarily by university junior and senior psychology majors who attended one hour of group supervision once a week. They also participated in a week of training sessions prior to the start of the intervention and received a grade and three credit hours for participation. The program implementers were required to read a manual describing the program and pass a test on the rules and procedures before they were able to implement the intervention. If they did not earn a 90% or above on the test, they were required to retake the test until they earned that score.
Structuring Out-of-School Time to Improve Academic Achievement
Review Details
Reviewed: July 2009
Structuring Out-of-School Time to Improve Academic Achievement Practice Guide- Randomized Controlled Trial
- Meets WWC standards without reservations
This review may not reflect the full body of research evidence for this intervention.
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Findings
Evidence Tier rating based solely on this study. This intervention may achieve a higher tier when combined with the full body of evidence.
Sample Characteristics
Characteristics of study sample as reported by study author.
-
Female: 33%
Male: 67% -
Urban
-
Race Black 65% White 35%
An indicator of the effect of the intervention, the improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention.
For more, please see the WWC Glossary entry for improvement index.
An outcome is the knowledge, skills, and attitudes that are attained as a result of an activity. An outcome measures is an instrument, device, or method that provides data on the outcome.
A finding that is included in the effectiveness rating. Excluded findings may include subgroups and subscales.
The sample on which the analysis was conducted.
The group to which the intervention group is compared, which may include a different intervention, business as usual, or no services.
The timing of the post-intervention outcome measure.
The number of students included in the analysis.
The mean score of students in the intervention group.
The mean score of students in the comparison group.
The WWC considers a finding to be statistically significant if the likelihood that the finding is due to chance alone, rather than a real difference, is less than five percent.
The WWC reviews studies for WWC products, Department of Education grant competitions, and IES performance measures.
The name and version of the document used to guide the review of the study.
The version of the WWC design standards used to guide the review of the study.
The result of the WWC assessment of the study. The rating is based on the strength of evidence of the effectiveness of the intervention. Studies are given a rating of Meets WWC Design Standards without Reservations, Meets WWC Design Standards with Reservations, or >Does Not Meet WWC Design Standards.
A related publication that was reviewed alongside the main study of interest.
Study findings for this report.
Based on the direction, magnitude, and statistical significance of the findings within a domain, the WWC characterizes the findings from a study as one of the following: statistically significant positive effects, substantively important positive effects, indeterminate effects, substantively important negative effects, and statistically significant negative effects. For more, please see the WWC Handbook.
The WWC may review studies for multiple purposes, including different reports and re-reviews using updated standards. Each WWC review of this study is listed in the dropdown. Details on any review may be accessed by making a selection from the drop down list.
Tier 1 Strong indicates strong evidence of effectiveness,
Tier 2 Moderate indicates moderate evidence of effectiveness, and
Tier 3 Promising indicates promising evidence of effectiveness,
as defined in the
non-regulatory guidance for ESSA
and the regulations for ED discretionary grants (EDGAR Part 77).