Comparison of Follow-up Data With Pretest Data

Comparison of Follow-up Data With Pretest Data Showed That Even Start Children and Parents Gained the Same Amount, but not More, Than Control Children and Parents

Similar to the findings documented in St.Pierre, Ricciuti, Tao, et al (2003), data collected from Even Start families in the EDS at follow-up showed that children and parents made gains on many different literacy measures. However, follow-up data collected from control group families showed that they performed as well as Even Start children and their parents. Follow-up data showed that children and parents in the control group made the same kinds of gains on literacy assessments, on parent reports of child literacy, on parent-child reading, on literacy resources at home, on family economic self sufficiency, and so on, that were seen for Even Start families. Thus, the Even Start projects included in this evaluation did not have any discernable impact on the language and literacy skills of children and parents that participated in the study.

As with the pretest-posttest analyses of EDS data reported by St.Pierre, Ricciuti, Tao, et al (2003), Even Start vs. control group follow-up differences were analyzed for almost 50 different outcome measures, and so we expected to observe some significant differences simply by chance. Four significant differences were found when comparing follow-up data to pretest data. These differences are not on the same measures for which we found significant differences in the pretest-posttest analyses. Further, three of the four significant differences would disappear if a less liberal significance criterion had been used (we used p<.10 for this study, to be sure that we identified all possible Even Start impacts). Because of the large number of outcomes assessed, because of inconsistency over time in the outcomes on which significant differences were found, and because of the liberal significance level used for these analyses (p<.10), we do not assign much meaning to the few significant differences that were observed, both when comparing pretest to posttest data and when comparing pretest to follow-up data.

Measures on Which Even Start and the Control Group Were No Different. No significant difference between the pretest and follow-up scores of Even Start and control group participants was found on 44 of the 48 outcome measures used in this evaluation. These include all 13 direct child outcomes (PPVT, WJR-Letter Word Identification, WJR-Dictation, WJR-Applied Problems, WJR-Incomplete Words, WJR-Sound Blending, WJR-Early Development Cluster, Story & Print Concepts, SSRS-Social Skills in Preschool, SSRS-Social Skills in Elementary School, SSRS-Problem Behaviors in Preschool, SSRS-Problem Behaviors in Elementary School, Vineland Communication Domain), the eight outcomes derived from school records (percent days attended in preschool, percent days absent in preschool, percent children ever tardy in preschool, percent children in special education in preschool, percent days attended in elementary school, percent days absent in elementary school, percent children ever tardy in elementary school, percent children in special education in elementary school), five of seven parent reports of child literacy (percent of children who know the alphabet, percent of children who know colors, percent of children who can count to 100, extent to which children under 2 years, 6 months read, extent to which child knows print concepts), seven of nine direct parent outcomes (WJR-Letter Word ID, WJR-Passage Comprehension, WJR-Reading Vocabulary, WJR-Reading Comprehension, WJR-Basic Reading Skills, percent of employed parents, annual household income), both parent reports of parent literacy at home (variety of parent reading, variety of parent writing), the four parent reports of parent-child reading (percent of parents who read to child daily, amount of reading to child, variety of reading to child, quality of reading to child), the three parent reports of literacy resources at home (number of books child has, variety of non-print resources, variety of print resources), and the two parent reports of parent support of the child's school (parent participation in school and parent opinion about school).

Measures on Which Even Start Did Better Than The Control Group. Comparing pretest data to follow-up data, there were two parent-report measures on which Even Start children gained more than control children. Parents of Even Start children reported larger gains than parents of control children on the extent to which their children read and on the extent to which their children exhibited age-appropriate writing activities than did parents of control children. The scale measuring extent to which children read has values from 0 to 9. The value of the scale increases by 1 if the child pretends to read, reads for enjoyment, has memorized a book, has a favorite book, can follow written directions, can describe something learned through reading, rereads sentences, reads or pretends to read to someone else, or recognizes own first name in writing/print. The scale measuring age-appropriate writing skills has values from 0 to 2. The value of this scale increases by 1 if the child pretends to write or writes some letters of the alphabet.

The Even Start vs. control group differences on these two variables were deemed statistically significant only by using a liberal significance criterion (p<.10) and did not translate into educationally large differences. At follow-up, Even Start children scored 6.07 on the parent-report scale measuring extent of reading, while control group children scored 6.10 (scale ranges from 0 to 9, sd = 2.22). At follow-up, Even Start children scored 1.53 on the parent-report measure of age-appropriate writing activities, while control group children scored 1.48 (scale ranges from 0-2, sd = 0.66).

At the follow-up, there were two measures on which Even Start parents gained more than control parents: the Woodcock-Johnson Word Attack subscale and attainment of a GED. On the Word Attack subscale, Even Start adults gained an average of 6.38 points between pretest and follow-up, while control adults gained an average of 2.50 points, which corresponds to an effect size of about .29 standard deviations (p<.10). Word Attack measures a parent's skill in applying phonic and structural analysis skills to the pronunciation of unfamiliar printed words. The parent reads aloud letter combinations that are linguistically logical but that form nonsense words or low-frequency words in English. In addition, more Even Start than control group adults attained a GED between pretest and follow-up (9.9% vs. 2.5%, respectively, of those who reported not having a high school diploma or GED at pretest).³ This is the only finding that is statistically significant at a traditional level (p<.05), and is consistent with previous Even Start research showing that the program has a positive effect on GED attainment (St.Pierre, Swartz, Gamse, Murray, Deck & Nickel, 1995).

Changes in Findings From the Pre-Post Analyses. When assessing the evaluation findings based on analyses of pretest vs. follow-up data we see that the only significant effect observed in analyses of pretest vs. posttest data (St.Pierre, Ricciuti, Tao, et al, 2003) is not replicated in the follow-up results. At the first posttest (in elementary school but not in preschool), Even Start children were rated (using the Social Skills Rating System) by their teachers as exhibiting significantly fewer problem behaviors than control group children. At the follow-up, this difference was no longer significant. It is difficult to interpret changes in teacher ratings from the first posttest to the follow-up since children were in different classrooms and were probably rated by different teachers. Further, additional children became old enough to enter preschool, and to have the opportunity to be rated.

Top