Project CRISS Reading Program and Grade 9 Reading Achievement in Rural High Schools

Regional need and study purpose

This randomized study tests the effectiveness of Project CRISS (Creating Independence through Student-owned Strategies), an education program designed to improve reading comprehension. Project CRISS uses extended teacher professional development and follow-up support to help teachers and students apply research-based learning principles to improve reading comprehension. Developed with the emerging knowledge base on learning theory and reading comprehension, the program has been field-tested and revised over two decades. In 1985, after field-testing in a Montana school district, the program was disseminated through the former National Diffusion Network of the U.S. Department of Education (Santa 1993).

The primary research question for this study is:

• What impact does Project CRISS have on the reading comprehension of grade 9 students in rural high schools?

This study also addresses two process questions on the implementation of CRISS and differences in classroom practices:

• To what extent is Project CRISS implemented with fidelity in the treatment schools? What adjustments or adaptations are made to the program developer's prescribed intervention during implementation?
• How do classroom instructional practices in the treatment schools compare with those in the control schools?

Aimed at teaching reading comprehension across multiple subjects, Project CRISS is intended to help struggling adolescent readers succeed in core high school subjects in the current environment of challenging standards-based content. Achieving reading proficiency in the early high school years is a high-priority need in the Northwest Region, especially among poor and minority students. This study's goal is to determine whether Project CRISS can improve the reading comprehension of early high school students. If so, students might have greater potential for success in content-area coursework that requires extensive reading and understanding of text.

The results will provide scientific evidence on Project CRISS in rural and small town high schools in the Northwest Region, as well as its effect size in such settings. The study can be used by other researchers and IES to add to the growing body of scientific evidence on "what works" to improve American education.

Intervention description

Essentially a teacher professional development intervention, Project CRISS is designed for high school teachers of all the core subjects, including English language arts, mathematics, science, and social studies. Teachers learn how to apply general Project CRISS learning strategies to content and curricular materials in these core content areas. For example, the trainer demonstrates learning principles—such as assessing student knowledge of a subject before introducing new material or using outlines or graphic organizers to comprehend long text. During the training sessions teachers work in small groups to decide how they can use these techniques in their specific content areas. It is hypothesized that as teachers increase their use of such research-based strategies, students learn more ways to comprehend text and become more self-directed readers.

The prescribed CRISS training and follow-up assistance allow teachers to learn, practice, combine, and adapt research-based reading comprehension strategies across content areas over a one-year period, followed by an additional year of reinforcement training and follow-up support. A local site facilitator (teacher in the school or district) is selected and trained to serve as the in-school resource. By year 2 of the intervention, teachers are expected to have developed lesson plans that consistently incorporate research-based learning and reading comprehension strategies across all content areas.

Project CRISS is an integration of research-based practices from cognitive science (Brune 1977; Bloom 1950; Simon 1979), reading comprehension research (Palincsar and Brown 1984; Duke and Pearson 2002), social learning research (Vygotsky 1978; Bandura 1977), and instructional strategies to help students understand text (Duffy et al. 1987; Brown et al. 1996; Duke and Pearson 2002). Concepts derived from these areas of research form the foundation of Project CRISS philosophy and operational principles. Strategies in the Project CRISS model reflect the conclusion in Reading Next, from the Alliance for Excellent Education: struggling high school students need to learn the strategies that research indicates are used by proficient readers through explicit classroom instruction (Biancarosa and Snow 2004). Reading Next also concludes that effective instructional principles should be embedded in content, with language arts teachers teaching from content-area texts and other content teachers providing instruction and practice in reading and writing skills in their subject areas.

There is some prior nonexperimental evidence from the developer on the effectiveness of Project CRISS. A study in two Utah school districts compared students in Project CRISS schools with students in demographically similar comparison schools at the elementary grade level (grade 4), middle grade level (grade 7), and the high school level (grades 9–11). Student reading comprehension was measured using a reading recall/comprehension test before and after they were taught either by Project CRISS-trained teachers in treatment schools or by teachers using more traditional "business-as-usual" methods in comparison schools.

Results showed consistent and positive findings across all grades and subjects taught. In the three high school teacher samples of different content areas (biology, social studies, and English language arts teachers) there were statistically significant effects in which the treatment students and comparison students performed roughly similarly on the pretest before the teacher training and in which the treatment students outperformed the comparison students on the posttest after the teacher training (Santa 2004).

The Utah study mirrors other developer research studies using similar methods and achieving similar results in Colorado, Florida, Virginia, and Washington (Santa 1995, 1993). Most of these studies were conducted on primarily White students from middle or lower middle class households.

Study design

Unlike past studies conducted by the developer, this study is being conducted by an independent party. It tests for effects on grade 9 students in schools that are randomly selected into either a Project CRISS treatment group or a non-Project CRISS control group. Random assignment means that schools have a 50/50 chance of being selected into the treatment group or control group. This method provides a more scientifically rigorous study of Project CRISS effectiveness than previous developer studies. This study of approximately 5,000 students in more than 50 high schools will add to the knowledge base of Project CRISS effectiveness and of adolescent reading strategies more generally.

A sample of 52 schools (26 treatment/26 control) has been recruited from rural and small town high schools (with 50 to 1,000 students) in northern California, Idaho, Montana, Oregon, Washington, and Wyoming. Schools were randomly assigned to a treatment or control group defined by state (six states) and high or low poverty census area (two groups). Because the intervention works to help all core content teachers apply and collaborate on schoolwide reading comprehension strategies, random assignment occurs at the school level. With a sample size of 52 schools, the study has the statistical power to detect an effect as low as 0.21 standard deviation units.

The design is a cluster randomized trial. Such a design allows for an accurate assessment of effects on students while controlling for any school-level factors that could bias the estimates of student effectiveness. The unit of assignment is the school, and the primary unit of analysis is the student. Students are nested within schools, but not within a teacher or classroom, because each high school student will receive the CRISS treatment from several teachers across core subject classes. Outcome measures will be taken on two occasions: at a baseline period (fall) and after students have been fully exposed to Project CRISS teaching strategies (spring).

In the treatment schools teachers will receive intensive Project CRISS training during year 1 and follow-up services during year 2. In treatment year 2—by which time teachers should understand the principles and mechanics of Project CRISS strategies and be implementing them in the classroom—grade 9 students will be tested in the fall and spring on reading comprehension. To induce schools to participate in a study requiring random assignment, control schools will be offered the Project CRISS intervention two years later.

Key outcomes and measures

The sole outcome measure is student reading comprehension. The comprehension subtest of the Stanford Diagnostic Reading Test (SDRT), 4th edition, has been selected as the standardized measure across states to assess student outcomes. The SDRT-4 is a group-administered reading assessment designed to pinpoint student strengths and weaknesses in reading. The comprehension subtest will be used with items that measure the understanding of three kinds of text: recreational reading, informational reading, and reading applied text from everyday situations.

Descriptive data on how faithfully Project CRISS is implemented by schools and teachers will be collected in the treatment schools, with some parallel measures in the control schools. Two short online questionnaires will be administered to grade 9 teachers and principals in both treatment and control schools. Questions on teacher and principal background information, such as years of experience and highest degree held, will be included to describe general staff characteristics at the schools and determine whether there are differences between treatment and control schools. The questionnaires also ask about schoolwide professional development programs related to reading comprehension other than Project CRISS. Four questions ask the extent to which teachers and principals participate in Project CRISS-recommended activities in the treatment school (conducted by a Project CRISS local facilitator) or parallel activities in the control schools that might be organized by an in-school literacy coach. These measures will help compare Project CRISS schools with the business-as-usual control schools.

A main source of implementation data in the treatment schools will be a monthly electronic log of specific activities. The Project CRISS local facilitator will be asked to report on Project CRISS activities, such as meetings, coaching, and other activities supposed to occur in addition to day-long workshops conducted by the Project CRISS national trainer. The monthly data will be examined across two school years of Project CRISS professional development to construct an index of how well the program was implemented. Attendance data at training events and other documented information from the Project CRISS national trainer will also be gathered to assess whether implementation was full or partial.

Classroom observations will be used to compare teacher behaviors related to the learning principles and strategies that are the basis of Project CRISS in subsample of treatment and control schools. Because the cost of sampling an adequate number of teachers to estimate accurate teacher impacts is prohibitive, teacher behavior is treated as an implementation variable rather than an impact measure. The observational results will describe the extent to which research-based learning principles espoused and taught through Project CRISS are visible in schools after two years of Project CRISS training and technical assistance. Again, the Project CRISS schools will be compared with the business-as-usual control schools.

A version of the Vermont Classroom Observation Tool (VCOT) is being used because it measures teacher behaviors that are very similar to those expected from Project CRISS. This instrument was developed for and used in other IES research, though some minor adaptations were made for this study with the consultation of the developer. All observers will receive intensive training and certification by the VCOT developer.

Data collection approach

The SDRT-4 comprehension subtest will be group administered to all grade 9 students in treatment and control schools in September and April of the school year in which Project CRISS teachers are in year 2 of professional development and implementation. At this time teachers should be fully familiar with the CRISS principles and have specific strategies for integrating them into their content areas based on the first year of intensive training and support. Test administrators at the Regional Educational Laboratory Northwest will work with local test coordinators to administer the 50-minute reading comprehension test. The study includes two cohorts of schools: one beginning Project CRISS in the 2007/08 school year, and another in following school year, 2008/09. Student data collection for the study will occur during the 2008/09 school year (cohort 1) and 2009/10 school year (cohort 2). Results from the combined cohorts (26 treatment/26 control) will be analyzed, and an impact analysis will be completed in 2010.

The teacher and principal questionnaires will be administered in the spring of year 1 and year 2 of Project CRISS teacher professional development. The local facilitator log collects monthly data throughout the two years of professional development. These monthly measures on specific Project CRISS activities will be aggregated into an overall school implementation index ranging from high to low.

VCOT teacher observations will occur at two times in a subsample of classrooms in treatment and control schools: at the beginning of the two-year training (to establish baseline equivalence) and during the second half of year 2 of Project CRISS continuing professional development. A subsample of 20 of the 52 schools will be randomly selected for observations (10 treatment/10 control). Five teacher/class periods will be randomly selected across content areas in each school, and each teacher will be observed for an entire class period on two successive days.

Analysis plan

The analysis of the reading achievement student outcome will have two components. Outcome data will first be subjected to a series of descriptive analyses to examine the general characteristics of the data and determine whether they meet typical assumptions for more sophisticated analyses. Outcome data will then be subjected to more sophisticated statistical analysis to infer whether there are statistically significant program effects on students, making appropriate corrections for the nesting of students in schools (the unit of random assignment). The estimation of an effect size will be expressed in standard deviation units so that Project CRISS can be compared with similar reading comprehension programs that have undergone rigorous scientific study.

Principal investigators

Jim Kushman, PhD
Regional Educational Laboratory Northwest

Contact information

Jim Kushman, PhD
Regional Educational Laboratory Northwest
101 SW Main Street, Suite 500
Portland, OR 97204
Voice: (503) 275.9621
Fax: (503) 275-9621
Email: kushmanj@nwrel.org

Region: Northwest

References

Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice-Hall.

Biancarosa, G. and Snow, C.E. (2004). Reading next—a vision for action and research in middle and high school literacy. A Report from Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education.

Bloom, B. (1950). Problem solving process of college students. Supplemental educational monographs, 73.

Brown, A., Pressley, M., Van Meter, P., and Schuder, T. (1996). A quasi-experimental validation of transactional strategies instruction with low achieving second graders. Journal of Educational Psychology, 88, 28–37.

Brune, J. (1977). The process of education. Cambridge, MA: Harvard University Press.

Duffy, G., Roehler, L., Sivan, E., Rackliffe, G., Book, C., Meloth, M., Vavrus, L., Wesselman, R., Putman, J., and Basiri, D. (1987). Effects of explaining the reasoning associated with using reading strategies. Reading Research Quarterly, 22, 347–68.

Duke, N.K., and Pearson, P.D. (2002). Effective practices for developing reading comprehension. In A.E. Farstrup and S.J. Samuels (Eds.), What research has to say about reading instruction. Newark, DE: International Reading Association.

Palincsar, A.M., and Brown, A. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 2, 117–75.

Santa, C. M (1993). Program abstract. Project CRISS. Retrieved November 3, 2008, from http://www.projectcriss.com/pdf_files/1_3-EVIDENCE-1993.PDF.

Santa, C. M. (1995). Evidence of effectiveness. Project CRISS. Retrieved November 3, 2008, from http://www.projectcriss.com/pdf_files/1_2-EVIDENCE-1995.PDF.

Santa, C. M. (2004). Project CRISS: evidence of effectiveness. Project CRISS. Retrieved November 3, 2008, from http://www.projectcriss.com/pdf_files/1_1-EVIDENCE-2004.PDF.

Simon, H. A. (1979). Models of thought. New Haven, CT: Yale University Press.

Vygotsky, L. (1978). Mind in society: the development of higher mental processes. Cambridge, MA: Harvard University Press.

Return to Index