The Effects of Hybrid Algebra I on Teaching Practices, Classroom Quality, and Adolescent Learning

Regional need and study purpose

The 2005 Appalachia Regional Advisory Committee (RAC) report submitted to the U.S. Department of Education listed improving teacher quality and identifying evidence-based curricula and programs as two of the top five areas of need in the Appalachia Region (CNA Corporation 2005). These priorities were confirmed by Regional Educational Laboratory Appalachia's need-sensing activities. Rural areas are often economically depressed and lack the cultural and social activities to which teaching candidates from more urban settings are accustomed, so they have difficulty attracting teachers (McClure, Redfield, and Hammer 2003). This leaves low-performing rural schools with a limited supply of possible teachers—many of whom were educated in the same low-performing schools. The RAC report also indicates that math instruction and technology use are important areas of need in the Appalachia Region. Algebra I is of particular interest because it is a gateway course to higher level math and science.

Kentucky's hybrid algebra I program includes sustained professional development for teachers as well as instructional materials and recommended classroom practices. The intervention uses a hybrid, or blended, approach to instruction: students are in a traditional face-to-face classroom setting but regularly use online algebra I instructional software. Teacher professional development is also hybrid, including four days of blended instruction in face-to-face summer sessions and five online sessions. The monthly facilitated online discussions provide teachers with an opportunity to share insights, problems, and solutions as they implement the blended approach in their grade 9 algebra I classrooms. During the school year, instructional specialists make two site visits to each school to help individual teachers implement the program in their classrooms.

The study addresses three main questions:

• Does the hybrid algebra I program increase student achievement in learning algebra I content and skills?
• Does the hybrid program increase the use of research-based best practices for algebra I as documented in the algebra I instructional standards? ¹
• Does the hybrid program improve classroom quality, as seen in greater student interest and engagement and in more academically focused class time?

The study also addresses two subsidiary questions:

• Do the achievement gains associated with the online curriculum vary among students with different characteristics (gender, race/ethnicity, eligibility for free or reduced-price lunch, limited English proficiency, learning disabilities/special education, and age)?
• Are the effects observed during the treatment period sustained in the posttreatment period?

Without a randomized study, information would not be available to assess the effectiveness of this approach—and whether its effectiveness justifies its expansion. Findings from the study will provide educators and policymakers with evidence of the effectiveness of this growing classroom practice.

Intervention description

The intervention uses online resources in face-to-face technology-enhanced classrooms to facilitate the use of algebra I standards-based instructional practices and to improve student learning in grade 9 algebra I classrooms. Each student spends at least 40 percent of class time using online courseware (such as two days a week in a computer lab for a course that meets five days a week or 40 percent of each period when classroom computers are available). As students use the computers, the teacher acts as a coach, assisting individual students or providing mini-lessons to larger groups of students as needed. The use of blended instructional practices is expanding rapidly in Kentucky and nationally. Although the estimated benefits vary widely depending on the by specific interventions examined, Waxman, Lin, and Michko (2003) reported a weighted mean effect size of 0.448 for cognitive outcomes, based on a metaanalysis of 42 studies.

Teachers have access in their classrooms to online instructional resources for direct instruction. Participating teachers engage in sustained professional development focusing on effective algebra I pedagogy and the use of technology to improve instructional practices and student learning. Professional development takes place in blended classrooms as well as online. It begins in the summer and continues through the school year, with monthly online facilitated discussions among participating teachers and two follow-up classroom visits by math instructional specialists.

The professional development is grounded in strategies that research indicates are effective. The program is school based and job embedded, continual and ongoing, content-focused, organized around groups of teachers, designed around active learning, and coherent—that is, the program is aligned with the state education system's goals for content and performance (Desimone et al. 2002; Elmore and Burney 1999; Fullen 2001; Garet et al. 2001; Joyce and Calhoun 1996; Joyce and Showers 1988; Loucks-Horsley et al. 1998; Supovitz, Mayer, and Kahle 2000; Supovitz and Turner 2000).

The Kentucky Department of Education selects and furnishes the online resources for the study through Kentucky Virtual School, which it operates. These resources include algebra I courseware for students, distributed by the National Repository of Online Courses, and professional development courseware developed and distributed by the Southern Regional Education Board. In addition, the Department provides teachers with training to use the online resources in their classrooms and enrolls teachers and students in its courses to access the online course materials. Math instructional specialists from the Collaborative for Teaching and Learning, a Louisville-based professional development provider, facilitate the professional development.

Study design

The study uses a two-cohort research design. In year 1, 25 Kentucky schools offering algebra I to students in grade 9 were recruited and randomly assigned to either 13 treatment or 12 control schools. In year 2, 16 new schools were recruited. In addition, six control schools from cohort 1 were re-randomized and included in cohort 2, to create a total sample of 47 schools across the two cohorts. Allowing for as much as 20 percent attrition from the randomized groups yields a conservatively projected sample of 38 schools, with an estimated 6,000 students, equally divided between treatment and control status. The minimum detectable effect size is 0.22.

All algebra I classes in a treatment school are assigned to their school's treatment condition. Most of the schools are in rural districts. More than 90 percent of students in the participating schools are White, and almost half are eligible for free or reduced-price lunch. Year 1 of the intervention includes teacher professional development that begins in the summer and continues throughout the school year as teachers implement the instructional practices. In year 2 of the intervention, treatment schools continue to follow the hybrid instructional model and have access to resources and technical support from the Kentucky Department of Education. But no formal professional development takes place after year 1. This is a year-long intervention that schools are permitted to use without any additional cost for two years.

Key outcomes and measures

The primary measures of the intervention's impact on student outcomes are designed to capture both the immediate impact on student achievement in algebra I and longer term effects on achievement and educational attainment. Outcome measures include an end-of-course test in algebraic understanding administered by trained site researchers in late spring of the intervention year, as well as the math portion of the PLAN® test—given to all Kentucky students in the fall of grade 10. The researchers use administrative records for intervention-year students from the Kentucky Department of Education to construct their enrollment rates in higher level math courses as well as high school continuation rates, in the postintervention school year. End-of-course tests are also given to assess the algebra I achievement of grade 9 students in the second year group in treatment and control schools. This assessment, given in the postintervention school year, provides an estimate of the longer term impact of the intervention on teaching effectiveness measured in terms of student achievement.

The study requires four types of data collection:

• Administrative data.
• Classroom observations.
• Brief one-time surveys of teachers.
• Assessments of student algebra I knowledge and skills.

Student administrative records. Using state administrative records, the researchers collect information on student enrollment in grade 9 algebra I, demographic attributes, and math test scores on statewide assessments given to eighth-graders in the year before the intervention. They also collect student-level indicators of academic outcomes, including subsequent enrollment in higher level math courses, scores on state-administered grade 10 math assessments, and school continuation for the year following the intervention (grade 10 for most students).

Classroom observations. Trained researchers observe (for about one hour) up to five algebra I classes during a one-day visit to each treatment and control school. To collect data during these visits, two valid and reliable instruments are used: the School Observation Measure collects data on overall classroom activities, and the Algebra I Quality Assessment records use of algebra I standards-based instructional practices.

Teacher surveys. The researchers use the Hybrid Algebra I Teacher Questionnaire and the Algebra I Control Teacher Questionnaire to collect teacher perceptions of their algebra I approach (hybrid compared with district curriculum) and of the algebra I standards-based instructional practices listed on the Algebra I Control Teacher Questionnaire.

Test scores on an end-of-course algebra assessment. To measure student knowledge of algebra I, the researchers use a nationally recognized, reliable paper-and-pencil 25-item test. They give the 40-minute test in all treatment and control schools during regular class times.

Other data. Math instruction specialists document treatment teachers' participation during professional development sessions. Archives from Blackboard, the platform housing the online courseware, track the number of student connections to online course materials during the school year.

Analysis plan

The study focuses on inferential analyses that can identify statistically significant differences in outcomes between the treatment and control groups. But the analysis plan also includes qualitative analyses of open-ended responses on teacher questionnaires and descriptive comparisons of the treatment and control samples.

Classroom observations and teacher surveys. Effects on teaching practices and classroom quality are evaluated through a classroom observation and short teacher survey, correcting estimates of statistical significance for multiple indicators of these effects. The data collection instruments provide nominal, ordinal, and qualitative data. Researchers use the Wilcoxon-Mann-Whitney test—a nonparametric method for comparing two independent groups—to analyze ordinal data. Although the Wilcoxon test draws statistical inference from the rank sum instead of the mean of each group, the researchers also provide the mean and the associated effect size. The nominal data are analyzed with chi-square tests of independence. Where appropriate, a multivariate analysis of variance and follow-up univariate analyses are conducted to compare treatment and control responses.

Student achievement. Using two-level hierarchical linear models, in which students are nested within schools, the study assesses how the intervention effects student achievement and educational attainment. The baseline models control only for pretest scores and the treatment condition at the school level. More comprehensive models control for individual student characteristics, including student eligibility for free and reduced-priced lunch, racial/ethnic minority status, and student age. Where individual student-level characteristics are missing, researchers use the sample average for the school. Where the pretest score is missing, researchers add a zero-one indicator—one if the pretest score is constructed, and zero otherwise. For students with a constructed pretest score, the sum of coefficients from the pretest score and this zero-one indicator provides an estimate of the association between the pretest and posttest.

To determine the intervention's effect on student outcomes, the researchers use an intent-to-treat design. Both school and student attrition are reported, and comparisons between the analysis sample and participants who left the sample are conducted. Students entering study schools after October 1 are not included in the study sample. And those who switch between study schools during the school year are analyzed with their original school.

Principal investigators

Linda Cavalluzzo, PhD
CNA

Deborah Lowther, PhD
Education Innovations

Contact information

Linda Cavalluzzo
cavallul@cna.org
(703) 824-2197

Region: Appalachia

References

CNA Corporation. (2005). A Report to the U.S. Department of Education on Educational Challenges and Technical Assistance Needs for the Appalachia Region. Alexandria, VA: U.S. Department of Education.

Desimone, L., Porter, A., Garet, M., Yoon, K., and Birman, B. (2002). Effects of professional development on teacher's instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24 (2), 81–112.

Elmore, R., and Burney, D. (1999). Investing in teacher learning: Staff development and instructional improvement. In L. Darling-Hammond and G. Sykes (Eds.), Teaching as the learning profession: Handbook of policy and practice. San Francisco, CA: Jossey-Bass.

Fullen, M. (2001). The new meaning of educational change. New York: Teachers College Press.

Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K. (2001). What makes professional development effective? Results from a national sample of teachers. American Education Research Journal, 38 (4), 915–45.

Joyce, B., and Calhoun, E. (1996). Learning experiences in school renewal: An exploration of five successful programs. Eugene, OR: University of Oregon. ERIC Clearinghouse on Educational Management, 5207.

Joyce, B., and Showers, B. (1988). Student achievement through staff development. White Plains, NY: Longman Press.

Loucks-Horsley, S., Hewson, P., Love, N., and Stiles, K. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press.

McClure, C. T., Redfield, D., and Hammer, P. C. (2003, December). Recruiting and retaining high-quality teachers in rural areas (AEL Policy Brief). Retrieved from, http://www.edvantia.org/publications/index1.cfm?§ion=publications&area=publications&id=482.

Supovitz, J., Mayer, D., and Kahle, J. (2000). Promoting inquiry-based instructional practice: The longitudinal impact of professional development in the context of systemic reform. Educational Policy, 14 (6), 331–56.

Supovitz, J., and Turner, H. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37, 963–80.

Waxman, H.C., Lin, M., and Michko, G. (2003). A Meta-analysis of the Effectiveness of Teaching and Learning with Technology on Student Outcomes. Learning Point Associates.

Return to Index