**Regional need and study purpose**

As the use of online courses continues to grow nationwide, more information is needed about the effectiveness of using online courses to expand schools' course offerings and student access to key gateway courses—introductory courses that are the foundation for further study in a field. This study investigates the use of an online course to expand access to algebra I to students in grade 8 who are ready to take the course but unable to do so because their schools do not offer a full algebra I course to eighth graders (often because the schools are small or in rural areas).

Eighth graders who take algebra I—a well documented gateway course (Atanda 1999; Kilpatrick, Swoffard, and Findell 2001)—are more likely to participate in advanced mathematics courses in high school (Stevensen, Schiller, and Schneider 1994). Yet more than 10 years after the U.S. Department of Education recommended expanding access to algebra I in middle schools, schools throughout the Northeast and Islands Region and the rest of the country are delivering algebra courses to only a small proportion of their eighth graders (U.S. Department of Education 1997). According to the 2007 National Assessment of Educational Progress (NAEP), 32 percent of eighth graders nationwide were enrolled in algebra I (U.S. Department of Education, National Center for Education Statistics 2007b^{1}). In the Northeast and Islands Region states, less than 25 percent of eighth graders take algebra I.

Online algebra courses offer a means for broadening access to algebra I in middle schools, especially where recruiting and retaining highly qualified teachers is difficult, such as in Maine and Vermont, which have many schools in rural areas. According to a 2007 report by the Rural School and Community Trust Policy Program, Maine has the highest percentage of students attending rural schools nationwide, at 52.9 percent, and Vermont has the second highest, at 51.3 percent (Johnson and Strange 2007). Rural schools face special challenges in offering a full range of courses due to limited resources, difficulties attracting and retaining qualified teachers, and small student bodies (Hannum et al. 2009). In Vermont only 22 percent of eighth graders take algebra I, and in Maine only 23 percent do. Virtual courses offer a promising solution: ensuring that students in small schools and isolated communities have access to critical courses, especially in science, technology, engineering, and mathematics (Tucker 2007).

The goal of this randomized controlled trial is to ascertain the effects of online algebra I on the achievement and subsequent course-taking patterns of eighth graders in schools that do not typically offer algebra I. In particular, this study addresses the following policy question: In schools that do not typically offer algebra I to eighth graders, is it beneficial to offer it as an online course, in terms of student achievement in mathematics and subsequent high school mathematics course-taking? This broad policy question is addressed through two primary and four secondary research questions.

Two primary research questions focus on the direct effects of the intervention on algebra-ready students, asking: What is the impact of offering algebra I online to algebra-ready students on

- Algebra-ready students' end of grade 8 algebra achievement?
- Algebra-ready students' subsequent high school course-taking patterns?

The direct effects of this intervention on such outcomes are key to determining whether this intervention is an effective way for schools to broaden access to algebra I.

Four secondary research questions focus on possible indirect effects of this intervention on algebra-ready students and their peers who were not considered ready for algebra. These questions are designed to test whether implementing online algebra for algebra-ready students has significant side effects (or unintended consequences). Implementation of the online course requires removing algebra-ready students from their general grade 8 mathematics class. This may affect algebra-ready students' general mathematics scores—an effect that is important to teachers and administrators concerned with student achievement on end of grade 8 accountability assessments. Removing algebra-ready students may also indirectly affect other students who are not considered ready for algebra I. Although these other students do not take the online algebra course, their mathematics instruction may be affected by the presence of the online course in their schools. For example, removing the algebra-ready students from the regular class may yield smaller grade 8 mathematics classes that are more tracked by ability than usual.

Four secondary research questions examine these indirect effects, asking: What is the impact of offering algebra I online to algebra-ready students on

- Algebra-ready student's end of grade 8 general mathematics achievement?
- Other students' end of grade 8 algebra achievement?
- Other students' end of grade 8 general mathematics achievement?
- Other students' planned grade 9 mathematics courses?

Answering these primary and secondary research questions will generate useful information about what happens to the entire population of eighth graders—including potential benefits and detriments—when a school uses an online course to give algebra-ready students access to algebra I. The observed impacts on the outcomes of algebra-ready and other students will help to determine whether the intervention is an effective choice for schools that do not typically offer algebra I to eighth graders.

This study is designed to produce policy-relevant, foundational information about the effectiveness of using an online course to provide access to a gateway algebra course that is not otherwise available. It will not provide information about the relative effectiveness of online algebra I courses compared with traditional face-to-face algebra I courses. The generalizability of the findings to different contexts will be presented with caution. In particular, the findings here will be interpreted by carefully considering the many aspects of the intervention tested, including the content of the course and the provision of highly qualified mathematics instructors.

^{1}Percentages are for grade 8 students enrolled in public schools who took the 2007 National Assessment of Educational Progress student survey. Students were considered enrolled in algebra I if they were enrolled in either a one-year algebra I course or the second year of a two-year algebra I course.

**Intervention description**

The algebra I course chosen for use in this study was developed by teams from the University of Nebraska funded by a 1996 U.S. Department of Education Star Schools grant. Class.com was created to make the courses available on an ongoing basis. Founded in 1999, Class.com is a privately held small business that partners with more than 4,400 high schools across the country.

The content of the online algebra I course is consistent with the mathematics content standards in Maine and Vermont and contains the same instructional components as a traditional class: defined learning objectives, curriculum materials, assignments, problem sets, quizzes, tests, and grades. The online teacher presents information, assigns problems, provides feedback, assesses students' learning, responds to their questions, and monitors and guides their progress. Students are expected to follow a schedule in completing each lesson and to prepare for quizzes and tests as scheduled.

Although similar in many ways, this online course differs from a traditional class in that students and the teacher in the online course communicate through asynchronous online exchanges. This means that students and the online teacher do not need to be, and typically are not, online at the same time. Most of the study's online classrooms comprised students from multiple schools who logged into the course at various times of the day. Typically, the teacher reviewed student work, replied to student messages and questions, and sent feedback to students at another time of day. Such learning, with its flexibility in place and time, is called anytime, anyplace learning. However, study schools scheduled a regular class period for students to access the online course.

Although the online teacher was responsible for the class, each participating school also had an onsite proctor who had daily direct contact with each student. Some proctors were mathematics teachers, and others were non-mathematics related school staff (such as a technology coordinator, librarian, or computer teacher). The proctor ensured that students had access to the technology required for the course, administered exams, supervised students' behavior, and served as a personal contact for students and parents when needed.

**Study design**

This study targeted schools in Maine and Vermont that do not offer a full algebra I course to eighth graders (meaning, they lack a separate section for algebra I with a dedicated teacher who serves all algebra-ready eighth graders). Seventy schools were recruited (50 in Maine, 20 in Vermont) that together served 2,030 eighth graders during the 2008/09 school year.

Using standardized test scores, student grades in the grade 7 math course, and teacher recommendations, schools decided which students were ready for algebra at the end of grade 7. The goal was to identify students for whom schools would offer the online algebra I course if it were available. Before random assignment, schools identified 479 entering grade 8 students as ready for algebra. Of these students, 23 moved away during the summer and were no longer attending the study schools as of August 2008. As of fall 2008, there were 456 algebra-ready students in the participating schools.

Schools identified an average of 6.8 algebra-ready students per school. With 70 schools, the study has the statistical power to detect effects on student achievement and high-school course-taking outcomes of 0.25–0.27 standard deviation or greater.^{2}

Schools were randomly assigned to one of two conditions: implementing the online algebra I course by offering it to their algebra-ready students (the intervention condition) or not (the control condition) during 2008/09. Schools in the control condition implemented their typical mathematics programs for eighth graders.

Schools were randomly assigned in blocks by state (Maine, Vermont) and school size: small (fewer than 17 eighth graders), medium (17–70), and large (more than 70). The table below breaks down schools by states, school size, and study condition.

**Random assignment by number of schools**

School size |
Maine (N = 50) |
Vermont (N = 20) |
||

Intervention schools | Control schools | Intervention schools | Control schools | |

Small (< 17 students) | 12 | 12 | 5 | 5 |

Medium (17–70 students) | 11 | 11 | 2 | 3 |

Large (> 70 students) | 2 | 2 | 3 | 2 |

*Source:* Reseachers' analysis based on data described in text.

^{2} To maintain the joint probability of falsely rejecting the null hypothesis for the two primary research questions at 5 percent, a multiple comparison adjustment will be used for the analyses of impacts on algebra achievement and high school course-taking for algebra-ready students.

**Key outcomes and measures**

The primary outcomes for this study are end of grade 8 algebra achievement scores and early high school (grades 9 and 10) mathematics course-taking. In addition, school, student, and teacher contextual information is collected and implementation fidelity and classroom instruction are evaluated using classroom observations.

Mathematics achievement scores. The pretest—a 30-item computer adaptive mathematics assessment (the Promise Assessment)—was administered to all grade 8 students in study schools in fall 2008 (year 1), when the student cohort started grade 8. Pretesting was completed by mid-October, with response rates of 96 percent for algebra-ready students and 95 percent for other students. Pretest items were drawn from a 1,000-item test bank of general mathematics items, with difficulty levels ranging from grades 5 to 8. The pretest scores will serve as a baseline covariate in analyzing the intervention's effects.

The posttest was administered at the end of grade 8 in spring 2009 to all grade 8 students in participating schools. Like the pretest, the posttest was a computer adaptive test (the Promise Assessment) containing 40 items equally divided into two sections. The first section measured general mathematics achievement and drew from the same item bank as the pretest. The second section measured algebra achievement and drew from a test bank of 270 algebra items. Response rates on the posttest were 96 percent for algebra-ready students and 93 percent for other students, accounting for student attrition during the 2008/09 school year.

**Grades 9 and 10 course-taking information.** Course information will be collected for two measures of high school course-taking: planned grade 9 courses (based on end of grade 8 enrollment) and a composite measure of high school course sequences based on grade 9 course types and grades and planned grade 10 courses. These measures will indicate whether students are participating in an advanced mathematics course sequence in high school.

Information was collected on planned grade 9 courses from all students—both algebra-ready and other students. At the end of grade 8 (year 1, spring 2009), information was collected on the grade 9 mathematics courses in which students had enrolled for fall 2009. Based on the Classification of Secondary School Courses (CSSC) and the methods of the 2005 National Assessment of Educational Progress High School Transcript Study, grade 9 courses will be coded as either at or below algebra I or above algebra I (U.S. Department of Education 2007a; Shettle et al. 2008). This indicator will serve as the primary variable in analyzing the impact of the intervention on planned high school course-taking.

Follow-up information for the composite measure of high school course sequences will be collected only from algebra-ready students. At the end of grade 9 (year 2, spring 2010), the title of the mathematics course that each algebra-ready student completed in grade 9, their grade, and their planned grade 10 course will be recorded. Again using the CSSC, the course type information will be coded, and a composite measure will be derived that indicates whether each algebra-ready student is in an advanced mathematics course sequence. Based on work by Schneider, Swanson, and Riegle-Crumb (1998), an advanced course sequence is defined as the completion of geometry with a grade of at least a C in grade 9 and enrollment in algebra II in grade 10 (see also Stevenson, Schiller, and Schneider 1994^{3}). This binary indicator is the primary variable for analyzing whether students with access to online algebra in grade 8 are more likely than those without access to participate in an advanced course sequence in high school.

**School, teacher, and student contextual information.** School- and student-level demographic data were obtained from administrative records from the states, districts, and schools. Information about students' attitudes toward and engagement in mathematics were collected from a student survey, also administered at the end of year 1. Teacher background characteristics were collected from surveys administered to all grade 8 mathematics teachers (including online teachers) in study schools at the end of year 1. The survey also provided information about teachers' classroom practices and measured the amount of algebra taught to eighth graders.

**Implementation fidelity and classroom instruction.** A random sample of 10 intervention and 10 control school classrooms was observed twice during year 1 of the study. The classroom observations were used to provide implementation information that will help ground the analytical findings and to gather contextual information about how teacher instruction and student engagement differ between an online classroom and a regular grade 8 mathematics classroom. These observational data will be used to help interpret findings and to generate relevant examples for the final report.

^{3} In high schools with a typical U.S. course sequence of algebra I -> geometry -> algebra II -> precalculus/trigonometry -> calculus.

Most of the data for this study were collected electronically from the study schools—including the pretest and posttest scores and end of grade 8 student and teacher surveys. Administrative data (including prior mathematics achievement test scores and student and school demographics) were collected from state departments of education and local districts. Initial high school course-taking data (planned grade 9 courses) were collected from participating middle schools. The follow-up high school course-taking data will be collected from the algebra-ready students' high schools. All coding of course-taking data will be conducted and housed electronically.

**Analysis plan**

The impact of online algebra will be assessed by comparing mathematics achievement and early high school course-taking outcomes for students from schools that offered the online course with those of students from schools that did not. Analyses will be conducted separately for algebra-ready students and students who were not identified by their schools as ready for algebra.

The impact of online algebra will be estimated using two-level random effects hierarchical linear models in which students are level 1 and schools are level 2. The basic analytic model will capture the difference in student outcomes between intervention schools and control schools, adjusting for student pretest scores, student characteristics (poverty status, special education status), and school characteristics (state, school size). Including student-level covariates will improve precision and adjust for possible differences in covariate values across conditions (Bloom, Richburg-Hayes, and Black 2005).

Confirmatory impact analyses will estimate the impact of online algebra on the mathematics achievement and course-taking for both algebra-ready and other students. The analyses following the year-long implementation of the online algebra course will estimate the short-term effects on mathematics achievement and initial course opportunities. The analyses conducted after the follow-up data collection will measure the long-term impact of having access to algebra I in grade 8 on subsequent course-taking patterns.

The analyses for primary research question 1 will estimate the impact of the intervention on algebra-ready students' algebra scores. A positive effect would indicate that algebra-ready students in intervention schools (with access to algebra I online) scored higher on the posttest than algebra-ready students in control schools, after controlling for pretest scores and relevant background characteristics. The impact estimate that captures the difference in achievement between algebra-ready students in intervention schools and those in control schools will be expressed as a standardized effect size, based on the standard deviation on the posttest for the control group. The model will also examine whether the effects of the intervention are uniform across student-level pretest scores. A cross-level interaction term (school-level intervention status by student-level pretest score) with a positive value would indicate that students with high pretest scores benefit more from the intervention than those with low pretest scores.

The analyses for primary research question 2 will estimate the impact of online algebra in grade 8 on the likelihood of students participating in an advanced course sequence in high school. This analysis will use a hierarchical generalized linear model, which is appropriate for binary outcomes (Raudenbush and Bryk 2002). The same control variables will be used as in the models for primary research question 1. A positive impact estimate would indicate that algebra-ready students from intervention schools are more likely to participate in advanced course sequences than algebra-ready students from control schools.

For the intervention to be considered successful, the impacts on algebra scores or high school course-taking must be positive. However, to obtain a more comprehensive assessment of effectiveness, further analyses will be conducted to address the secondary research questions.

The analyses for secondary research question 1 will mirror those for primary research question 1, using general mathematics achievement scores as the outcome. The estimated impact on general mathematics achievement will reveal whether algebra-ready students in intervention schools suffer any negative consequences compared with algebra-ready students in control schools.

The analyses for secondary research questions 2–4 address whether there are any negative consequences for students who remain in the general grade 8 mathematics class. The analyses for research questions 2 and 3 examine potential differences in algebra and general mathematics achievement between other students in intervention schools who were not considered ready for algebra and their counterparts in control schools where they participated with algebra-ready students in general grade 8 mathematics classes. The same hierarchical linear model equations are used for these analyses as those used to estimate impacts on algebra-ready students, with other students replacing algebra-ready students as the analytic sample.

Secondary research question 4 will use the same hierarchical generalized linear model as that for primary research question 2, with other students not ready for algebra as the analytic sample and planned grade 9 course types as the outcome. These analyses will determine whether the intervention has an effect on other students' likelihood of taking a grade 9 course below the level of algebra I (such as prealgebra), at the level of algebra I, or above the level of algebra I.

Together, these primary and secondary confirmatory impact analyses will generate policy-relevant findings about the overall impact of adding online algebra I to the course offerings of a school's grade 8 mathematics program.

In general, the intervention will be considered an effective way to broaden access if estimated impacts are positive for algebra-ready students' algebra scores or high school course-taking outcomes and not negative for algebra-ready students' general mathematics scores, other students' algebra and general mathematics scores, and other students' planned grade 9 courses. That is, the intervention will be considered successful if it yields benefits for algebra-ready students on primary outcomes, without significant negative consequences for them on other outcomes or on their peers who were not considered ready for algebra. Descriptive analyses documenting implementation of the intervention will augment these confirmatory impact analyses.

**Principal investigators**

Margaret Clements, PhD, Education Development Center, and Teresa Duncan, PhD, American Institutes for Research

**Contact information**

Dr. Jessica Heppen

Study Director

American Institutes for Research

1000 Thomas Jefferson Street, NW

Washington, DC 20007

Phone: (202) 403-5488

Fax: (202) 403-5188

Email: jheppen@air.org

**Region**: Northeast and Islands

**References**

Atanda, R. (1999). *Do gatekeeper courses expand education options?* (NCES 1999-303). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

Bloom, H.S., Richburg-Hayes, L., and Black, A.R. (2005). *Using covariates to improve precision: Empirical guidance for studies that randomize schools to measure the impacts of educational interventions.* New York: MDRC.

Hannum, W.H., Irvin, M.J., Banks, J.B., and Farmer, T.W. (2009). Distance education use in rural schools. *Journal of Research in Rural Education, 24* (3). Retrieved July 9, 2009, from http://jrre.psu.edu/articles/24-3.pdf.

Johnson, J., and Strange, M. (2007). *Why rural matters 2007: The realities of rural education growth*. Arlington, VA: The Rural School and Community Trust. Retrieved July 13, 2009, from http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/36/26/08.pdf.

Kilpatrick, J., Swoffard, J., and Findell, B. (Eds.). (2001). *Adding it up: Helping children learn mathematics.* Washington, DC: National Research Council, Center for Education, Division of Behavioral and Social Sciences and Education.

Raudenbush, S.W., and Bryk, A.S. (2002). *Hierarchical linear models: Applications and data analysis methods* (2nd ed.). Thousand Oaks, CA: Sage.

Schneider, B., Swanson, C.B., and Riegle-Crumb, C. (1998). Opportunities for learning: Course sequences and positional advantages. *Social Psychology of Education, 2*(1), 25–53.

Shettle, C., Cubell, M., Hoover, K., Kastberg, D., Legum, S., Lyons, M., Perkins, R., Rizzo, L., Roey, S., and Sickles, D. (2008). *The 2005 High School Transcript Study: The 2005 High School Transcript Study user's guide and technical report* (NCES 2009–480). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.

Stevenson, D.L., Schiller, K.S., and Schneider, B. (1994). Sequences of opportunities for learning. *Sociology of Education, 67* (3), 184–198.

Tucker, B. (2007). *Laboratories of reform: Virtual high schools and innovation in public education.* Washington, DC: Education Sector.

U.S. Department of Education. (1997, October 20). Mathematics equals opportunity. Retrieved September 15, 2005, from http://www.ed.gov/pubs/math/index.html.

U.S. Department of Education, National Center for Education Statistics. (2007a). *America's high school graduates: Results from the 2005 NAEP High School Transcript Study*. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.

U.S. Department of Education, National Center for Education Statistics. (2007b). *National Assessment of Educational Progress (NAEP), 2007 Mathematics Assessment*. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.