Co-Principal Investigators: Inns, Amanda; Pape, Stephen

Purpose: For this project, researchers will carry out a meta-analysis of rigorous evaluations of approaches that use technology to improve student mathematics achievement in grades K to 12. Using the meta-analytic techniques, researchers will seek to identify conditions under which various types of technology applications are most effective in teaching mathematics in grades K to 12. The results t will provide researchers and education leaders with up-to-date information on effective uses of technology, including computer assisted instruction, cooperative learning, intelligent tutoring systems, games, simulations, virtual reality, inquiry/discovery, project-based learning, and media-infused instruction.

Project Activities: The researchers will conduct a systematic review of all experimental and quasi-experimental evaluations of educational technology approaches with impacts on mathematics outcomes. The team will use modern means of accounting for key moderators during analysis.

Products: The research team will develop peer-reviewed articles targeting researchers, as well as articles aimed at practitioner-focused journals. All publications will be open source. Additional dissemination materials will include conference presentations and summaries shared through blogs and newsletters

Structured Abstract

Setting: Included studies will report evaluations of interventions in grades K–12 education settings and locations across the U.S. and similar countries such as Canada, Europe, Israel, Australia, and New Zealand.

Sample: The review will include studies of students in grades K to 12, in all types of schools in the U.S. and similar countries.

Factors:  Included studies will have evaluated mathematics achievement outcomes of all types of approaches making extensive use of technology, such as computer assisted instruction, cooperative learning, intelligent tutoring systems, games, simulations, virtual reality, inquiry/discovery, project-based learning, and media-infused instruction. Researchers will examine study conditions that include characteristics of students (e.g., age, socioeconomic status), instructional focus (e.g., arithmetic, geometry, problem solving), and type of community (urban, suburban, rural; disadvantaged/advantaged). The researchers will use modern meta-analytic methodologies including rigorous inclusion criteria similar to those used by the What Works Clearinghouse and meta-regression to identify key conditioning factors.

Research Design and Methods: The meta-analysis method to be used comprises five key steps: (a) retrieve all potential studies; (b) screen and review studies by pre-set criteria; (c) code data and features of qualified studies; (d) compute effect sizes; and (e) implement statistical analyses. To be included in the meta-analysis, studies must be experimental or quasi-experimental evaluations of educational technology approaches with impacts on mathematics outcomes studies. Studies also must have a duration of at least 12 weeks. Qualifying studies will be accepted from North America, Europe, Israel, Australia, and New Zealand. Studies must use "intent to treat" designs, meaning that all students in the original sample are included in the final analysis. All coding will be done by independent pairs of reviewers, who will discuss disagreements and come to consensus, involving other authors if needed.

Control Condition: Included studies must include a business-as-usual or alternative treatment condition.

Key Measures: The key outcomes in this project are quantitative measures of mathematics. For each study there may be multiple outcomes in multiple domains of mathematics. All outcomes from each study will be coded for inclusion.

Data Analytic Strategy: The research team will use a multivariate meta-regression model with robust variance estimation to conduct the meta-analysis. First, the researchers will estimate a null model to produce the average effect size for studies of technology programs. Second, the research team will estimate the identified malleable factors and moderators of interest and covariates using a meta-regression model. To better understand the heterogeneity in the effect sizes, in addition to the means, the researchers will calculate 95% prediction intervals around the mean effect sizes for all studies.