Skip Navigation

Regional Educational Laboratory Program

Exploring How Methodological Decisions Affect the Variability of Schools Identified as Beating the Odds


Description: Although school performance that exceeds expectations can be defined at the broad conceptual level, there is no one right approach to identifying these schools. This report considers one state's approach to identifying schools that are performing better than expected —called "beating the odds" or BTO schools in this report—and examines how a set of schools labeled as BTO might change when the approaches to identification change. The state uses two statistical methods to designate schools as BTO. The first method identifies a school if it outperforms its predicted level of performance given school demographics; the second method identifies a school if it outperforms other demographically similar schools. This report investigates how the identification results vary, under each of the two statistical methods.

Research Questions: This study will address the following three research questions:

  1. How do the BTO identification results produced by a given method for a given year vary when alternative measures and school configurations are used?
  2. How do the BTO identification results produced by the prediction method (schools outperforming their predicted performance) vary from those produced by the comparison method (schools outperforming other demographically similar schools) for a given year?
  3. How do the BTO identification results produced by a given method vary from year to year?

Study Design: This study applies baseline and alternative specifications of the state's methods to generate lists of BTO schools. For a given method and year, the study generates a list of BTO schools based on the baseline model, as established by the state. Then, the study generates BTO school lists by altering just one of three specifications—school performance measures, school configuration, and school characteristic indicators—at a time while keeping the other two specifications unchanged from the baseline model before generating another list by applying all three alternative specifications at the same time. Finally, the study compares the BTO lists generated under the alternative models with the baseline model to gauge the extent that the identification of schools varies given a method and a year.

Projected Release Date: Summer 2014

Research Alliance: Beating the Odds, REL Midwest

Keywords: School Improvement, High Performing School, Data Use and Systems, Michigan

Study-Related Products: Applied Research Methods report

Principal Investigators & Affiliation: