Search Results: (16-30 of 34 records)
| Pub Number | Title |  Date |
|---|---|---|
| REL 2016119 | Stated Briefly: How methodology decisions affect the variability of schools identified as beating the odds
This "Stated Briefly" report is a companion piece that summarizes the results of another report of the same name. Schools that show better academic performance than would be expected given characteristics of the school and student populations are often described as "beating the odds" (BTO). State and local education agencies often attempt to identify such schools as a means of identifying strategies or practices that might be contributing to the schools' relative success. Key decisions on how to identify BTO schools may affect whether schools make the BTO list and thereby the identification of practices used to beat the odds. The purpose of this study was to examine how a list of BTO schools might change depending on the methodological choices and selection of indicators used in the BTO identification process. This study considered whether choices of methodologies and type of indicators affect the schools that are identified as BTO. The three indicators were (1) type of performance measure used to compare schools, (2) the types of school characteristics used as controls in selecting BTO schools, and (3) the school sample configuration used to pool schools across grade levels. The study applied statistical models involving the different methodologies and indicators and documented how the lists schools identified as BTO changed based on the models. Public school and student data from one midwest state from 2007-08 through 2010-11 academic years were used to generate BTO school lists. By performing pairwise comparisons among BTO school lists and computing agreement rates among models, the project team was able to gauge the variation in BTO identification results. Results indicate that even when similar specifications were applied across statistical methods, different sets of BTO schools were identified. In addition, for each statistical method used, the lists of BTO schools identified varied with the choice of indicators. Fewer than half of the schools were identified as BTO in more than one year. The results demonstrate that different technical decisions can lead to different identification results. |
4/6/2016 |
| NCSER 2015002 | The Role of Effect Size in Conducting, Interpreting, and Summarizing Single-Case Research
The field of education is increasingly committed to adopting evidence-based practices. Although randomized experimental designs provide strong evidence of the causal effects of interventions, they are not always feasible. For example, depending upon the research question, it may be difficult for researchers to find the number of children necessary for such research designs (e.g., to answer questions about impacts for children with low-incidence disabilities). A type of experimental design that is well suited for such low-incidence populations is the single-case design (SCD). These designs involve observations of a single case (e.g., a child or a classroom) over time in the absence and presence of an experimenter-controlled treatment manipulation to determine whether the outcome is systematically related to the treatment. Research using SCD is often omitted from reviews of whether evidence-based practices work because there has not been a common metric to gauge effects as there is in group design research. To address this issue, the National Center for Education Research (NCER) and National Center for Special Education Research (NCSER) commissioned a paper by leading experts in methodology and SCD. Authors William Shadish, Larry Hedges, Robert Horner, and Samuel Odom contend that the best way to ensure that SCD research is accessible and informs policy decisions is to use good standardized effect size measures—indices that put results on a scale with the same meaning across studies—for statistical analyses. Included in this paper are the authors' recommendations for how SCD researchers can calculate and report on standardized between-case effect sizes, the way in these effect sizes can be used for various audiences (including policymakers) to interpret findings, and how they can be used across studies to summarize the evidence base for education practices. |
1/7/2016 |
| REL 2015077 | Comparing Methodologies for Developing an Early Warning System: Classification and Regression Tree Model Versus Logistic Regression
The purpose of this report was to explicate the use of logistic regression and classification and regression tree (CART) analysis in the development of early warning systems. It was motivated by state education leaders' interest in maintaining high classification accuracy while simultaneously improving practitioner understanding of the rules by which students are identified as at-risk or not at-risk readers. Logistic regression and CART were compared using data on a sample of grades 1 and 2 Florida public school students who participated in both interim assessments and an end-of-the year summative assessment during the 2012/13 academic year. Grade-level analyses were conducted and comparisons between methods were based on traditional measures of diagnostic accuracy, including sensitivity (i.e., proportion of true positives), specificity (proportion of true negatives), positive and negative predictive power, and overall correct classification. Results indicate that CART is comparable to logistic regression, with the results of both methods yielding negative predictive power greater than the recommended standard of .90. Details of each method are provided to assist analysts interested in developing early warning systems using one of the methods. |
2/25/2015 |
| REL 2015071 | How Methodology Decisions Affect the Variability of Schools Identified as Beating the Odds
Schools that show better academic performance than would be expected given characteristics of the school and student populations are often described as "beating the odds" (BTO). State and local education agencies often attempt to identify such schools as a means of identifying strategies or practices that might be contributing to the schools' relative success. Key decisions on how to identify BTO schools may affect whether schools make the BTO list and thereby the identification of practices used to beat the odds. The purpose of this study was to examine how a list of BTO schools might change depending on the methodological choices and selection of indicators used in the BTO identification process. This study considered whether choices of methodologies and type of indicators affect the schools that are identified as BTO. The three indicators were (1) type of performance measure used to compare schools, (2) the types of school characteristics used as controls in selecting BTO schools, and (3) the school sample configuration used to pool schools across grade levels. The study applied statistical models involving the different methodologies and indicators and documented how the lists schools identified as BTO changed based on the models. Public school and student data from one midwest state from 2007-08 through 2010-11 academic years were used to generate BTO school lists. By performing pairwise comparisons among BTO school lists and computing agreement rates among models, the project team was able to gauge the variation in BTO identification results. Results indicate that even when similar specifications were applied across statistical methods, different sets of BTO schools were identified. In addition, for each statistical method used, the lists of BTO schools identified varied with the choice of indicators. Fewer than half of the schools were identified as BTO in more than one year. The results demonstrate that different technical decisions can lead to different identification results. |
2/24/2015 |
| REL 2014064 | Reporting What Readers Need to Know about Education Research Measures: A Guide
This brief provides five checklists to help researchers provide complete information describing (1) their study's measures; (2) data collection training and quality; (3) the study's reference population, study sample, and measurement timing; (4) evidence of the reliability and construct validity of the measures; and (5) missing data and descriptive statistics. The brief includes an example of parts of a report's methods and results section illustrating how the checklists can be used to check the completeness of reporting. |
9/9/2014 |
| REL 2014014 | Developing a Coherent Research Agenda: Lessons from the REL Northeast & Islands Research Agenda Workshops
This report describes the approach that REL Northeast and Islands (REL-NEI) used to guide its eight research alliances toward collaboratively identifying a shared research agenda. A key feature of their approach was a two-workshop series, during which alliance members created a set of research questions on a shared topic of education policy and/or practice. This report explains how REL-NEI conceptualized and organized the workshops, planned the logistics, overcame geographic distance among alliance members, developed and used materials (including modifications for different audiences and for a virtual platform), and created a formal research agenda after the workshops. The report includes links to access the materials used for the workshops, including facilitator and participant guides and slide decks. |
7/10/2014 |
| REL 2014051 | Going public: Writing About Research in Everyday Language
This brief describes approaches that writers can use to make impact research more accessible to policy audiences. It emphasizes three techniques: making concepts as simple as possible, focusing on what readers need to know, and reducing possible misinterpretations. A glossary of common concepts is included showing the approaches applied to a range of concepts common to impact research, such as ‘regression models’ and ‘effect sizes.’ |
6/24/2014 |
| NCES 2013046 | U.S. TIMSS and PIRLS 2011 Technical Report and User's Guide
The U.S. TIMSS and PIRLS 2011 Technical Report and User's Guide provides an overview of the design and implementation in the United States of the Trends in International Mathematics and Science Study (TIMSS) 2011 and the Progress in International Reading Literacy Study (PIRLS) 2011, along with information designed to facilitate access to the U.S. TIMSS and PIRLS 2011 data. |
11/26/2013 |
| NCES 2013190 | The Adult Education Training and Education Survey (ATES) Pilot Study
This report describes the process and findings of a national pilot test of survey items that were developed to assess the prevalence and key characteristics of occupational certifications and licenses and subbaccalaureate educational certificates. The pilot test was conducted as a computer-assisted telephone interview (CATI) survey, administered from September 2010 to January 2011. |
4/9/2013 |
| NCES 2011463 | The NAEP Primer
The purpose of the NAEP Primer is to guide educational researchers through the intricacies of the NAEP database and make its technologies more user-friendly. The NAEP Primer makes use of its publicly accessible NAEP mini-sample that is included on the CD. The mini-sample contains real data from the 2005 mathematics assessment that have been approved for public use. Only public schools are included in this subsample that contains selected variables for about 10 percent of the schools and students in this assessment. All students who participated in NAEP in the selected public schools are included. This subsample is not sufficient to make state comparisons. In addition, to ensure confidentiality, no state, school, or student identifiers are included. The NAEP Primer document covers the following topics:
Also note that the NAEP Primer consists of two publications: NCES 2011463 and NCES 2011464 |
8/4/2011 |
| NCES 2011464 | NAEP Primer Mini-Sample
The purpose of the NAEP Primer is to guide educational researchers through the intricacies of the NAEP database and make its technologies more user-friendly. The NAEP Primer makes use of its publicly accessible NAEP mini-sample that is included on the CD. The mini-sample contains real data from the 2005 mathematics assessment that have been approved for public use. Only public schools are included in this subsample that contains selected variables for about 10 percent of the schools and students in this assessment. All students who participated in NAEP in the selected public schools are included. This subsample is not sufficient to make state comparisons. In addition, to ensure confidentiality, no state, school, or student identifiers are included. The NAEP Primer document covers the following topics:
Also note that the NAEP Primer consists of two publications: NCES 2011463 and NCES 2011464 |
8/4/2011 |
| NCES 2011049 | Third International Mathematics and Science Study 1999 Video Study Technical Report, Volume 2: Science
This second volume of the Third International Mathematics and Science Study (TIMSS) 1999 Video Study Technical Report focuses on every aspect of the planning, implementation, processing, analysis, and reporting of the science components of the TIMSS 1999 Video Study. Chapter 2 provides a full description of the sampling approach implemented in each country. Chapter 3 details how the data were collected, processed, and managed. Chapter 4 describes the questionnaires collected from the teachers in the videotaped lessons, including how they were developed and coded. Chapter 5 provides details about the codes applied to the video data by a team of international coders as well as several specialist groups. Chapter 6 describes procedures for coding the content and the classroom discourse of the video data by specialists. Lastly, in chapter 7, information is provided regarding the weights and variance estimates used in the data analyses. There are also numerous appendices to this report, including the questionnaires and manuals used for data collection, transcription, and coding. |
7/27/2011 |
| NCEE 20114019 | Baseline Analyses of SIG Applications and SIG-Eligible and SIG-Awarded Schools
The Study of School Turnaround is an examination of the implementation of School Improvement Grants (SIG) authorized under Title I section 1003(g) of the Elementary and Secondary Education Act and supplemented by the American Recovery and Reinvestment Act of 2009. "Baseline Analyses of SIG Applications and SIG-Eligible and SIG-Awarded Schools" uses publicly-available data from State Education Agency (SEA) websites, SEA SIG applications, and the National Center for Education Statistics' Common Core of Data to examine the following: (1) the SIG related policies and practices that states intend to implement, and (2) the characteristics of SIG eligible and SIG awarded schools. This first report provides context on SIG. |
5/9/2011 |
| NCEE 20104003 | Precision Gains from Publically Available School Proficiency Measures Compared to Study-Collected Test Scores in Education Cluster-Randomized Trials
In randomized controlled trials (RCTs) where the outcome is a student-level, study-collected test score, a particularly valuable piece of information is a study-collected baseline score from the same or similar test (a pre-test). Pre-test scores can be used to increase the precision of impact estimates, conduct subgroup analysis, and reduce bias from missing data at follow up. Although administering baseline tests provides analytic benefits, there may be less expensive ways to achieve some of the same benefits, such as using publically available school-level proficiency data. This paper compares the precision gains from adjusting impact estimates for student-level pre-test scores (which can be costly to collect) with the gains associated with using publically available school-level proficiency data (available at low cost), using data from five large-scale RCTs conducted for the Institute of Education Sciences. The study finds that, on average, adjusting for school-level proficiency does not increase statistical precision as well as student-level baseline test scores. Across the cases we examined, the number of schools included in studies would have to nearly double in order to compensate for the loss in precision of using school-level proficiency data instead of student-level baseline test data. |
10/26/2010 |
| NCSER 20103006 | Statistical Power Analysis in Education Research
This paper provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research. For multilevel evaluation studies in the field of education, it is important to account for the impact of clustering on the standard errors of estimates of treatment effects. Using ideas from survey research, the paper explains how sample design induces random variation in the quantities observed in a randomized experiment, and how this random variation relates to statistical power. The manner in which statistical power depends upon the values of intraclass correlations, sample sizes at the various levels, the standardized average treatment effect (effect size), the multiple correlation between covariates and the outcome at different levels, and the heterogeneity of treatment effects across sampling units is illustrated. Both hierarchical and randomized block designs are considered. The paper demonstrates that statistical power in complex designs involving clustered sampling can be computed simply from standard power tables using the idea of operational effect sizes: effect sizes multiplied by a design effect that depends on features of the complex experimental design. These concepts are applied to provide methods for computing power for each of the research designs most frequently used in education research. |
4/27/2010 |
Page 2
of 3