Denis Newman, Empirical Education Inc., Co-PI; Richard Sawyer, AED, Co-PI; Jean Scott, SERVE Center, Study Manager
Study Purposes and Questions:
Examines the impact of the AMSTI program for the 4th through 8th grades in 80 Alabama public schools. The primary research questions concerning impact and implementation are: IMPACT
What is the impact of AMSTI on student achievement in math, science, and reading during Year 1 and Year 2 implementation?
What is the impact of AMSTI on teacher instructional practices during Year 1 and Year 2 implementation?
Does two years of AMSTI have a greater impact on student achievement than one year?
Does AMSTI have a lasting effect on student achievement after the intervention has been diminished?
How does the impact of AMSTI vary across characteristics of teachers, students, and the school technology environment?
IMPLEMENTATION
Is the delivery of regional professional training consistent with the state design of the AMSTI program? Are the technology materials that AMSTI uses made available to classrooms on a timely basis?
What is the relationship between the training provided and actual classroom instructional practices in the region?
How does classroom implementation of AMSTI vary with teacher characteristics, student characteristics, and the technology environment of the school?
Intervention:
The Alabama Math, Science and Technology Initiative (AMSTI) was developed by the Alabama State Department of Education (ALSDE) to improve the quality of mathematics and science instruction with provision of comprehensive professional development, in-school coaching and supports, and provision of technology and instructional materials. Teachers and principals are required to attend a two week Summer Institute prior to each of the first two school years during which AMSTI is implemented. The treatment includes: on-site teacher trainings in the Fall of each year; mentoring by "lead" math and science teachers with newer faculty; site visits from math and science specialists from the local university to provide assistance; and teacher team meetings to plan and discuss their use of and experience with AMSTI. AMSTI also provides technology-oriented and hands-on resources and tools that include assessments and various toolkits from the GLOBE software program.
Design and Samples:
This study is a grouped randomized control trial in 5 regions of Alabama. Schools are randomized into 2 groups. One group of 20 schools will receive 2 years of the AMSTI program in 2006-2007 and 2007-2008. A second group of 20 schools will receive the AMSTI program in the second school year 2007-2008 and continue through the third year 2008-2009. A replication study begins in 2007-2008 and is similar in design and size as the original study. Data from the original and replication studies will be included in the final analysis and results.
Outcome Measures:
Student achievement is measured with 4 years of longitudinal data from the Alabama Reading and Mathematics Test (ARMT grades 3 through 8) and the Stanford 10 (grades 4 and 8) on science.
The differences in instructional practices and school context between AMSTI and comparison groups are measured using data from teacher and principal surveys and interviews and classroom observations.
Measures of interactions of impact on student and teacher characteristics data will utilize the above measures as well as demographic data collected from schools. Covariates include pretest scores, student SES and ethnicity, teacher education, amount of inquiry instruction, and teacher professional development.
During the 3 years of each study (original and replication studies), randomly assigned groups allow for 3 different kinds of comparisons: in year 1 the study compares 1 year of treatment with no treatment; in year 2 the study compares 2 years of treatment with 1 year of treatment; in year 3, the study compares full treatment plus a year of lasting effects with 2 years of treatment.
Schools assigned to the control group remain in that condition for only 1 year and then join the treatment condition. However, full treatment is a 2 year process. To infer what the treatment effect is after 2 years we rely on a quasi-experimental approach whereby we obtain a counterfactual value by assuming that the impact of the first year of AMSTI applies to the treatment group as well as the controls once they enter treatment a year later. The resulting estimate does not become biased with changes in secular trends and is accurate so long at the treatment effect is constant.
The Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI)