Project Activities
The study will use a quasi-experimental design to evaluate the effects of AIW. The project team will analyze achievement data spanning school year (SY) 2005–06 to SY 2016–17 for all schools that have implemented AIW. In addition, the team will analyze student and teacher outcomes for matched samples of AIW and non-AIW schools and teachers. Researchers will conduct three sets of analyses that will be used to investigate the intervention effect on: (a) academic achievement, (b) student engagement and authentic work performance, and (c) teacher outcomes.
Structured Abstract
Setting
Participating schools are located across the State of Iowa.
Sample
In the population achievement component of the study, study participants will include Iowa students across approximately 52 high schools that have adopted the intervention. In the matched samples component of the study, study participants will include 350 teachers and their 4,200 9th- and 10th-grade students across 50 high schools.
Authentic Intellectual Work (AIW) is a state initiative intended to provide students with opportunities to engage in the rigorous manipulation of knowledge through discussion, thinking, and writing. This intervention is designed to influence student outcomes by changing teaching. In the key component of the intervention, teachers collectively score their own assignments, student work, and videotaped lessons in scoring meetings (4 to 6 hours per month) using a set rubric with support from state and local coaches. AIW began in SY 2007-08 and adoption has been gradual. Currently, 101 schools have adopted this intervention (including 52 high schools) and 5 of these schools are no longer participating. In SY 2013-14, 30 new schools are expected to adopt this intervention. There are 26 lead coaches and 40 local coaches.
Initial research
The study will use a quasi-experimental design with two components—a population achievement component involving analysis of achievement data spanning SY 2005–06 to SY 2016–17 for the population of all schools that have implemented AIW and a matched samples component involving the analysis of student and teacher outcomes for matched samples of AIW and non-AIW schools and teachers.
For the population achievement component, researchers will examine the intervention effects on student achievement in mathematics, English, science and social studies using data from the Iowa Department of Education (IDE) Statewide Longitudinal Data System from 2005–06 to 2016–17 for the populations of Iowa students and high schools which have adopted the intervention with varying implementation. Researchers will calculate intervention timing and dosage variables and utilize an extended time series of pre-intervention outcome measures in a difference-in-differences design.
For the matched samples component study, intervention schools will be matched with comparison schools on school-level variables, including aggregated teacher-level and student-level covariates (e.g., average teacher experience, average achievement scores including accountability measures, school size, percentage receiving free lunch, measures of school racial composition). Within each intervention school, researchers will select a random sample of seven teachers and for each teacher, researchers will select a focal class. Teacher-level matching will be conducted within pairs of matched schools using class characteristics (i.e., schooling level, course description, course level, available credit, and sequence) and teacher characteristics (e.g., gender and years teaching experience). Within the focal classes of sampled teachers, researchers will approach all students to participate in the study. Potential differential student participation in the study across conditions will be addressed by inverse probability of treatment weighting which accounts for observed differences in student achievement (e.g. achievement scores, free lunch status). Researchers will assess the effect of intervention after exposures of one and two years by comparing the outcomes of 9th and 10th graders across conditions.
Control condition
Schools in the comparison condition have not yet adopted the AIW intervention.
Key measures
Student and teacher participation in the intervention will be extracted from the IDE Statewide Longitudinal Data System. Student engagement will be measured with surveys and authentic work performance will be measured using scored samples of student work. Student achievement will be assessed using the Iowa Tests of Basic Skills and the Iowa Tests of Educational Development. Teacher instructional practice will be assessed using a rubric for coding teacher assignments and scored samples of student work and using close-ended web-administered questionnaires.
Data analytic strategy
Researchers will employ two models for estimating the impact of AIW on student achievement scores: a value-added model that analyzes the effect of AIW on academic achievement within intervention schools, and a difference-in-differences model comparing the achievement of students from intervention and comparison schools. Researchers will examine the intervention effect on student engagement and authentic work performance using a three-level model (students nested within teachers nested within schools). In addition, the research team will examine the intervention effect on teachers using a three-level model (repeated assessments nested within teachers nested within schools).
People and institutions involved
IES program contact(s)
Project contributors
Partner institutions
Iowa Department of Education
Products and publications
Products: The products of this project will be evidence of the efficacy of the AIW initiative in Iowa for high school students. Peer reviewed publications will also be produced.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
