Project Activities
First, researchers will assess the construct validity of CVAMs by testing their ability to accurately predict teacher effects on short-term outcomes using quasi-experimental teacher switching designs. Second, researchers will investigate the predictive validity of CVAMs relative to existing TVAMs and NVAMs for long-run outcomes. Third, researchers will investigate convergent validity by exploring whether VAMs that incorporate non-test outcomes capture distinct teaching skills by linking them to other measures of teacher practice and student behavior. To conduct this work, the researchers will use data from grades 4 through 10 in all public schools across Massachusetts and North Carolina, linking students to their teachers in core subjects. They will compare predicted student outcomes in grade-school-year cells—where predictions are formed from estimated teacher performance in other years—with actual student outcomes.
Structured Abstract
Setting
Data will come from Massachusetts and North Carolina.
Sample
The sample includes data from all students and their teachers in grades 4 through 10 in the core academic subjects (English language arts, mathematics, science, and social studies) in public schools.
This measure of teacher contribution to student outcomes will incorporate test as well as non-test information into value-added models (VAMs). The VAMs will control for student demographics, classroom composition, classroom and school prior outcomes and demographics, and include teacher fixed effects to isolate only within-teacher variation to estimate the coefficients on the controls and the inclusion of school or academic track effects. Test information will include student scores on statewide standardized test data for math and ELA by grade and year. Non-test information will include (a) student absences, (b) student disciplinary infractions, (c) student grades, and (d) student grade promotion.
Research design and methods
Using administrative data from each state, the researchers will construct value-added measures including teacher TVAMs, teacher NVAMs, and teacher CVAMs and accounting for the sorting of students into tracks in secondary school. They will examine psychometric properties of NVAMs by constructing teacher effects on each proposed non-test measure individually and then estimate the variance in teacher effects using the covariance between estimated teacher effects in consecutive years (test-retest reliability). Using data aggregated to the school-grade-year level, they will examine variation in teacher impacts on each of the proposed non-test inputs induced by teacher staffing changes. They will explore the relation between each of the non-test measures individually and measures of underlying student behaviors and skills and will explore various ways of accounting for classroom or school context (e.g., classroom control variables or school fixed effects). For teacher CVAMs, the researchers will test for teacher effects that are persistent across school years and classrooms. They will examine the extent to teacher CVAMs make out-of-sample predictions of the outcomes of their students and will examine the extent to which teacher CVAMs predict student long-run outcomes such as college enrollment and college quality.
Control condition
Due to the nature of the project, there is no control condition.
Key measures
The researchers will measure teacher professional and instructional practice using classroom observation rubric scores collected by school districts as part of teacher evaluation. They will use state and district administrative records of (1) student absences, (2) student disciplinary infractions, (3) student grades, and (4) student grade promotion as proxies of non-cognitive skills. They will also use student performance on standardized tests, high school grade point average, performance in Advanced Placement courses, performance on the SAT tests, high school graduation, college enrollment, and college quality as indicators of student academic outcomes.
Data analytic strategy
The researchers will use principal components methods to combine non-test outcomes into a single behavioral measure. The CVAM approach will use the estimated relations between short- and long-run outcomes observed in a subset of our data to generate predicted values of student long-run educational attainment outcomes based on their short-run outcomes. These predicted outcomes will thus represent a combination of test and non-test outcomes in the value-added analyses. The researchers will use jackknife (leave-one-out) forecasts of the CVAMs. Utilizing multi-level modeling, they will conduct a teacher-switching quasi-experiment to measure whether aggregate student outcomes in a school are correlated with forecasted teacher performance as teachers enter or exit that school. They will also explore the sensitivity of validity estimates to choices regarding the measurement of non-test outcomes and the inclusion or exclusion of various explanatory variables.
Cost analysis strategy
Researchers will calculate cost of this measure by collecting number of person-hours needed to data management and calculate VAM estimates and hourly personnel cost.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
This project will result in a fully developed and validated measure for teacher contribution to student outcomes for grades 4 through 10. The project will also result in peer-reviewed publications and presentations as well as additional dissemination products that reach education stakeholders such as practitioners and policymakers. The researchers will produce publications about the construct validity of CVAMs, the predictive validity of CVAMs, and the convergent validity of CVAMs. Initial papers will be released as CALDER working papers and later submitted to academic conferences and journals. In addition, the researchers will produce a shorter research brief for policymakers that describes the research findings.
Publications:
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
