Description:

Co-Principal Investigator: Deanna Iceman Sands, University of Colorado, Denver

Purpose: There is a substantial and growing body of evidence that using assessment results formatively to shape learning activities can have a significant impact on student learning. However, there are no instruments available to measure formative assessment practices in the classroom. In this project, the researchers will investigate formative assessment practices and accompanying instructional practices that occur in middle-level science classrooms. Specifically, this project will develop and carry out the technical evaluation of instruments designed to measure the quality of formative assessment practices as they are used in the classroom.

Project Activities: This project seeks to develop and test reliable, valid, and practical measures of formative assessment practices in middle school science classrooms. The research is guided by three measurement questions: (a) What critical attributes define the formative assessment practices of very good teachers; (b) What measures can provide valid and reliable information about their formative assessment practices; and (c) Which of these measures are more cost effective and user friendly for different users? These measurement questions will be approached through an assessment and development framework that will be carried out in three phases and involve 132 teachers and their students across diverse (race, ethnicity, language, and socio-economic status) school districts in an urban metropolitan area of Colorado. In Phase I, researchers will define the construct (formative assessment practices) based on theory and empirical research and identify the personal attributes of teachers required to implement formative assessment practices. In Phase II, researchers will collect information to refine the construct and develop and pilot benchmark and surrogate instruments. In Phase III, information will be collected that allows for the psychometric evaluation of the benchmark and surrogate instruments developed in Phase II. The design incorporates qualitative and quantitative methods, involves four iterations to improve the methods of data collection and instruments developed, and collects diverse information to evaluate reliability and validity of the instruments using different sources of evidence.

Products: Researchers will produce scholarly reports of the findings. In addition, the researchers will develop and validate various instruments, including benchmark measures, surrogate measures, technical reports, and a project manual.

Structured Abstract

Setting: Participating students are located in Colorado in urban school districts characterized by a wide range of student diversity including ethnicity (24–73 percent non-white), socio-economic status (17–80 percent free/reduced price lunch), and linguistic background (8–65 percent).

Sample: Study participants include 132 middle-school science teachers over the course of the four-year project, with an estimated 30 students per classroom.

Intervention: The proposed project will develop and validate the following instruments: (1) benchmark measures of formative assessment practices, and the resources and social processes that support those practices, which include in-depth analyses of videos, in-depth direct classroom observations, and in-depth interviews; (2) surrogate measures, which are more cost-, time-, and effort-efficient than the gold standard benchmark measures and include a formative assessment classroom observation protocol by a third-party, formative assessment self-report teacher log protocol, and student and teacher self-report questionnaires; (3) technical reports on both the benchmark and surrogate measures; and (4) a manual detailing how to use the surrogate measures, the resources needed to implement each instrument, training required, and number of observations needed to obtain dependable measures.

Research Design and Methods: The research design incorporates qualitative and quantitative methods, involves four iterations to improve the methods of data collection and instruments developed, and collects diverse information to evaluate reliability and validity of the data using different sources of evidence. Assessment instruments will be developed across 3 phases. In Phase I, researchers will define the construct, formative assessment practices, based on theory and empirical research and identify the personal attributes of teachers required to implement formative assessment practices. In Phase II, researchers will collect information (through diverse sources such as direct observations, interviews of teachers and students, videotapes) to refine, using an iterative design process, the construct and develop and pilot benchmark and surrogate instruments (i.e., the classroom observation protocols and teacher logs). In Phase III, researchers will conduct two iterations, each with 50 teachers and their students, to evaluate the psychometric properties of the benchmark and surrogate instruments developed in Phase II. A random sample of 10 classrooms will also be videotaped and tapes will be coded by 2 independent raters and used as one piece of evidence to evaluate validity of the surrogates. In addition, there will be teacher and student self-report questionnaires and measures of student learning and motivation.

Control Condition: Due to the nature of the research design, there is no control condition.

Key Measures: Students' learning outcomes will be assessed using the Colorado Student Assessment Program and district benchmark assessments. Student motivation (learning goal orientation) will be assessed using researcher-developed questionnaires. Data sources include videotapes of classroom teaching; classroom observations; videotaped teacher interviews to assess teacher beliefs and conduct cognitive task analysis; video-stimulated recall where teachers view and answer questions about four selected scenes from the videotapes of instructional episodes; videotaped student interviews of six randomly selected students per classroom; and classroom artifacts.

Data Analytic Strategy: Surrogate instruments will be evaluated based on the following dimensions: (1) directedness; (2) extent of judgment; (3) sensitivity; (4) source of information; (5) practices tapped. To identify which attributes of the practices result in students being more engaged in the learning activities, and which one seems to have a higher impact on students' learning, the researchers will code videotapes using four criteria: importance, frequency, difficulty, and variability. Researchers will conduct cognitive analysis of teacher and student interviews about their conception or responses to the questions/items of formative assessment practice measures. In Phase II and III, researchers will calibrate the item response theory models to examine how well the theoretically defined construct is consistent with the empirical data; conduct item analysis to investigate whether items are consistent with the instrument as a whole; conduct differential item functioning study to show whether particular items may function differently for examinees, of the same level of the construct, from different subgroups; and conduct higher order confirmatory factor analysis. In Phase III, researchers will provide convergent evidence using generalizability studies to evaluate the exchangeability of video coding, the observation protocol, and teacher logs; and examine the relationship between the criterion variables (learning and achievement outcome and student motivation) and the scores of formative assessment practice measures using a 3-level hierarchical linear model (student, teacher, school). Variables generated from different measures of formative assessment practices will be introduced to a multi-level model as independent variables.

Products and Publications

Book chapter

Ruiz-Primo, M.A., Solano-Flores, G., and Li, M (2014). Formative Assessment as a Process of Interaction Through Language. In C. Wyatt-Smith, V. Klenowski and P. Colbert (Eds.), Designing Assessment for Quality Learning (pp. 265–282). Springer Netherlands.

Journal article, monograph, or newsletter

Kroog, H. and Ruiz-Primo, M.A. (2017). Are Clouds a Solid, Liquid, or Gas?. Science and Children, 54(8): 41–44.