Skip Navigation
Funding Opportunities | Search Funded Research Grants and Contracts

IES Grant

Title: Domain-Specific Assessment: Bringing the Classroom into Community College Accountability
Center: NCER Year: 2008
Principal Investigator: Yarnall, Louise Awardee: SRI International
Program: Postsecondary and Adult Education      [Program Details]
Award Period: 4 years Award Amount: $1,568,413
Type: Measurement Award Number: R305A080109

Co-Principal Investigator: Geneva Haertel

Purpose: The project will develop and validate domain-specific assessments for students in their first two years of college that require them to apply basic content knowledge in a field in complex expert ways. The goal is to measure how well students can reason using content knowledge versus learning the narrow procedural/technical skills often measured by current assessments. By determining whether students understand why and when to apply knowledge, procedures, and strategies unique to a domain, these assessments can help determine how prepared students are to apply high-level skills in more advanced courses and the workplace.

Project Activities: Two domain-specific assessments will be developed for community college courses in economics, biology, and/or physics. The assessments will be developed using evidence-centered design through the online system Principled Assessment Designs for Inquiry (PADI). Panels of experts in each content area along with cognitive scientists and assessment experts will identify the big ideas and other forms of knowledge for each domain. Assessment experts will refine these ideas into assessment patterns and then operationalize them into observable performances. In conjunction with community college instructors, they will develop assessment scenarios and scoring rubrics. The assessment scenarios will be revised after pilot testing using student think-alouds and a small-scale field test. A four-part validation of the assessments will include: (1) an alignment study with other existing assessments (College BASE subject subtests and the Major Field Tests in the subjects), (2) a cognitive analysis to determine if the new assessments (along with the BASE and Major Field Tests) elicit evidence of students' use of the expected knowledge, (3) an instructional sensitivity analysis to determine if students who have greater exposure to the content do better on the assessments (and on the BASE and Major Field Tests), and (4) a correlation between results on the new assessments and student performance as measured by the BASE, Major Field Tests, GPA, and placement tests.

Products: The expected products from this project include two fully developed and validated assessments in two of the three domains (economics, biology, and/or physics) for students in their first two years of community college.

Structured Abstract

Setting: The development of the assessments and the alignment study will take place at SRI International using panels of experts and SRI personnel. The other three validation studies of the assessments will occur at a community college in northern California and will include students taking courses in two of three domains (economics, biology, and/or physics).

Population: The alignment validation study will use experts from the community college and four-year universities located in northern California. The other validation studies will include student volunteers in the domains at the community college. Forty students (twenty volunteers from each domain) will take part in the cognitive analysis. For the instructional sensitivity study, 300 volunteers (150 in each domain) will take part with half being in the early state of their study and half near program completion. The correlation study will use the data collected in the instructional sensitivity study.

Intervention: The intervention being developed is two domain specific assessments for students in their first two years of college that require them to apply basic content knowledge in a field in complex expert ways.

Research Design and Methods: Development of the assessments will be based on evidence-centered design through using the online system Principled Assessment Designs for Inquiry (PADI). Content experts in each domain, cognitive scientists, assessment experts, and college instructors will identify the most important knowledge and skills from each domain and develop instruments to assess students for them. Four types of validation studies will be done: (1) an alignment study, (2) a cognitive analysis, (3) an instructional sensitivity analysis, and (4) a correlational study.

Control Condition: The validation of the assessments will compare the new assessments to other existing assessments (College BASE subject subtests and the Major Field Tests in the subjects) for the alignment study and will compare the results for students finishing up their program with students just beginning their program in the same domain for the instructional sensitivity analysis.

Key Measures: Key measures in the development of the assessments include identification of the key ideas and other important knowledge for each domain. In the validation studies, these big ideas and other knowledge will be looked for: in other assessments and by other panels of domain experts (alignment), in student evidence for their use (cognitive), as gained by students from instruction (instruction), and through a correlation of student results on these assessments to other student outcomes (correlational).

Data Analysis Strategy: Each validation study will use its own analysis strategy. The alignment study will use a two-way analysis of variance model with raters (as a random effect) crossed with source of assessment (new assessment, BASE, or Major Field Test) as a fixed effect. The cognitive analysis will use a two-level hierarchical model (items nested within students). The instructional analysis will use a two-way analysis of variance model with group membership and test type as fixed factors. The correlation analysis will correlate student sub-scores on the assessments with other student outcomes.