Project Activities
Two domain-specific assessments will be developed for community college courses in economics, biology, and/or physics. The assessments will be developed using evidence-centered design through the online system Principled Assessment Designs for Inquiry (PADI). Panels of experts in each content area along with cognitive scientists and assessment experts will identify the big ideas and other forms of knowledge for each domain. Assessment experts will refine these ideas into assessment patterns and then operationalize them into observable performances. In conjunction with community college instructors, they will develop assessment scenarios and scoring rubrics. The assessment scenarios will be revised after pilot testing using student think-alouds and a small-scale field test. A four-part validation of the assessments will include: (1) an alignment study with other existing assessments (College BASE subject subtests and the Major Field Tests in the subjects), (2) a cognitive analysis to determine if the new assessments (along with the BASE and Major Field Tests) elicit evidence of students' use of the expected knowledge, (3) an instructional sensitivity analysis to determine if students who have greater exposure to the content do better on the assessments (and on the BASE and Major Field Tests), and (4) a correlation between results on the new assessments and student performance as measured by the BASE, Major Field Tests, GPA, and placement tests.
Structured Abstract
Setting
The development of the assessments and the alignment study will take place at SRI International using panels of experts and SRI personnel. The other three validation studies of the assessments will occur at a community college in northern California and will include students taking courses in two of three domains (economics, biology, and/or physics).
Sample
The alignment validation study will use experts from the community college and four-year universities located in northern California. The other validation studies will include student volunteers in the domains at the community college. Forty students (twenty volunteers from each domain) will take part in the cognitive analysis. For the instructional sensitivity study, 300 volunteers (150 in each domain) will take part with half being in the early state of their study and half near program completion. The correlation study will use the data collected in the instructional sensitivity study.
Intervention
The intervention being developed is two domain specific assessments for students in their first two years of college that require them to apply basic content knowledge in a field in complex expert ways.
Research design and methods
Development of the assessments will be based on evidence-centered design through using the online system Principled Assessment Designs for Inquiry (PADI). Content experts in each domain, cognitive scientists, assessment experts, and college instructors will identify the most important knowledge and skills from each domain and develop instruments to assess students for them. Four types of validation studies will be done: (1) an alignment study, (2) a cognitive analysis, (3) an instructional sensitivity analysis, and (4) a correlational study.
Control condition
The validation of the assessments will compare the new assessments to other existing assessments (College BASE subject subtests and the Major Field Tests in the subjects) for the alignment study and will compare the results for students finishing up their program with students just beginning their program in the same domain for the instructional sensitivity analysis.
Key measures
Key measures in the development of the assessments include identification of the key ideas and other important knowledge for each domain. In the validation studies, these big ideas and other knowledge will be looked for: in other assessments and by other panels of domain experts (alignment), in student evidence for their use (cognitive), as gained by students from instruction (instruction), and through a correlation of student results on these assessments to other student outcomes (correlational).
Data analytic strategy
Each validation study will use its own analysis strategy. The alignment study will use a two-way analysis of variance model with raters (as a random effect) crossed with source of assessment (new assessment, BASE, or Major Field Test) as a fixed effect. The cognitive analysis will use a two-level hierarchical model (items nested within students). The instructional analysis will use a two-way analysis of variance model with group membership and test type as fixed factors. The correlation analysis will correlate student sub-scores on the assessments with other student outcomes.
People and institutions involved
IES program contact(s)
Products and publications
Products: The expected products from this project include two fully developed and validated assessments in two of the three domains (economics, biology, and/or physics) for students in their first two years of community college.
Supplemental information
Co-Principal Investigator: Geneva Haertel
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
