Co-Principal Investigators: Wind, Stefanie; Man, Kaiwen; Uttal, David; Wai, Jonathan

Purpose:  The goal of this project is to conduct validity research on a new measure intended for students in grades 2–8, the Cognitive Abilities Test (CogAT): Spatial Assessment, which was developed in collaboration with Riverside Insights and the digital design company Digital Artefacts. This assessment measures students' spatial reasoning skills and is used to inform instructional planning and evaluation. Spatial reasoning is critical to student achievement, particularly in early math skills. At later grades, it is a gateway skill to persistence in engineering, chemistry, geoscience, and other STEM fields. Despite this important role, the field still lacks well-validated measures of spatial reasoning suitable for grades 2–8.

Project Activities: In Year 1, the research team will conduct a large-scale validity study to develop norms and technical guides. In Year 2, the research team will carry out cognitive lab and eye-tracking studies to gather information about inter-individual differences in solution strategies. In Year 3, the research team will conduct a quasi-experimental test/re-test study to examine test reliability and sensitivity to interventions that occur between test and retest. Throughout the project, the research team will also collect data to examine convergent and predictive validity. In Year 4, the research team will analyze their data and disseminate project findings.

Products: The products include the CogAT Spatial Test; a technical manual and norms guide; teacher resources for interpreting and using test scores; training webinars for teachers and administrators; findings from a cost analysis; and presentations, reports, and publications for a variety of education stakeholders.

Structured Abstract

Setting: Schools across the nation will participate in large-scale validity data collection. Intensive data collection will occur in Illinois, Alabama, and Arkansas within urban, rural, and suburban districts.

Sample: The sample includes approximately 2300 students in grades 2–8 (typically ages 7–14). In addition, a sample of 30 teachers—varying in gender, race, ethnicity, area of certification, and years of teaching experience—will be recruited from participating schools and organizations to participate in qualitative research on the instructional needs of students with high/low spatial test scores.

Intervention/Factors/Assessment: The CogAT Spatial Test uses two visualization item formats and two rotation formats to represent the domain of spatial reasoning. Within each facet, one traditional and one interactive item format has been developed. Scoring processing and scaling of the test is an ongoing research focus. Proposed uses of CogAT Spatial include (1) identifying students with exceptional spatial strengths who may excel at specific STEM curricula; (2) identifying students in need of remediation of spatial skills that may impede academic success; and (3) evaluating interventions designed to increase spatial reasoning.

Research Design and Methods: In Year 1, the research team will conduct a large-scale validity study to develop norms and technical guides. The goal of this study is to gather the validity evidence that will support intended uses, including predicting math and science outcomes; reliability for measuring the effect of interventions; and fairness for groups that vary in gender, race, ethnicity, socioeconomic status, and English proficiency. In Year 2, the research team will conduct a cognitive lab study where they will measure students' cognitive processes used while solving the spatial test items using think aloud procedures. They will also conduct an eye-tracking study to identify strategies used to solve items. In Year 3, the research team will conduct a quasi-experimental test/re-test study. Students will take the CogAT Spatial assessment, participate in either spatially intensive programs (intervention condition) or other enrichment experiences (control condition), and then take the assessment again. Test re-test reliability as well as intervention sensitivity will be examined. Throughout the project, the research team will also assess predictive validity by collecting data at a summer STEM program to explore the relationship between performance on the assessment and students' STEM interest. In addition, the research team will interview teachers about the characteristics about students with high and low spatial skills as a measure of convergent validity.

Control Condition: Due to the nature of this work, most of the studies proposed do not have control conditions. The quasi-experimental study uses a control condition where students participate in enrichment experiences that are not focused on spatial skills, such as creative writing.

Key Measures: The research team will collect students' test scores, grades, and ratings in math and science; the CogAT Form 8; the Purdue Test of Spatial Visualization; the Santa Barbara Sense of Direction Scale; the Video Game Experience Survey; think aloud responses; and eye-tracking metrics.

Data Analytic Strategy: The research team will use the data to address a variety of validity research goals, each with their own analytic methods, including multi-group confirmatory factor analysis, correlational analysis, IRT methods of detecting DIF, structural equation modeling to explore predictive invariance, and qualitative coding to understand students' strategies to solving the items.

Cost Analysis: Through the validity study, the research team will develop an accurate cost estimate through analysis of the resources (including technology, training, and other ingredients) required to effectively implement the test and use scores effectively to improve instruction.