Previous Award Number: R305A200199
Previous Awardee: State University of New York (SUNY), Buffalo

Co-Principal Investigators: Dumas, Denis; Organisciak, Peter

Purpose: This research project aims to develop a new instrument and a computer application that measures original thinking among late-elementary students (Grades 3–5) through automated scoring. The Measurement of Original Thinking in Elementary Students (MOTES) will adopt text-mining models that have been tested only with adults and test their use with elementary students. Measuring original thinking in children has a long history. However, manual scoring of existing original thinking tests limits their accessibility—because many schools may not afford the costs and logistics of such tests—and introduces measurement error into the scores. Therefore, fewer children are measured for original thinking, leading directly to the under-representation of minority students and low-SES learners in gifted and talented programs. There is a need for screening tools that allow measuring original thinking in a large number of students quickly and at lower cost.

Project Activities: Researchers will develop a massive corpus of text (target: 1 million unique words) that is built to approximate the semantic language-space of a child within the range of grades 3 to 5, which is the range when creativity assessment in schools is most common. This corpus will be used to score the four different divergent thinking (DT) tasks. After administering these tasks to a large and diverse sample of elementary school students, the tasks will be scored using text-mining methods and examined in terms of fairness of the individual items and reliability and validity of the scales. Researchers will also complete a cost analysis.

Products: The MOTES will be available to education practitioners and researchers who can obtain instant scores, for free or for very low cost. The team will also present findings from the research in conference presentations and in peer-reviewed publications.

Structured Abstract

Setting: The MOTES will be developed in collaboration with participating urban and suburban schools across the State of New York. Most participating schools are from the western and central New York area.

Sample: Researchers will collect data from a total of 400 students grades 3 through 5. The sample will include comparable (~200) number of students with low versus middle or high socio-economic status (SES).

Assessment: Researchers will develop and validate the MOTES, which consists of four different types of divergent thinking tasks (i.e., Uses, Instances, Consequences, and Scenarios) with 5 tasks in each (a total of 20 tasks). The responses will be scored based on the text-mining models using the new corpus that will be developed as part of this project.

Research Design and Methods: In order to investigate the reliability and validity of MOTES scores, researchers will use a full course of validation methods, including content validity questionnaires for practicing elementary school teachers, targeted cognitive interviews with elementary students, multi-dimensional latent measurement models, modern reliability coefficients (e.g., H and Omega), multi-group measurement models for testing differential item functioning (DIF) in the model parameters, as well as criterion validity correlations to a range of external criteria including academic achievement measures, teacher report of original thinking, and existing standardized creative thinking assessments. At every phase of this validation methodology, the MOTES items and text-mining corpus will be revised to produce maximally valid and reliable scores for use in applied research and educational practice.

Key Measures: To validate the MOTES, which is the new battery of divergent thinking tasks described above, three major types of data will be used: (a) students' academic achievement data including grade point average (GPA), standardized achievement tests, and subject-specific grades; and (b) Torrance Tests of Creative Thinking – Figural (TTCT-Figural-Form A) and (c) teacher evaluation of students' originality. Academic achievement data will be collected from the school directly and matched with data to be collected from students and teachers. TTCT-Figural Form A consists of three activities: Picture Construction, Picture Completion, and Lines. Students are given 10 minutes for each activity.

Data Analytic Strategy: Researchers will fit a number of differently specified multi-dimensional measurement models (i.e., confirmatory factor models) in order to determine the validity of the instrument, with the best-fitting dimensional structure being used to examine DIF across groups of high- and low-SES students.

Cost Analysis: A cost analysis will be conducted to determine the costs associated with implementing the MOTES.The research team predicts cost savings to schools and districts when it is used in universal screening efforts for gifted identification due to automatized scoring.