Project Activities
First, researchers will refine and extend an already-developed response-to-text assessment involving two texts, one of which already includes a prototype to automatically assess writing. Next, they will develop an automated assessment for the second text, and evaluate whether it can provide formative feedback to inform quality of student writing and teacher instruction. The research team will then conduct a series of studies to validate the tool by measuring the inter-rater agreement between human and automatically generated assessments of students' analytic writing.
Structured Abstract
Setting
Development and validation of the assessment will occur in two large suburban districts in New York State. The schools serve a socioeconomic and ethnically diverse population, with half of the students coming from minority backgrounds.
Sample
The sample will consist of 82 English Language Arts teachers in Grade 5 and 6 and students in these classrooms. During Year 1, the study will include 40 teachers and their students. During Year 2, the study will include 12 teachers and their students. During Year 3, the sample will include 30 teachers and their students distributed across the 3 grade levels.
Assessment
The assessment will measure students' ability to reason about texts while writing and to use evidence effectively to support claims. The assessment is intended to be administered by teachers in 60 minutes. The teacher reads a pre-determined text for 15 minutes and students follow along with their own copy of the text. Students then have 45 minutes to write their responses on an online form. The tool will automatically assess students' writing by extracting information on the number of pieces of evidence, the concentration of evidence provided, the specificity, and the word count.
Research design and methods
During Year 1, the researchers will develop and test algorithms to compare scores of the computer generated reports and the human generated reports. To expand and replicate the validity investigation, in years 2 and 3 the researchers will collect evidence for whether feedback from the automated essay scoring feature helps students' improve using evidence to support writing and whether the feedback influences the beliefs and practices of teachers.
Control condition
There is no control condition for this project.
Key measures
Measures will target two constructs within analytic text-based writing: students' effective use of evidence and their organization of ideas and evidence in support of a claim. Key student outcome measures include the New York State Common Core tests comprised of multiple-choice items, short constructed responses, items requiring extended writing, and the Smarter Balanced Summative Assessment Tests in literacy.
Data analytic strategy
Researchers will analyze the data using multi-level models, with classrooms as the unit of analysis and achievement measured at the individual student level and nested within teachers. Analyses will control for prior achievement and background characteristics. The focus of the research is on the effect of the assessment on achievement in analytic writing.
People and institutions involved
IES program contact(s)
Products and publications
Products: Researchers will fully develop an online automated machine scoring tool to assess student writing, both in the use of evidence and organization of writing in response to two texts. The tool will automatically generate formative feedback based on students' written responses. The tool is designed for students in grade 5 and 6. The research team will also develop a reporting dashboard to provide formative assessment feedback and reports to teachers to inform instruction.
Conference Proceeding
Zhang, H., Magooda, A., Litman, D., Correnti, R., Wang, E., Matsmura, L.C., ... and Quintana, R. (2019, July). eRevise: Using Natural Language Processing to Provide Formative Feedback on Text Evidence Usage in Student Writing. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 9619-9625).
Preprint
Zhang, H., and Litman, D. (2019). Word Embedding for Response-To-Text Assessment of Evidence. arXiv preprint arXiv:1908.01969.
Zhang, H., and Litman, D. (2019). Co-attention Based Neural Network for Source-Dependent Essay Scoring. arXiv preprint arXiv:1908.01993.
Supplemental information
Co-Principal Investigators: Richard Correnti and Lindsay Clare Matsumara
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
