|Title:||Exploring Writing Achievement and Its Role in Success at 4-Year Postsecondary Institutions|
|Principal Investigator:||Burstein, Jill||Awardee:||Educational Testing Service (ETS)|
|Program:||Postsecondary and Adult Education [Program Details]|
|Award Period:||4 years (7/1/2016-6/30/2020)||Award Amount:||$1,387,363|
Co-Principal Investigator: Daniel McCaffrey
Purpose: Postsecondary-level writing challenges students and the instructors who teach it. In this project, the researchers will address this challenge by exploring writing skills and achievement, the malleable factors that predict writing quality, and their links to postsecondary success.
Project Activities: Using both secondary and primary data, the researchers will examine interactions among factors of writing achievement and explore how these factors relate to college retention and completion. They use a theoretical model of writing that assumes that multiple skills contribute to writing achievement including writing-domain knowledge (e.g., grammar, organization), domain-general knowledge (e.g., critical thinking), and intrapersonal factors (e.g., motivation).
Products: The researchers will produce preliminary evidence of potential malleable factors that affect writing achievement to help inform the development of interventions. They will also produce peer-reviewed publications.
Setting: Study 1 will use secondary data collected from 22 diverse 4-year postsecondary institutions from across the U.S. Study 2 will use data from students attending one of eight historically black colleges or universities in six different states and a large 4-year university.
Sample: The data for Study 1 were collected from 1,791 students who completed both the ETS HEIghten Critical Thinking and Written Communications Assessments from a previous pilot study of the assessment. The data for Study 2 sample will come from 2,000 students enrolled in developmental and standard writing classes or disciplinary classes (e.g., humanities or science classes) and who represent a diverse group in terms of gender, ethnicity, first-generation college student status, and socioeconomic status.
Intervention: Underlying this research is the assumption that writing skills rely on writing-domain knowledge, general-domain knowledge, and intrapersonal factors. Combined, these three components may predict a student's writing level and, hence, his or her ability to complete writing courses and persist in postsecondary classes. In terms of writing-domain knowledge, the researchers are focusing on five main constructs:
Students' skill in these constructs may vary depending on student background characteristics. For example, some students may have deficits in some, but not all, constructs or be weak or strong across the board. In terms of domain-general knowledge, the researchers are exploring whether general critical thinking skills correlate with writing ability. In terms of intrapersonal factors, they are exploring whether a student's level of engagement, goal-setting, interest, motivation, and self-efficacy in writing will correlate with his or her writing ability.
Research Design and Methods: The researchers will use both existing secondary data sets (Study 1) and primary data (Study 2) for two separate sub-studies. They will examine samples of students' writing using ETS's existing automated writing evaluation (AWE) capabilities and student writing samples from writing prompts used in HEIghten Written Communication Assessment (Studies 1 & 2) and disciplinary course assignments (Study 2). The AWE tool analyzes writing samples for the 5 writing-domain constructs, analyzing 175 features to build profiles for a writer's use of the constructs. These profiles will provide detailed information about writing skills that may be essential for postsecondary success. In Study 2, the researchers will also collect additional measures of student domain-general knowledge and intrapersonal skills and examples of course-based writing. Students in Study 2 will come from multiple disciplines, such as science, humanities, or psychology to help ensure a broad range of genres and writing samples.
Control Condition: Given the exploratory nature of this project, there is no control condition.
Key Measures: The researchers will evaluate writing-specific and domain-general knowledge using subtests of ETS's HEIghten assessment and intrapersonal factors using existing motivation and interest scales. They will also collect administrative data on students' overall GPA and GPA from writing intensive courses, course completion, major course enrollment and completion, and retention in college for five semesters following primary data collection.
Data Analytic Strategy: The researchers will use multiple analytic methods such as factor analysis, regression, and structural equation modeling to study the relationships among the AWE-identified linguistic features and among those features, critical thinking, intrapersonal factors, writing skills, and other postsecondary outcomes.
Burstein, J., Riordan, B., & McCaffrey, D. (2020). Expanding Automated Writing Evaluation. In, Yan, D., Rupp, A. A., & Foltz, P. W. (Eds.). (2020). Handbook of Automated Scoring: Theory into Practice (pp 329–346). CRC Press. 329–346
Hazelton, L., Nastal, J., Elliot, N., Burstein, J. & McCaffrey, D. (in press). Formative automated writing evaluation: A standpoint theory of action. Journal of Response to Writing.
Ling, Guangming, Elliot, N., Burstein, J. C., McCaffrey, D. F., MacArthur, C. A., & Holtzman, S. (2021). Writing motivation: A validation study of self–judgement and performance. Assessing Writing, 48.
Burstein, J., McCaffrey, D., Elliot, N., Beigman Klebanov, B., Molloy, H., Houghton, P. & Mladineo, Z. (2020). Exploring Writing Achievement and Genre in Postsecondary Writing . In Companion Proceedings in the 10th International Conference on Learning Analytics & Knowledge (LAK20), pp 53–55. Full text
Burstein, J., McCaffrey, D., Klebanov, B. B., & Ling, G. (2017). Exploring Relationships Between Writing and Broader Outcomes With Automated Writing Evaluation. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 101–108. Full Text
Burstein, J., McCaffrey, D., Beigman Klebanov, B., Ling, G., & Holtzman, S. (2019). Exploring Writing Analytics and Postsecondary Success Indicators. In Companion Proceedings 9th International Conference on Learning Analytics & Knowledge (LAK19), pp. 213–214. Full Text
McCaffrey, D., Holtzman, S., Burstein, J., and Beigman Klebanov, B. (in press). What can we learn about college retention from student writing? To appear in Companion Proceedings in the 11th International Conference on Learning Analytics & Knowledge (LAK21).