Co-Principal Investigators: Rod Roscoe, James Blasingame

Purpose: The Writing Pal (W-Pal) provides writing strategy instruction, game-based writing practice, essay writing practice, and formative feedback to adolescent writers. The purpose of this project is to explore how components of the fully developed W-Pal are related to student writing strategy acquisition and writing proficiency. Researchers will examine the extent to which students' strategy acquisition and interactions with the W-Pal are influenced by individual differences such as prior writing skill and strategy use, writing self-efficacy, and attitudes about writing. Additionally, researchers will explore the effects of different formats for strategy instruction, different modes and formats of practice, and different essay feedback formats on students' writing strategy acquisition. Information gained from this study will facilitate later efficacy studies of the W-Pal.

Project Activities: Seven studies will be carried out in this project to ascertain which components of the intervention are associated with positive outcomes. Two correlational studies will examine the association of students' individual differences with writing strategy acquisition. Five experimental studies will explore of the effects of various formats and modes of instruction, practice, and feedback. During the first correlational study, researchers will implement W-Pal in authentic classrooms with approximately 1,000 9th- to 12th- grade students. The team will collect data for several months as students interact with the W-Pal. The second correlational study will be a replication of the first, conducted after modifications have been made to the W-Pal following the first correlational study and the experiments. All five of the experiments will include between 96 and 184 high school students, and each will have multiple conditions in order to test the effects of various formats or features of the W-Pal, including whether animated video lessons are more effective than illustration instructional texts, and whether having control or no control over the amount of feedback received effects students' strategy acquisition.

Products: The final product of this study will be additional information regarding the format and modes of the fully developed W-Pal. Peer-reviewed publications will also be produced.

Structured Abstract

Setting: The correlational studies will be conducted in urban and suburban school districts in Arizona. The experimental studies will be conducted in a laboratory at Arizona State University.

Sample: Participants will include approximately 2,700 high school students.

Intervention: The W-Pal is a fully developed educational technology tool aimed at improving adolescents' writing strategy acquisition and students' writing quality. The technology incorporates direct instruction of strategies, opportunities for extended practice over time, and formative feedback. The W-Pal incorporates strategy lessons including step-by-step guides, examples, and mnemonic devices, practice games, and an essay writing tool that provides automated feedback. Lessons covered by the W-Pal include: freewriting; planning; introduction building; body building; conclusion building; paraphrasing; cohesion building; and revising. Student responses to the W-Pal have been overwhelmingly positive, and past research has shown that W-Pal writing scores increase from pre-test to post-test. Part of the current project's purpose is to explore those aspects of the W-Pal which received less-positive feedback from students.

Research Design and Methods: The two correlational studies will include several months of the W-Pal implemented in authentic classrooms. Students will complete pre-test and post-test measures of writing achievement and strategy use. Additionally, researchers will collect system use data to examine the associations between system use and writing quality. The first experimental study will compare the differential effects of animated video lessons versus illustrated instructional texts on student writing quality. The second experimental study will compare the effects of experiencing lessons and practice games in either a fixed, linear sequence or an open-choice, non-linear sequence. The third experimental study will compare the impacts of three practice formats: non-game practice; game-based practice; and essay-based practice. The fourth experimental study will explore the effects of receiving more versus less feedback from the W-Pal, and of students having more control or no control over the amount of feedback received. The fifth and final experimental study will compare strategy-focused essay feedback to trait-based essay feedback (i.e. spelling, grammar, mechanics).

Control Condition: Due to the design of the two correlational studies, there will be no control conditions. While the experimental studies are designed to have multiple treatment groups for the purpose of comparison, none of them have control conditions.

Key Measures: Students will write test essays in a number of the studies, and test essay topics will be similar to SAT and ACT test prompts. Each essay will be scored based on the SAT essay scoring rubric. Lesson-specific strategy knowledge tests and strategy writing tasks will be used to assess students' strategy knowledge and use. The Gates-MacGinitie Reading Test will be used to assess reading skill and vocabulary knowledge, and the Writing Attitudes and Strategies Self-Report Inventory (WASSI) will be used to measure students' writing self-efficacy. Students' comfort with writing will be assessed with the Daly-Miller Writing Apprehension Test. Finally, log file data from the W-Pal system will provide measures of system use, and student surveys will provide measures of student perceptions of the W-Pal.

Data Analytic Strategy: The two correlational studies are aimed at examining the associations between individual characteristics, system use, and writing quality and data will be analyzed using hierarchical linear modeling. Any mediational analyses will be conducted with structural equation modeling. The five experiments all employ between-groups multiple analyses of variances and repeated-measures analyses of variance as appropriate. In some of the studies, analyses of covariance will be used when covariates or additional independent variables are included to account for the effects of individual differences.

Related IES Projects: The Writing Pal: An Intelligent Tutoring System that Provides Interactive Writing Strategy Training (R305A080589)

Products and Publications

Book chapter

Johnson, A.M., Jacovina, M.E., Russell, D.G., and Soto, C.M. (2017). Challenges and Solutions When Using Technologies in the Classroom. Adaptive Educational Technologies for Literacy Instruction. Taylor and Francis eBooks.

Allen, L. K., Jacovina, M. E., and McNamara, D. S. (2016). Computer-Based Writing Instruction. Handbook of Writing Research (pp. 316–329). The Guilford Press.

Jacovina, M. E., and McNamara, D. S. (in press). Intelligent Tutoring Systems for Literacy: Existing Technologies and Continuing Challenges. Intelligent Tutoring+Systems: Structure, Applications and Challenges.

McNamara, D. S., Jacovina, M. E., and Allen, L. K. (2015). Higher Order Thinking. Handbook of Individual Differences in Reading: Reader, Text, and Context (pp. 164–). Routledge, Taylor and Francis, Ltd.

Stone, M., Kent, K., Roscoe, R., Corley, K.M., Allen, L.K., and McNamara, D.S. (2017). The Design Implementation Framework: Iterative Design from the Lab to the Classroom. End-user considerations in educational technology design.

Journal article, monograph, or newsletter

Allen, L.K., Snow, E.L., and McNamara, D.S. (2016). The Narrative Waltz: The Role of Flexibility in Writing Proficiency. Journal of Educational Psychology.

Crossley, S.A., and McNamara, D.S. (2016). Say More and be More Coherent: How Text Elaboration and Cohesion can Increase Writing Quality. Journal of Writing Research, 7: 351–370.

Crossley, S.A., Kyle, K., and McNamara, D.S. (2015). To Aggregate or Not? Linguistic Features in Automatic Essay Scoring and Feedback Systems. The Journal of Writing Assessment, 8(1).

Crossley, S.A., Kyle, K., and McNamara, D.S. (2016). The Development and use of Cohesive Devices in L2 Writing and Their Relations to Judgments of Essay Quality. The Journal of Second Language Writing, 32: 1–16.

McNamara, D. S. (2013). The Epistemic Stance between the Author and Reader: A Driving Force in the Cohesion of Text and Writing. Discourse Studies, 15(5): 579–595.

McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., and Dai, J. (2015). A Hierarchical Classification Approach to Automated Essay Scoring. Assessing Writing, 23: 35–59.

McNamara, D. S., Jacovina, M. E., Snow, E. L., and Allen, L. K. (2015). From Generating in the Lab to Tutoring Systems in Classrooms. American Journal of Psychology, 128(2): 159–172.

Proske, A., Roscoe, R. D., and McNamara, D. S (2014). Game-Based Practice versus Traditional Practice in Computer-Based Writing Strategy Training: Effects on Motivation and Achievement. Educational Technology Research and Development, 62(5): 481–505.

Roscoe, R.D., Jacovina, M.E., Harry, D., Russell, D.G., and McNamara, D.S. (2015). Partial Verbal Redundancy in Multimedia Presentations for Writing Strategy Instruction. Applied Cognitive Psychology, 29(5): 669–679.

Roscoe, R.D., Snow, E.L., Allen, L.K., and McNamara, D.S. (2015). Automated Detection of Essay revising Patterns: Application for Intelligent Feedback in a Writing Tutor. Technology, Instruction, Cognition, and Learning, 10(1): 59–79.

Snow, E.L., Allen, L.K., Jacovina, M.E., Crossley, S.D., Perret, C.A., and McNamara, D.S. (2015). Keys to Detecting Writing Flexibility Over Time: Entropy and Natural Language Processing. Journal of Learning Analytics, 2(3): 40–54.

Weston-Sementelli, J.L., Allen, L.K., and McNamara, D.S. (2016). Comprehension and Writing Strategy Training Improves Performance on Content-Specific Source-Based Writing Tasks. International Journal of Artificial Intelligence in Education: 1–32.

Proceeding

Allen, L. K., Mills, C., Jacovina, M. E., Crossley, S., D'Mello, S., and McNamara, D. S. (2016). Investigating Boredom and Engagement During Writing Using Multiple Sources of Information: The Essay, the Writer, and Keystrokes. In Proceedings Sixth International Conference on Learning Analytics and Knowledge (pp. 114–123). Edinburgh, UK: ACM.

Jacovina, M. E., Snow, E. L., Allen, L. K., Roscoe, R. D., Weston, J. L., Dai, J., and McNamara, D. S. (2016). How to Visualize Success: Presenting Complex Data in a Writing Strategy Tutor. In Proceedings of 8th International Conference on Educational Data Mining (EDM). Madrid, ES.