Project Activities
Focused program of research
The WRITE AI Center will carry out an exploratory study on the use of generative AI in postsecondary writing instruction, combining a national survey with in-depth case studies. Extensive information on the top 100 AI writing tools will be made publicly available in a searchable database. Insights from this study will also inform development and pilot testing of PapyrusAI, a generative AI based writing tool designed to enhance student writing and AI literacy. Partnering with six community colleges across three states, the research team will iteratively develop PapyrusAI, assess its feasibility and usability, evaluate its impact on student writing and on instructors’ and students’ AI literacy and self-efficacy, analyze its cost of implementation, and produce an implementation support guide.
Research Training: The WRITE AI Center will develop an online research hub to support both comprehensive learning through an annual fellows program and just-in-time access for the broader research community. Training programs will focus on 1) developing robust frameworks for evaluating the instructional effectiveness, scalability, biases, and ethical implications of generative AI tools; 2) accessing, cleaning, and preparing platform data with attention to privacy and quality; 3) measuring learning engagement and platform behavior using advanced data analytics; 4) conducting causal research via A/B testing and other innovative methodologies; and 5) employing mixed-methods approaches that integrate advanced statistical and qualitative techniques to understand generative AI tool usage and impact.
National leadership and outreach activities
To guide policymakers, educational leaders, platform developers, educators, and the broader public in adopting research-based approaches to the use of generative AI in postsecondary education the WRITE AI Center will 1) disseminate actionable information and research findings through a website, webinars, training programs, social media, research briefs, preprints, publications, and presentations; 2) foster collaboration by convening stakeholders in an annual summit, innovation labs, and national conferences; and 3) create and share resources to support effective use of generative AI in postsecondary education.
Structured Abstract
Setting
The Center’s activities will be national with respect to all activities except the development and pilot work, which will be carried out in California, Texas, and Virginia. In those states, we will work with 6 community colleges, purposely selected for representing varied rural, suburban, and urban communities.
Sample
The exploratory case study will be based on a nationally representative sample of public postsecondary institutions. Development work will be done with 12 teachers and the RCT (pilot) done with 60 teachers across our colleges. Participants will be drawn from first-year composition courses, plus developmental and disciplinary writing classes in the development phase, and each teacher will use our instructional resources and PapyrusAI tool (the “intervention”) in at least one course of approximately 20 students.
Research design and methods
The exploratory research study will consist of a national survey intended to generate generalizable information on the most used generative AI tools for writing instruction. We will gather information on each tool’s instructional purposes, goals and course applications; implementation including available guides/manuals, costs, infrastructure requirements, and training needs; focus on safety, bias, and fairness; and available evidence of each tool’s impact on postsecondary students’ academic outcomes. The findings will be provided and updated on the Center’s website not less than annually in a publicly accessible and usable reference for researchers and practitioners.
The development and pilot study will begin with two years of design-based research with 12 teachers across our 6 community colleges. We will use an iterative design process with cycles of requirement analysis and design followed by development. The pilot study will consist of a randomized controlled trial to examine the potential promise of our intervention. Treatment teachers will participate in professional development and use the instructional resources and PapyrusAI for instruction in at least one first-year composition course. Control teachers will conduct classes business as usual and will not have access to the instructional resources or PapyrusAI until the following year.
Key measures
The research team will conduct pre and post assessments of both teacher and student AI literacy and self-efficacy based on validated measures. They will assess student writing quality pre and post, as well as distal measures such as grade in next writing course and persistence in enrollment.
Data analytic strategy
The exploratory study will use standard descriptive statistics to analyze tool use overall, and by type of institution, by region, and by content area. Qualitative analyses will be used in both the exploratory study (for open responses to the questionnaire) and the development study to understand themes related to tool use and guide our development efforts. For the RCT, researchers will use a three-level model of students nested within classes nested within teachers, using fixed effects for districts and states. The team will explore moderation by student characteristics by including interaction terms between focal moderators (e.g., English language status, initial writing score, and full or part-time enrollment) and the treatment indicator.
Cost analysis strategy
The cost study will document the type and quantity of resources needed to implement the intervention, including the cost of AI (API costs, Amazon Web Services hosting costs) and technology access (percentage of use), amount of technology support utilized, and hours of PD using the Ingredients Method.
People and institutions involved
IES program contact(s)
Project contributors
Partner institutions
MDRC
Virginia Community College System
Houston City College
Foothill-De Anza Community College District
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
