Description:

Co-Principal Investigator: Samarapungavan, Ala; Litman, Diane

Purpose: In this project, the researchers will refine an existing mobile application, CourseMIRROR, for use in postsecondary STEM lecture courses. This application aims to improve deep learning by encouraging students to reflect on course content and receive immediate feedback on their reflections. Often, in large lecture courses, students' ability to reflect on course content and get feedback on these reflections is limited by class size and instructor availability. At the same time, instructors often don't have access to students' reflections, so they cannot correct misunderstandings or build on class knowledge. By leveraging natural language processing and mobile learning technologies, CourseMIRROR aims to overcome these barriers and help students and instructors gain insights into what was or was not learned.

Project Activities: The researchers will iteratively refine and test the CourseMIRROR application for use in postsecondary introductory physics and gateway math courses. They will integrate student and instructor feedback as they develop both the application and the supporting tools (e.g., dashboard) and materials (e.g., professional development materials). Once they have a fully functional version of CourseMIRROR, they will pilot it in classes at a large university and a community college.

Products: Researchers will produce a fully developed CourseMIRROR application for postsecondary students and peer-reviewed publications.

Structured Abstract

Setting: The research will take place at universities in Indiana and Pennsylvania and in a community college in Indiana.

Sample: Approximately 120 students will participate during the development process and 600 undergraduate students will participate in the pilot studies. Approximately 14 university and community college instructors will also take part in the development process with additional instructors helping with the pilot study.

Intervention: CourseMIRROR is a mobile application for smartphones, tablets, or computers that prompts students to write about course lecture concepts at the end of each class. It uses a natural language processing algorithm to analyze students' input and generate summaries of their reflections. Students can use these reflections to identify lecture themes, identify their misunderstandings, and gauge their knowledge. Instructors can leverage the reflections to identify where the class may be struggling, thus allowing them to intervene more effectively. 

Research Design and Methods: Building off an existing prototype of CourseMIRROR, the research team will further develop and test the mobile technologies, web tools, instructor interfaces, and natural language processing algorithms. The final product will also include a professional development module to help instructors implement CourseMIRROR. The design process includes multiple quasi-experimental studies to refine elements of the reflection process (e.g., improving the prompts and feedback students get while writing their reflections) as well as qualitative work such as classroom observations and interviews and focus groups with teachers and students.  This iterative process will include gathering both student and instructor feedback, both quantitative and qualitative (e.g., observation data). The final pilot study will include sections of a university introductory physics course and a community college gateway math course. Instructors of these courses will have a treatment class, which uses CourseMIRROR; and a business-as-usual course, which does not. The researchers will collect student baseline data on their achievement, motivation, and learning strategies and pre- and post-test data.

Control Condition: The comparison course in the pilot study will be a course taught by the same instructor as the treatment course, but the students in it will not use CourseMIRROR.

Key Measures: Student outcome measures include course exam scores, standardized conceptual tests (e.g., the ACT Compass tests), questionnaires (e.g., the Motivated Strategies for Learning Questionnaire), and classroom observations and instructor and student interview data. The researchers will use student demographic data and administrative data (e.g., course-taking patterns) to determine whether impacts vary by student characteristics.

Data Analytic Strategy: The research team will analyze the student outcome data using analysis of variance models for each dependent measure and a multivariate analysis of variance model with multiple dependent measures.

Products and Publications

Journal article, monograph, or newsletter

Menekse, M. (2019). The Reflection-Informed Learning and Instruction to Improve Students' Academic Success in Undergraduate Classroom. The Journal of Experimental Education, 1–17.

Proceedings

Anwar, S. (2019). Impact of Educational Technology-Based Learning Environment on Students' Achievement Goals, Motivational Constructs, and Engagement. In Proceedings of the 2019 ACM Conference on International Computing Education Research (pp. 321–322). ACM.