Precision Education: The Virtual Learning Lab

The Virtual Learning Lab will conduct a focused program of research to learn whether precision education can improve students' ability to pass an end-of course exam. The researchers will address the challenge of improving low-achieving students' achievement in algebra, using Algebra Nation, a free online learning platform for students and teachers designed to promote mastery of basic algebra, as a virtual laboratory. The team will personalize instruction using learning analytics; develop sensor-free estimates of engagement during learning (i.e., using students' interactions with the technology and contextual cues to infer engagement); and design professional development to help teachers use learning analytics to differentiate their instruction. This research program will serve as a national model for developing online systems that can use prior student data to optimize, update, and refine the instruction provided to current users.

Measurement Study 1: In Years 1 and 2, the research team will use data from prior students of Algebra Nation to develop recommendations for new users of the platform. They will use student records from across the state of Florida for three school years (a separate group of students each year) in which Algebra Nation was implemented, including students' prior year's test scores, demographic information, and end-of-year test scores. The research team will mine the data to identify paths taken through Algebra Nation by students who performed well and those who did not. These analyses will inform the development of a personalized recommendation system, which will be implemented in Algebra Nation. The research team will pilot test a personalized version of Algebra Nation in six to eight classrooms to evaluate feasibility and to solicit feedback from students about whether the personalized recommendations for resources and activities are helpful to them and are at the appropriate difficulty level. Analyses will also examine optimal classroom implementations of Algebra Nation, which will provide the foundation for the development of an effective professional development program to help teachers implement Algebra Nation in their classrooms and use the rich student data available to personalize instruction.

Measurement Study 2: In Years 1 and 2, the research team will develop an automated measure of engagement, which will be used to enhance the personalized learning models that will be implemented in Algebra Nation. The engagement measure will be developed by applying machine learning methods (i.e., methods that use data to develop models and algorithms that can make predictions) to data about students' actions in Algebra Nation and relevant contextual cues. In small-scale studies in school computer labs, students will interact with Algebra Nation, and self-reports of engagement will be administered. Researchers will use these data to validate the accuracy of the engagement measure. In a subset of the computer lab sessions, the researchers will conduct live field observations and record videos of student engagement, affect, and behavior. These data will be used to obtain evidence for the convergent validity of the automated engagement measure.

Experimental Study 1: In Year 3, the research team will conduct a pilot study using a quasi-experimental design. They will implement the personalized version of Algebra Nation in 30 classrooms (approximately 750 students) and will use 120 classrooms that used the original Algebra Nation in the previous year as the control condition. The Florida end-of-year course exam for algebra will serve as the primary outcome measure, along with estimates of engagement derived from log file data using the automated techniques described above, with the prediction that estimated engagement levels should be higher for students who work with the personalized version of Algebra Nation.

Experimental Study 2: In Year 4, the research team will conduct a randomized controlled trial that will compare the personalized version of Algebra Nation to the original, non-personalized version. A total of 111 schools, with two classes taught by different teachers from each school, will participate in this study. Researchers will recruit classrooms in Florida and other states, depending on the progress of Algebra Nation's national expansion plan. Within each school, the research team will randomly assign one classroom to the personalized version of Algebra Nation and the other to the original version of Algebra Nation. Each classroom is expected to include 25 students on average, for a student sample of 2,825. Key measures will include the researcher-developed measure of engagement, the Florida Comprehensive Assessment Test, and the Florida Standards Assessment. In addition, the research team will collect student data (demographic data, archival data, process data from interactions with the Algebra Nation system, and student performance on the end-of-course algebra exam, class grades, and pass/fail outcomes), data about teachers' experience and credentials, and log file data to assess fidelity of implementation of Algebra Nation. The research team will use multi-level modeling to analyze the data. Data analysis and dissemination will continue through Year 5.