Adaptive Experiments for Enhancing Digital Education: Benefits and Statistical Challenges

Adaptive digital field experiments are continually increasing in their breadth of use in fields like mobile health and digital education. Using adaptiveexperimentation in education can help not only to explore and eventually compare various arms but also to direct more students to more helpful options. For example, they might explore whether one explanation type (e.g., critiquing an existing explanation or revising one’s own explanation) would lead students to gain a better understanding of a scientific concept and assign it more often. In such an experiment, data is rapidly and automatically analyzed to increase the proportion of futureparticipants in the study allocated to better arms.One way of implementing adaptivity is through algorithms designed to solve multi-armed bandit (MAB) problems, such as Thompson Sampling (TS). The MAB problem is to effectively choose among K available options or arms, in order to maximize the expected outcome of interest (or reward). In this work, we present real-world case studies of applying TS in education. Specifically, we explore its use for motivating students, through different email reminders, tofinalize their online homework. To evaluate the potential of MAB in education, we leverage the power of simulations to further explore the behavior of TS both when there is no difference between arms and when some difference exists. We empirically show that, while adaptive experiments can result in an increased benefit for students, by assigning more people to better arms, they can also cause problems for statistical analysis. Notably, this assignment strategy suggests an alert in drawing statistical conclusions, resulting in an inflated Type I error and a decreased power (failure to conclude there is a difference in arms when there truly is one). We explain why this happens and propose some strategies to mitigate these issues, in the hope to provide building blocks for future research to better balance the competing goals of reward maximization and statistical inference.