Annie Johansson
PhD Candidate
Optimizing Personalized Learning at Scale
Department of Psychology
University of Amsterdam
Three Strikes and Who is Out? Individual Differences in Error-Induced Quitting
Preprint (PDF) · GitHub repository · PsyArXiv
Abstract
No matter how well a learning system is designed, it becomes irrelevant when students quit. Once learners disengage, learning ceases. Crucially, we show that sequential errors are an important cause of quitting from learning. Little is known about how students differ in their sensitivity to errors. Using intensive longitudinal practicing data from over 200,000 primary-school students in a large-scale online learning environment, we show that sequential errors strongly increase the probability of quitting from learning. Importantly, we find large variability in this effect, ranging from no or small tendencies to quit to high sensitivities to quitting following sequential errors. We also found interaction effects involving age, difficulty level, time of day, and response times. We validate these results in an independent dataset and show that individual differences are stable across two arithmetic practice domains. Our results show that students differ in their tolerance to failure, and we discuss how to deal with these individual differences.
Figure. Left: Baseline quitting rates (intercept) and effects of sequential errors on quitting (slope) for 300 randomly sampled users, across both the addition and subtraction domain. Data points are ordered from lowest to highest effect estimate. Horizontal lines denote the fixed effect; vertical lines represent the 95% confidence interval of each user's effect. Points whose estimate is not significantly different from the average main effect are shown with reduced opacity. Right: Scatterplots representing the correlation between random effects in the addition and subtraction game. The shaded region represents the 95% confidence interval.
Key Findings
- Sequential errors strongly predict quitting. Making several errors in a row is one of the strongest predictors of a student stopping their practice session — a finding that holds across over 200,000 students.
- The effect of errors on quitting differs across contextual and individual factors. Specifically age, difficulty level, time of day, and response times moderate the effect. Importantly, these individual differences were found to be stable across two arithmetic practice domains, suggesting they reflect consistent personal traits rather than situational fluctuations.
- Individual differences are large. On top of group-level differences, students differ dramatically in how sensitive they are to sequential errors on an individual level. Some show little to no tendency to quit after mistakes, while others are highly likely to quit following them.
- Differences are stable across domains. A student's sensitivity to error-induced quitting in addition practice predicts the same sensitivity in subtraction — showing this is a stable individual characteristic, not noise.
Data & Methods
We analyze longitudinal practice data from Prowise Learn, a large-scale adaptive online learning platform used by primary school students across the Netherlands. Our dataset contains responses from over 200,000 students practicing arithmetic. First, we estimated dynamic transition probabilities between persisting and quitting states using a Markov model. Then we use mixed-effects models to estimate a student-level random effect of sequential errors on quitting. Findings are validated in an independent dataset.