When estimating the number of dots in a set, adults show bias and variability that scale with numerosity. Increasing variance in estimation is thought to reflect constant Weber noise on perceptual magnitude representations, while the increasing bias reflects miscalibrated mappings of number words onto magnitudes. Here we argue that response variability in numerical estimation increases with numerosity in part due to uncertainty and slow drift in the mapping of numbers onto magnitudes. We show that individuals' number-to-magnitude mapping functions drift slowly over the course of the experiment, with a shared-variance half-life of over 100 trials ($\sim 10$ min). We thus propose a model that treats the word-to-magnitude mapping function as a major source of estimation variability, and that accounts for cross-subject differences in estimation bias and variability, as well as changes to estimation performance within a given subject over time. In doing so, we reconcile the existing literature on the sources of estimation variability, and provide evidence that uncertainty in the word-to-magnitude mapping function is a key limiting factor in estimation performance.