Speech perception requires ongoing perceptual category learning. Each talker speaks differently, and listeners need to learn each talker’s particular acoustic cue distributions in order to comprehend speech robustly from multiple talkers. This phonetic adaptation is a semi-supervised learning problem, because sometimes a particular cue value occurs with information that labels the talker’s intended category for the listener, but other times no such labels are available. Previous work has shown that adaptation can occur in both purely supervised (all labeled) and purely unsupervised (all unlabeled) settings, but the interaction between them has not been investigated. We compare unsupervised with (semi-) supervised phonetic adaptation and find, surprisingly, that adult listeners do not take advantage of labeling information to adapt more quickly or effectively, even though the labels affect their categorization. This suggests that, like language acquisition, phonetic adaptation in adults is dominated by unsupervised, distributional learning.