Speech perception is made much harder by variability between talkers. As a result, listeners need to adapt to each different talker's particular acoustic cue distributions. Thinking of this adaptation as a form of statistical inference, we explore the role that listeners' prior expectations play in adapting to an unfamiliar talker. Specifically, we test the hypothesis that listeners will have a harder time adapting to talkers whose cue distributions fall outside the range of normal variation across talkers. We also show that it is possible to infer listeners' shared prior expectations based on patterns of adaptation to different cue distributions. This provides a potentially powerful tool for directly probing listeners' prior expectations about talkers that does not rely on speech produced by many different talkers, which is costly to collect and annotate, and only indirectly related to listeners' subjective expectations.