Individuals use both linguistic and non-linguistic features of the speech signal to identify talkers. Importantly, listeners have more difficulty identifying talkers in unfamiliar languages compared to a native language (language familiarity effect, or “LFE”). In the present study, the source of LFE on talker identification was investigated as listeners identified talkers in their native language as well as non-native languages. Experiment 1 explored the influence of L2 proficiency on talker identification across languages. Experiment 2 investigated individual differences in L1 phonetic perception and their contribution to talker identification by comparing listeners’ performance across different language conditions that varied in the availability of linguistic cues. Results imply that familiarity with a specific language (L1 or L2) did not explain individual variation in LFE. Rather, in addition to the native language benefit, talker identification may be supported by general sensitivity to speech sound structures, modulated by the availability of higher-level linguistic information.