In the search for an understanding of human communication, researchers often try to isolate listener and speaker roles and study them separately. Others claim that it is the intertwinedness of these roles that makes human communication special. This close relationship between listener and speaker has been characterized by concepts such as common ground, backchanneling, and alignment, but they are only part of the picture. Underlying these processes, there must be a mechanism for making inferences about our interlocutors' understanding of words and gestures that allows us to communicate robustly and efficiently without assuming that we take the same words to have the same meaning. In this paper, I explore this relationship between language and concepts and propose an interactive mechanism that can facilitate these latent conceptual inferences. Finally, I show how this proposal paves the way for a more precise account of the role of interaction in communication.