Speakers must take their addressee’s knowledge into account in choosing to refer to an object using a name or a description. Do speakers keep track of partner-specific information about the common ground status of names? And if so, what mechanisms support this ability? We present a series of experiments that investigate the nature of the memory representations involved in supporting speakers’ ability to distinguish shared from privileged information. The results of these experiments suggest that category information can be used as a cue to aid retrieval of ground status, and that shared experience plays an important role in helping speakers to distinguish privileged information from shared information.