We examined how the intrinsic orientation of spatial layouts and the conversational partner’s viewpoint shape how people organize spatial information in memory and subsequently describe it. In 24 pairs, Directors first studied an array with a symmetrical structure while either knowing their Matcher’s subsequent viewpoint or not. When describing the array to the Matcher, the array’s intrinsic orientation was aligned with the Director, the Matcher, or neither partner. Memory tests preceding descriptions revealed that Directors misaligned with the structure organized information according to a priori knowledge, being more likely to use the structure as an organizing orientation when knowing that Matchers were aligned with it. The perspective of Directors’ descriptions was also influenced both by the partners’ alignment with the structure and their advance knowledge of that. Altogether, speakers are guided by converging social and representational cues to adapt flexibly the organization of their memories and perspectives of their descriptions.