Three experiments investigated the frame of reference used in memory to represent vertical spatial layouts perceivable from a single viewpoint. We tested for the selection of three different reference systems: the body orientation, the visual vertical of the surrounding room, and the direction of gravity. Participants learned and retrieved differently colored objects on a vertical board with body and room orientations varying in relation to gravity and each other systematically. Across all three experiments participants were quicker or more accurate in memory recall when they saw the vertical spatial layout in the same orientation in relation to their body vertical as during learning, irrespective of the direction of gravity or visual room upright. These results indicate that spatial long-term memories for small-scale vertical relations are mainly defined in an egocentric reference system with respect to the body vertical despite the availability of alternative highly salient allocentric reference directions.