Time and space are intimately related, but what is the real nature of this relationship? Is time mapped metaphorically onto space, or do the two domains share a common representational format? In the present paper, participants touched (but could not see) physical sticks while listening to an auditory note. Judgements of stick length were affected by concurrent note duration, but not vice versa. When participants were allowed to see as well as touch the sticks, however, the effects reversed. These findings run counter to the spatial metaphor account of time, which claims that effects of space on time should always be stronger than those of time on space. Rather, our findings support the spatial representation account, in which time and space share a common neural substrate that may be affected by concurrent temporal or spatial information, depending on the perceptual acuity of the modality used to perceive space.