Across a wide range of face perception tasks, observers show drastically worse performance when faces are oriented upside-down versus upright. However, the meaning of orientation must be established in relation to a particular frame of reference. In relation to which reference frame(s) does the face inversion effect occur? Here we describe a simple, novel method for investigating potentially independent effects of retinal and environmental reference frames on face processing. Participants performed one of two face-processing tasks (emotional expression classification and recognition memory) as they lay horizontally, which served to disassociate the retinal and environmental reference frames. In both experiments we found a large effect of retinal orientation on performance and a small but reliable effect of environmental orientation. In a follow-up control study, we consider an alternative explanation based on our experimental setup. We argue that environmental orientation influences face processing, which is revealed when retinal orientation is kept constant.