To successfully navigate the complex social world, people often need to solve the problem of perspective selection: Between two conflicting viewpoints of the self and the other, whose perspective should one take? In two experiments, we show that four-year-olds use others’ knowledge and goals to decide when to engage in visual perspective taking. Children were more likely to take a social partner’s perspective to describe an ambiguous symbol when she did not know numbers and wanted to learn than when she knew numbers and wanted to teach. These results were shown in children’s own responses (Experiment 1) and in their evaluations of others’ responses (Experiment 2). By preschool years, children understand when perspective taking is appropriate and necessary and selectively take others’ perspectives in social interactions. These results provide novel insights into the nature and the development of perspective taking.