We used a novel experimental paradigm to investigate cognitive principles underpinning human communication. Through a computer game simulating different virtual scenes, pairs of participants sent and interpreted non-linguistic, minimal signals to achieve common goals. Participants’ signalling and interpretative actions demonstrated both flexibility and sensitivity to variations in the context of the shared visual scene: the same signal in one context could ‘flip’ its meaning in a new context. Such ‘invertible’ signals in the lab have their counterparts in patterns of real-world natural language use—from the phenomenon of enantiosemy (words/phrases that contain their ‘opposite’ meaning) to the pragmatics of satire and irony. But the emergence of such signals in our experiment challenges both correlational associative and recursive-mentalizing (‘mind-reading’) accounts of human communication and language. Instead, we point to a pragmatics-central perspective in which what is vital is our capacity for joint inference and coordination.