We develop an ideal observer model of human visual short-term memory. Compared to previous models that have posited constraints on memory performance intended solely to account for observed phenomena, in the present research we derive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model using rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information transmission subject to a fixed capacity. The resulting model provides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regarding human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our ideal observer model encompasses two existing, but competing accounts of VSTM as special cases.