Damage to the Medial Temporal Lobe (MTL) impairs declarative memory and perception. The Representational-Hierarchical (RH) Account explains such impairments by assuming that MTL stores conjunctive representations of items and events, and that individuals with MTL damage must rely upon representations of simple visual features in posterior visual cortex. A recent study revealed a surprising anti-perceptual learning effect in MTL-damaged individuals: with exposure to a set of visual stimuli, discrimination performance worsened rather than improved. We expand the RH account to explain this paradox by assuming that visual discrimination is performed using a familiarity heuristic. Exposure to a set of highly similar stimuli entails repeated presentation of simple visual features, eventually rendering all feature representations equally (maximally) familiar and hence inutile for solving the task. Since the unique conjunctions represented in MTL do not occur repeatedly, healthy individuals are shielded from perceptual interference. We simulate this mechanism with a neural network.