Nearly all models of visual working memory have focused on fitting average memory performance across many simple displays of colored squares. Unfortunately, fitting average performance can mask an underlying failure of the model to account for performance by particular observers in particular displays. We had N=475 observers perform exactly the same set of continuous color report trials (e.g.,Zhang&Luck,2008), and found a significant amount of observer-to-observer consistency in which displays are hardest and easiest to remember items from. While existing models of visual working memory fail to explain this variance, we present an expanded model that accounts for perceptual grouping, the encoding of summary statistics, and attentional selection, and show that such a model considerably improves the fit to the data. We suggest that in order to understand the architecture and capacity of visual working memory, models must take into account differences in the information observers encode about each particular display.