How is musical memory organized? While classic studies of music perception appealed to schematic or symbolic knowledge structures, recent work suggests that listeners form highly-detailed auditory representations of music. Studies of metrical restoration—memory fill-in of the “beat” of a metrically-ambiguous melody—suggest some organizing dimensions in musical memory. However, many potential dimensions remain unexplored. The current study looked for effects of mode (major vs. minor)—a substantial organizing force in Western music—and timbre (what instrument is playing) on metrical restoration. Both mode and timbre can signify particular musical styles. In Experiment 1, listeners showed timbre specificity in metrical restoration, but not mode specificity. However, in Experiment 2, when timbres were extremely unique (one per melody), restoration effects were not observed, suggesting that too much variability leads to diffuse representations which are too weak to support metrical restoration. Implications for the nature of musical memory are discussed.