Quantitative analysis has usually highlighted the random nature of linguistic forms (Zipf, 1949). We zoom in on three structured samples of language (numerals; playing cards; and a corpus of artificial languages from Kirby, Cornish & Smith 2008) to quantitative explore and illustrate the idea that linguistic forms are nonrandom in that their structure reflects the structure of the meanings they convey. A novel methodology returns frequency spectra showing the distribution of character n-gram frequencies in our language samples. These spectra, purely derived from linguistic form, clearly reflect the quantitative structure of the underlying meaning spaces, as verified with a new information theoretical metric of compositionality. Moreover, analyses of a diachronic corpus of languages show that linguistic structure gradually adapts to match the structure of meanings over cultural transmission.