Categorical perception describes the phenomenon that visual stimuli can be discriminated more easily when they belong to distinct rather than common linguistic categories. Here we investigate the role of the meaningfulness of linguistic categories in categorical perception. To disentangle the effects of labels and semantic contents of verbal categories we employed a learning paradigm in which participants acquired information about initially unfamiliar objects. Linguistic knowledge was manipulated by labeling object pairs either with the same or different names. Furthermore, the labels could be associated with in-depth knowledge or learned in isolation. Two days after learning, the EEG was recorded while participants performed a lateralized object discrimination task. Verbal labels affected object processing already at about 120 ms, unaffected by semantics, while separate semantic effects were found at about 200 ms, suggesting that the influence of verbal categories on perception may not be modulated by semantic information associated with the categories.