There is currently little understanding on whether significant word retrieval difficulties appear during midlife and if so, whether they relate to decrease in grey matter (GM) density that accompanies aging. To answer this question, we studied retrieval of proper names in 125 cognitively healthy middle-aged persons (49.7, ±3.2) comparing their performance on a tip-of-the-tongue (TOT) task with that of 86 young persons (25.4, ±3.5) from the Cam-Can data repository (http://www.mrc-cbu.cam.ac.uk/datasets/camcan/). The middle age (MA) group was worse in word retrieval (U = 23950.5, p = 0.003) and had less GM volume in a range of left fronto-temporal areas relative to the young group, but there were no statistically significant correlations between volumes of the regions known to be implicated in word retrieval and MA’s TOT scores. Thus, midlife word retrieval decline is not associated with GM volume reduction; more likely it is due to changes in connectivity.