Studies show sex differences in working memory (WM) measured by using functional magnetic resonance imaging (fMRI). Effects have been associated with hormonal regulation of prefrontal (PFC) and parietal (PAR) cortices, regions implicated in WM function. Determining the pathophysiology of these sex differences has implications for understanding individual differences in WM. Using fMRI, we assessed WM using an N-back task and acquired hormonal status in 13 males and 13 females. Findings demonstrated sex differences in brain activity and connectivity between PAR and PFC, which were associated with female hormonal status. We suggest that hormones may regulate the ‘gain’ of neuronal activity in PFC and PAR, leading to less diffuse activation in women compared to men, the effect for which we propose a neural network model.