Previous research has shown that auditory training helps non-native speakers learn to perceive difficult phonemic contrasts in a second language (L2) (Hirata, 2004), but there is much room for improvement. Given that hand gestures influence many aspects of native language processing (Hostetter, 2011), we examined whether imitating versus observing gestures helps to improve native English speakers’ ability to perceive novel phoneme contrasts in Japanese as an L2. Participants were assigned to either a gesture observe or gesture imitate training condition. There was no overall training advantage of imitating gestures over simply observing them. However, in a preliminary analysis of a sub-group of participants with low scores on the auditory pretest, observing gestures was actually more beneficial than imitating them on an ERP post-test of auditory perception. The results suggest that producing gestures does not always help with learning, and for particularly challenging auditory perceptual tasks, may actually interfere with it.