Cognitive dysfunction caused by some neurodegenerative diseases is associated with an increased risk of traffic accidents. Previous studies have reported inconsistent results for prodromal and early stages of dementia. Few studies have directly compared the effects on driving performance of amnestic subtype of mild cognitive impairment (aMCI) with those by normal aging in elderly drivers. The present study examines the association between cognitive decline and driving ability in elderly drivers with aMCI. The participants were 19 healthy young adults (HYA), 26 healthy elderly adults (HEA), and 12 elderly patients with aMCI. All performed a road-tracking, a car-following, and a harsh-braking task on a driving simulator (DS). Elderly participants also completed cognitive assessment tasks including measures of memory performance. All MCI participants showed a well-defined memory decline, and demonstrated significantly decreased performance on the car-following and road-tracking tasks as compared with the HYA group. However, the aMCI group also demonstrated significantly decreased performance on the car-following task as compared with HEA. In elderly participants, the car-following performance was positive correlated with the score on the Trail Making Test-B. This evidence indicates a difference for driving ability between individuals with symptomatic memory impairment and the aging-related memory of normal controls. This difference may be associated with flexibility of visual attention and executive function.