Parallel Distributed Processing (PDP) models have been widely used for modeling cognitive tasks where accuracy or reaction time were the dependent performance measures. However, only few PDP models have attempted to model more brain-related data like event related potentials (ERPs). In this paper, we take a step towards using ERP data for model fitting by proposing a PDP model, which can successfully replicate various known ERP effects. Specifically, we introduce a PDP-equivalent of the N400 ERP measure and apply it to a simple PDP model of early bilingual word acquisition as bilingual word acquisition tasks provide several well-established N400 effects that can be used for model validation. We then analyze the dynamics of the network to show why and how the network can capture each of the targeted N400 effects. Furthermore, we qualitatively compare model-generated and empirical N400 peak values for L2 words.