What Syntactic Structures block Dependencies in RNN Language Models?

AbstractWe provide new evidence that RNN language models are sensitive hierarchical syntactic structure by investigating the filler—gap dependency and constraints on it, known as island constraints. Previous work is inconclusive about whether RNNs learn to attenuate their expectations for gaps in island constructions in particular or in any sufficiently complex syntactic environment. We demonstrate that two state-of-the-art RNN models are are able to maintain the filler–gap dependency through unbounded sentential embeddings and yet are sensitive to the hierarchical relationship between the filler and the gap. Next, we demonstrate that the models are able to maintain possessive pronoun gender expectations through island constructions. We also evaluate three untested islands constraints: coordination islands, left branch islands, and sentential subject islands. Models are able to learn left branch islands and learn coordination islands gradiently, but fail to learn sentential subject islands and coordination islands.

Return to previous page