Query-guided visual search
- Junyi Chu, Brain and Cognitive Sciences, MIT, Cambridge, Massachusetts, United States
- Jon Gauthier, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States
- Roger Levy, Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States
- Josh Tenenbaum, Brain and Cognitive Sciences, MIT, Cambridge, Massachusetts, United States
- Laura Schulz, Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States
AbstractHow do we seek information from our environment to find solutions to the questions facing us? We pose an open-ended visual search problem to adult participants, asking them to identify targets of questions in scenes guided by only an incomplete question prefix (e.g. ”Why is...”, “Where will...”). Participants converged on visual targets and question completions given just these function words, but the preferred targets and completions for a given scene varied dramatically depending on the query. We account for this systematic query-guided behavior with a model linking conventions of linguistic reference to abstract representations of scene events. The ability to predict and find probable targets of incomplete queries may be just one example of a more general ability to pay attention to what problems require of their solutions, and to use those requirements as a helpful guide in searching for solutions.