Are emoji a poor substitute for words? Sentence processing with emoji substitutions
- Neil Cohn, Department of Communication and Cognition, Tilburg University, Tilburg, Netherlands
- Tim Roijackers,
- Robin Schaap,
- Jan Engelen,
AbstractWith the integration of emoji into digital keyboards, people are increasingly using multimodal interactions between text and image in real-time interactions. One technique of using emoji is to substitute them into sentences. We here investigate the online processing of these interactions, by modulating either the grammatical category of those substitutions (Experiment 1: nouns vs. verbs) or the type and location of substitutions (Experiment 2: emoji vs. logos, within sentences vs. at their end). We found a processing cost for self-paced reading times of images compared to words, which indeed extended past the emoji itself, but no difference in comprehensibility ratings between word and congruent-image substitutions. Overall, these results suggest that, despite costs of switching modalities, text and images can be integrated into holistic multimodal expressions.