Kohei Haneda – Syntax-Vision Interface: The Impact of Visual Cues in Real-time Ellipsis Resolution

** This ELiTU talk will take place from 13:00-14:00 instead of the regular time. **

Kohei Haneda (International Doctorate for Experimental Approaches to Language And Brain, University of Groningen, Macquarie University, and University of Potsdam),
Anja Schüppert (University of Groningen),
Roel Jonkers (University of Groningen),
and Anita Szakay (Macquarie University)

Abstract: Comprehenders can capitalize on non-linguistic visual information to maneuver their way through online sentence comprehension. But can they use such visual cues to resolve an elliptical gap? Processing ellipsis generally requires that a linguistic antecedent be retrieved at the time of encountering an elliptical gap. When one says, “Last Friday, Joanna was knitting a scarf. Elise was too”. The verb phrase from the first sentence serves as the antecedent, which is later retrieved when a parser encounters “was too”. What if this linguistic antecedent is replaced with a visual cue such as an image depicting an action? What would happen if, instead of reading, “Joanna was knitting a scarf”, a comprehender viewed an equivalent image of a woman knitting a scarf, and then read “Elise was too”? In two web-based Self-Paced Reading experiments, 60 L1 English and 60 L1 Dutch readers viewed images depicting various actions followed by sentences with elliptical gaps. English participants saw Verb-Phrase Ellipsis, while Dutch participants saw Sluicing. Participants’ reading times at the ellipsis sites and spillover regions were compared to those of the controls to examine whether or not the elliptical resolution with a visual antecedent was successful. English results indicated that visually-situated processing of ellipsis may have a processing advantage over its linguistic counterpart, suggesting that mental representations constructed multimodally may be less cognitively taxing than purely linguistic ones. Dutch results, however, showed no such effect; instead, processing Sluicing remained equally cognitively demanding for both linguistic and visual antecedent conditions. The combined findings point to the possibility that visual cues may facilitate online ellipsis resolution when the to-be-retrieved antecedent is relatively small informationally (a verb phrase); however, such a benefit vanishes when the antecedent is too large (a whole clause), potentially delineating the capacity of visual information to infiltrate into real-time sentence processing.

Keywords: Ellipsis, Visual cues, Sentence processing, Language comprehension, Multimodality