Merel Scholman – Talking hands: Prototypical gestural patterns for discourse structure

** This ELiTU talk will take place from 16:00-17:00 instead of the regular time. **

Discourse is structured by coherence relations: logical connections between propositions such as causality, contrast, and temporal sequences (Sanders, Spooren & Noordman, 1992). These relations are often explicitly marked by connectives such as “because,” “although,” and “after” (Knott, 1996). Most research on connectives focuses on written text, but spoken communication is inherently multimodal. Hand gestures serve various functions: iconic gestures depict concrete concepts (speakers may illustrate “exploding” by moving their hands apart), while other gestures structure discourse (speakers may represent “on the one hand” and “on the other hand” by positioning their hands sequentially on different sides).

Concrete referents are often accompanied by iconic gestures (McNeill, 1992), but it has hardly been investigated what systematic gesture patterns hold for more abstract discourse markers such as connectives (Scholman & Laparle, 2025). In this line of work, we therefore investigate to what extent hand gestures accompanying connectives exhibit systematic patterns related to the underlying coherence relation. If connectives elicit consistent gesture patterns, this would point to an integrated multimodal system in which hand gestures fulfill both iconic and structural discourse functions.

In this talk, I will present data collecting in a controlled production task with participants (n=91) at a science festival. Participants read aloud fragments from talk shows (2 sentences per fragment) and produced hand gestures at marked moments. For each fragment, we elicited two gestures: first for an iconic word (e.g., “explosion”), then for a connective (e.g., “because”). We used eight items with different coherence relations (causality, contrast, temporal sequences, etc.). Video recordings were coded for gesture type and formal features. Analyses focused on consistency per connective type and relation type.

Most of the connectives were often expressed using a gesture known as PUOH (palm-up-open-hand, 32%), indicating that speakers often resorted to a specific default gesture. At the same time, when looking at individual connectives, the results reveal item-specific modulation of this default pattern. Contrastive connectives such as but show increased use of block- and finger-point schemas, whereas temporal connectives such as then and after that display more distributed profiles across multiple schemas. This suggests that while PUOH functions as a general fallback strategy for abstract meanings, individual connectives afford additional, more specific schematizations. In the presentation, I will go deeper into specific gesture profiles for different types of discourse structures, and the conclusions we can draw from this regarding the cognitive representation of discourse structure.

References:

Knott, A. (1996). A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh.

McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago press.

Sanders, T. J., Spooren, W. P., & Noordman, L. G. (1992). Toward a taxonomy of coherence relations. Discourse processes, 15(1), 1-35.

Scholman, M., & Laparle, S. (2025). Can gestures speak louder than words? The effect of gestural discourse markers on discourse expectations. Discourse Processes, 1-22.

Microsoft Teams meeting

Join: https://teams.microsoft.com/meet/32358990684274?p=q1VZbZK5MjPdZbo2G3

Meeting ID: 323 589 906 842 74

Passcode: bo7nm9UE