Iris Mulders & Eddy Ruys- LLMs as informants

** This ELiTU talk will take place on a Tuesday from 15:30-16:30. **

Abstract: Generative large language models (LLMs) produce unconstrained text with few grammatical errors. This is surprising in view of what is known as “the logical problem of language acquisition.” We assessed the linguistic competence of LLMs by asking them for judgments like one would human informants, performing two sets of experiments. The first set tested two LLMs on their ability to judge sentences as (un)grammatical (observational adequacy), and found that GPT-4’s performance could not be distinguished from our human participants. The second set of experiments tested five LLMs on their ability to distinguish structurally ambiguous from unambiguous sentences in Dutch, and to assign the correct interpretations to the unambiguous variants (descriptive adequacy). In order to minimize the effects of pragmatics, an additional experiment was performed with stimuli containing pseudowords. We found that the LLMs indeed show an ability to recognize structural ambiguity and to choose the correct semantic interpretations, although they perform worse than humans, and show different biases.

Microsoft Teams meeting

Join: https://teams.microsoft.com/meet/318165565974357?p=axgnuzO3d59W7zhc2m

Meeting ID: 318 165 565 974 357

Passcode: Vw3uM7gg