Language processing in natural discourse: exploring the interplay between language production and comprehension of grammatical functions in short story contexts
In this project, I investigate spoken language processing - production and comprehension of transitive sentences such as (i) ‘Hon vattnar blommorna’. I focus on how the processing of grammatical functions of subject, ‘Hon’ in (i), and object, ‘blommorna’ in (i), is influenced by whether sentences occur in appropriate discourse contexts consisting of short stories.
The project consists of three experimental studies that investigate the processing of subject-initial - as (i) - and object-initial - as (ii) ‘Blommorna vattnar hon’ – sentences when they occur in short stories. Study 1 investigates how speakers express transitive events, and whether they more frequently use structurally ambiguous sentences as (ii) when the discourse context may facilitate comprehension. Study 2 investigates whether the discourse context do facilitate comprehension in these cases by measuring processing costs during comprehension of structurally ambiguous sentences as (ii), based on the listener’s brain activity. Using eye-tracking, study 3 investigates whether listeners can predict grammatical functions of structurally ambiguous sentences as (ii), when the sentences occur in facilitating discourse contexts.
The project will provide novel insights into the relationship between sentence level- and discourse level- language processing, and the relationship between language production and comprehension. It will also complement earlier tentative findings since it´s using naturalistic experimentation.
Final report
1. Introduction
The project was initiated at the start of 2019 and was finished june 2022. The covid pandemic had a strong impact on data collection and thus affected the time line of the project, and the plans needed to change accordingly. Despite of this, the project has broaden our theoretical understanding of language processing and provided novel and groundbreaking results that have contributed to the international research front. The project has resulted in international collaborations and several international disseminations in high-profile journals and conferences.
This summary is structured as follows. In section 2, ("Research aim") a short introduction of the aim of the research project and it's research questions are presented. In Section 3 ("The impact of the covid pandemic"), the ramifications of the covid pandemic on the project is presented. In section 4 ("Individual studies"), an overview of the development, methodological setup and results (if available) of the individual studies are presented. Section 5 ("Summary") summarizes the findings of the studies taken together. In Section 5 ("Project outreach and dissemination"), the international disseminations and collaborations that have resulted from the project are summarized.
2. Research aim
The project investigates spoken language processing of transitive sentences in Swedish such as as (i) "Flickan retar pojken" (‘The girl teases the boy’). It investigates how the grammatical functions of subject, "Flickan" (‘the girl’), and direct object, "pojken" (‘the boy’), are linguistically encoded during production, on the one hand, and propositionally decoded during comprehension, on the other. Grammatical functions allow us to communicate how participants or items are related to events or states. Our understanding of how we process them is therefore crucial for our knowledge about how we understand and produce language more generally.
Much research has been done on how various information types (i.e., linguistic cues) affect production and comprehension of grammatical functions in isolated sentences. Less is known, however, about the underlying processes that utilize such linguistic cues to grammatical functions during on-line comprehension. Linguistic cues could either have a privileged role in language processing, or their effects could originate from implicit expectations based on their distributions in previous language input. It is also unclear how sentence-level linguistic cues to grammatical functions are utilized in natural discourse-pragmatic contexts, such as short stories, where other discourse-level cues are available. The aim of this project is therefore to investigate how the processing of grammatical functions in Swedish is influenced by the interplay of linguistic cues, and how this interplay is affected by the availability of a natural discourse context. The project is concerned with the following research questions:
• Is the processing of linguistic cues to grammatical function expectation-based, i.e., is it based on the distribution of cues in previous language input?
• Do speakers balance their use of sentence-level cues to grammatical functions in a communicatively efficient manner even when discourse level cues are available in the discourse contexts?
• Can processing difficulties due to cue conflicts during the comprehension of grammatical functions be avoided or mitigated if a natural discourse context is provided?
• Can comprehenders predict unexpected grammatical function assignments during language comprehension if a natural discourse context is provided?
The project consists of a expectation-based, Bayesian model of incremental grammatical function assignment (study 0), a sentence production experiment (study 1), an event-related-brain potentials (ERP) experiment (study 2), and a visual world eye-tracking study (study 3).
3. The impact of the covid pandemic
The work on the stimulus materials for studies 1-3 took longer than anticipated. The stimulus materials were finished at the beginning of the second year of the project, 2020 (it was very hard to anticipate how long this work would take). There was therefore no time to pilot the EEG and the eye-tracking study during 2019.
When the covid pandemic broke out in the beginning of 2020, data collection had to be postponed further as the phonetics lab was closed. A research assistant who would collect the data was hired at the beginning of 2020, but his contract was terminated and postponed until the start of the fall semester of 2020. Data collection for study 1 was therefore finished at the end of october 2020. Data collection for study 2 stopped in the end of january 2021, and data collection for study 3 was finalized in the beginning of july 2021.
Because of this delay, the project plans needed to be adapted. In april 2020, in the midst of the pandemic, work on a expectation-based, Bayesian computational model of incremental argument interpretation therefore begun. This work, henceforth referred to as "Study 0", was conduction in collaboration with professor Florian Jaeger at the department of Brain and Cognitive Sciences at Rochester University, NY.
At the time of writing, a journal article on Study 0 (i.e., the Bayesian computational model and an empirical validation of the model predictions based on previously collected data) has been published in the academic journal "Frontiers in Psychology". Study 1 has been finished and published in the academic journal "Language, Cognition and Neuroscience". Also data from study 3 has been analyzed and the analysis of data from study 2 has begun.
4. Individual studies
4.1. Study 0: an expectation-based model of incremental grammatical function assignment
4.1.1. Introduction and background
Several studies have shown that a multitude of linguistic cues are utilized during on-line language comprehension, such as during grammatical function assignment. This involves morphosyntactic cues (e.g., case marking, word order and agreement), lexical-semantic cues (e.g., animacy), and verb-semantic cues (e.g., volitionality). Less is known about the cognitive processes that utilize these cues. Linguistic cues could have a privileged role in language processing in the sense that they constitute a special class of information that is utilized by the sentence comprehension system (e.g., Bornkessel-Schlesewsky & Schlesewsky 2006). It could also be that there is nothing special about linguistic cues in this sense. Instead, their value as cues might stem from statistical properties in their distribution in previous language input. On this view, language processing is expectation-based: the language comprehension system utilizes statistical regularites in the input during on-line interpretation. Processing difficulties stem from violations of expections in the incoming language input.
In study 0, I present an incremental, Bayesian, expectation-based model of incremental grammatical function assignment. This model is trained on the distribution of linguistic cues and their interactions in transitive sentences, collected from a corpus of transitive sentences in written Swedish (see Hörberg, 2016; 2018). The model is then evaluated against empirical reading time (RT) data from a self-paced reading study that was collected during my PhD (Hörberg, 2016). This is done by comparing the expectation-based model against a linguistic model, in terms of how well these models are able to explain the RT data.
In the following, I provide an overview of the expectation-based model. I briefly describe the self-paced reading experiment and then finally present the results and the analyses, comparing the expectation-based model against the linguistic model.
4.1.2. An expectation-based model of incremental grammatical function assignment
Comprehenders incrementally update their implicit expectations about the underlying sentence interpretation as new input becomes available (Hale, 2001; Levy, 2008). In expectation-based models, sentence interpretation involves continuously shifting from a prior to a posterior probability distribution over possible interpretations, a process known as "Bayesian belief-updating". The processing cost associated with new input is in part determined by the amount of new information provided by the input—the degree of shift in expectations or beliefs about the interpretation (Levy, 2008). This shift can be quantified in terms of Bayesian surprise, which is equivalent to the Kullback-Leibler (KL) divergence of the posterior distribution with respect to the prior distribution.
The model is concerned with canonical Swedish transitive clauses with subject-object (SO) order, and object-initial sentences with object-verb-subject (OS) order (see Hörberg, 2018). It estimates the Bayesian surprise of encountering the first NP, the verb(s), and the second NP of such clauses, with respect to grammatical function assignment (SO or OS). The model thus captures the change in expectations about whether the first or the second NP is the subject based on the linguistic cues available at constituent Ci (e.g., the second NP, NP2) with respect to the cues available at the previous constituent Ci-1 (e.g., NP1 and the verb).
In order to calculate Bayesian surprise of a constituent, the relevant prior and posterior probability distributions need to be estimated. In the model, this is done on the basis of corpus data. The data consists of 16,552 transitive sentences from the Svensk Trädbank treebank (Nivre & Megyesi, 2007). These sentences are canonical transitive sentences with SVO order, object-initial transitive sentences with OVS order, and adverbial-initial sentences with VSO or VOS order and are annotated for morphosyntactic (e.g., case-marking, auxiliary verbs), syntactic (embedding, verb-initial vs. verb-medial word order), prominence (e.g., animacy, person, givenness, definiteness), and verb semantic cues (e.g., volitionality, sentience). Bayesian surprise is estimated at three sentence regions: NP1, the verb, and NP2, derived from the probability distribution of OS vs. SO order at four different points in the sentence: (i) at the clause onset prior to any sentence input, (ii) after NP1 has been processed, (iii) after NP1 and the verb has been processed, and (iv) after NP1, the verb, and NP2 has been processed. The Bayesian surprise at NP1 is the KL divergence between the distribution of OS vs. SO after NP1 has been processed (ii) and the distribution of OS vs. SO at the clause onset prior to NP1 (i), etc. These distributions of OS vs. OS order at (i)-(iv) were estimated by fitting four separate Bayesian mixed-effects logistic regressions (GLMMs). Each of these four GLMMs included all annotated cues available up to that point in the sentence.
4.1.3. Testing model predictions
The models' predictions were tested against reading times from a self-paced reading experiment conducted by Hörberg (2016). The stimulus materials consist of transitive sentences with 1st or 2nd person personal pronoun subjects that vary with respect to word order (SVO vs. OVS), object animacy (animate vs. inanimate), and verb class (volitional vs. experiences). The experiment also contained filler sentences included to ensure that subjects also occured as nouns. In the experiment, participants read sentences one word at a time, exposing the subsequent word themselves with a button press. Time intervals between button presses are assumed to reflect processing costs. These should correlate with the models' predicted Bayesian surprise. The effects of linguistic cues on RTs show a qualitativey similar pattern to their effect on Bayesian surprise. As confirmed by statistical analyses, Bayesian surprise predicts RTs, showing also a quantitative relationship between Bayesian surprise and RTs.
As a final step, we also compared the expectation-based models' ability to predict RTs to the predictive power of a “linguistic” model, predicting RTs directly from the sentence properties (i.e., word order, animacy, and verb class) directly. This analysis found that our expectation-based model performs equally well as the linguistic model. In other words, Bayesian surprise largely captures the same predictive information about RTs as the individual linguistic cues. This provides strong support for the expectation-based account since our estimate of Bayesian surprise by neccessity contains less information (in terms of fewer degrees of freedom) about RTs than the information that is provided by the linguistic cues directly. This finding provides strong support for the notion that processing costs associated with linguistic cues are mediated through Bayesian surprise, and that grammatical function assignment is expectation-based.
4.2. Study 1: speakers balance cues to grammatical functions in informative discourse contexts
4.2.1. Introduction and background
Study 0 shows that linguistic cues are utilized in a probabilistic manner during on-line comprehension. Comprehenders' processing costs are predicted by the extent to which linguistic cues violate expectations regarding grammatical function assignment. Study 1 investigates whether speakers balance their use of linguistic cues to grammatical functions in a communicatively efficient manner. In other words, study 1 investigates whether speakers avoid redundant cues in order to reduce production costs, but only to the extent that the message provides enough cues for the listener to be able to determine grammatical functions. Although some studies have provided evidence for the notion that grammatical encoding is influenced by communicative efficiency (Kurumada & Jaeger 2015; Hörberg, 2018), this evidence has primarily been indirect (i.e., based on corpus- or sentence recall data). Further, no study has investigated whether speakers balance their use of cues in a communicatively efficient manner in natural discourse contexts, where other discourse-level cues are available. It could be that discourse-level cues are sufficient for interpretation, rendering the need for additional cues redundant. Study 1 thus asks whether speakers balance their use of cues to grammatical functions in order to avoid both redundancies and ambiguities when producing sentences in story contexts. As shown in study 0, animacy (i.e., whether an NP refers to a living thing or not) functions as an important cue to grammatical functions (i.e., subjects are most frequently animate). Study 1 thus investigates whether speakers balance their use of morphosyntactic cues (e.g., word order and case marking) with respect to animacy.
4.2.2. Materials and methods
Study 1 is a picture description task where participants describe transitive events occuring in short stories that are accompanied with cartooned images (similar to Prat-Sala & Branigan, 2000). The stories consist of scenes, presented one scene at a time, that are divided into three parts. Part 1 consists of one scene which introduces the protagonist of the story ("This is Eva") and presents the situational context ("She is the manager of a staffing agency"). Part 2 and 3 contain two separate scenes. In the first scene, either two animate antagonists ("Before lunch, Eva interviewed a man and a woman") or two inanimate objects ("Before lunch, Eva received several invoices and an interim report") are presented. In the second scene, the protagonist interacts with the antagonists or the objects. Importantly, these events involve an opposition in the way the protagonist interacts with the antagonists or objects and thus license the use of a contrastive construction. Here, the opposition in the involvement of the object referents is signaled by fronting of the direct object, resulting in OVS word order (Hörberg, 2018). These events can thus be described with a transitive sentence with either SO ("She liked the man, but did not like the woman" / "She paid the invoices, but forgot the report") or OS word order ("The man she liked, but the woman she did not like" / "The invoices she paid, but the report she forgot"). Part 2 and 3 differ with respect to the animacy of the object referents, and are therefore referred to as the "animate" and the "inanimate" event. Participants are exposed both to the animate and the inanimate event, starting with either the former or the latter. In the first event, participants listen to a SVO sentence or an OVS sentence that describes the event and that functions as a prime sentence. In the second event, participants describe the event depicted in the image themselves. With event animacy (animate vs. inanimate) crossed with prime type (SVO vs. OVS), there are four picture description conditions: Animate / SVO priming, Animate / OVS priming, Inanimate / SVO priming, and Inanimate / OVS priming.
The stories were recorded in the anechoic chamber at the Department of Linguistics at Stockholm University. They were spoken by a 45-year-old female professional speech coach from the Stockholm area. The pictures used in the stories were drawn by a professional artist. These consisted of drawings of 59 individuals who were the protagonists and/or antagonists in the stories, 33 inanimate objects, 32 inanimate events and 32 animate events. Two sets of 16 full stories with an animate and an inanimate event were constructed. Four story versions, corresponding to each of the four conditions, were constructed for each of the stories, resulting in 16 (item) × 2 (set) × 4 (condition) = 128 stories in total.
A total of 65 native Swedish speakers (39 female) performed the experiment. Their age had a median of 27 years and ranged from 18 to 68 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent. Responses that were not in accordance with the instructions was excluded from further analysis.
4.2.3. Predictions
As explained above, the events-to-be-described differ with respect to the animacy of the object referent. If speakers are sensitive to this animacy manipulation in their encoding, events with inanimate object referents should more frequently be described with OVS word order. Further, if speakers are sensitive to the availability of cues to grammatical functions more generally, OVS word order should also be used more frequently with other morphosyntactic cues, and such cues should further be used more frequently in sentences with two animate NPs.
4.2.4. Results and discussion
As confirmed by statistical analyses (not reported), OVS word order is produced more frequently in descriptions of inanimate events. The statistical analyses also show that OVS word order is used more frequently when the direct object is inanimate (independent of event type), when the subject NP is case marked (i.e., consists of personal pronoun), and when other syntactic cues to grammatical functions are available (e.g., such as an adverbial). Additional analyses also found that subject case marking is used more frequently in sentences with animate direct objects, independent of word order. There is, however, no effect of prime type. OVS word order is used more or less equally frequently in the descriptions of both SVO and OVS prime stories.
Taken together, these findings show that speakers balance their use of both semantic and morphosyntactic cues to grammatical functions when producing sentences in a discourse context that is informative about grammatical functions in itself. OVS word order is preferred when other cues to grammatical functions are available. In other words, speakers prefer to use alternative word order patterns such as OVS?which are potentially ambiguous with respect to grammatical functions?only when other cues are available, even though the discourse context provides enough information for the listener. Speakers also seem to balance their use of other cues such as case marking and animacy.
4.3. Study 2: Are processing difficulties during grammatical function assignment mitigated by an informative natural discourse context?
4.3.1. Introduction and background
Previous studies (e.g. Hörberg et al. 2013) and study 0 show that processing difficulties during the comprension of transitive sentences stem from the extent to which linguistic cues violate expectations regarding grammatical function assignment. However, these studies have only investigated processing of sentences presented in isolation, independent of the discourse context. The discourse context can provide additional cues to grammatical functions, rendering the need for cues on the sentence level redundant. Processing might therefore not be affected by the availability of sentence-level cues in coherent and informative discourse contexts.
The aim of study 2 was thus to investigate whether processing difficulties during grammtical function assignment in spoken language comprehension that result from sentence-level cue conflicts are affected by the information available in the discourse context.
4.3.2. Materials and methods
The study is an event-related brain potentials (ERP) study, where participants listen to short stories. In order to ensure that participants stay attentive to the stories, they are asked comprehension questions after some of the stories. The critical dependent measure is the ERP response to the subject pronoun in OVS sentences. In such sentences, the subject pronoun disambiguate the sentence towards OVS, engendering an ERP response (see Hörberg et al. 2013). This response should be affected by the availability of other sentence-level cues (i.e., animacy) and / or discourse-level cues (i.e., discourse coherence, see below).
The stories are highly similar to those of study 1. The main difference is that they only contain one event which either is animate (i.e., "animate stories") or inanimate (i.e.,"inanimate stories"). This event is described with either an SVO or OVS sentence. In the first scene, the protagonist and the story context is presented ("This is Eva. She works as the manager of a staffing agency"). The second scene introduces either two animate antagonists ("Before lunch, she interviewed a man and a woman") or two inanimate objects ("Before lunch, she received several invoices and an interim report"). In the third, critical scene, a contrastive event is presented in which the protagonist interacts either with the animate antagonists (animate stories) or the inanimate objects (inanimate stories). These events are described with a transitive sentence with either SO ("She liked the man, but not the woman" / "She paid the invoices, but forgot the report") or OS ("The man she liked, but the woman she did not like" / "The invoices she paid, but the report she forgot") word order.
In order to control for the influence of discourse context coherence on the processing of the critical sentences, the experiment also contains control stories that are matched to the critical stories. These only differ from the critical stories in the second scene in which the antagonists or objects are introduced. In the control stories, the two antagonists or objects are never introduced, and instead a more general statement is used ("Before lunch, she interviewed some applicants" / "Before lunch, she had a lot to do"). With this context, the upcoming contrastive event of the final scene becomes infelicitous, and should be unexpected. The processing of the potentially ambiguous OVS sentences should therefore not be facilitated by the context in the control stories.
With the critical sentence type (SVO vs. OVS) crossed with story animacy (Animate vs. Inanimate) and discourse context type (Critical vs. Control), this results in eight critical conditions: SVO sentences with inanimate objects, SVO sentences with animate objects, OVS sentences with inanimate objects, and OVS sentences with animate objects, either occurring in critical contexts or in control contexts. The experiment also contained filler stories where the critical scenes are described with SVO sentences in which the subject is lexical (e.g. the woman) and the object is pronominal (e.g., honom). This ensures that participants are exposed to SVO sentences with both pronominal (critical sentences) and lexical subjects (filler sentences), so that an initial lexical NP not always entails OVS word order. The comprehension questions following some of the stories were open-ended questions (e.g. "What does Eva work as?") that were answered by selecting one of two alternatives (e.g., "manager" or "teacher"). These were used to ensure that participants stayed attentive to the story content throughout the experiment. Two sets of 16 full stories were constructed. Four story versions, corresponding to each of the four conditions, were constructed for each of the stories, resulting in 16 (item) × 2 (set) × 4 (condition) = 128 stories in total. Two sets of filler stories were also created and included, resulting in a total of 160 stories.
A total of 47 native Swedish speakers (30 female) performed the experiment. Their age had a median of 29 years and ranged from 18 to 58 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent.
The data from Study 2 is yet to be analyzed.
4.4. Study 3: Comprehenders can predict unexpected grammatical function assignments during language comprehension from animacy
4.4.1. Introduction and background
Study 3 investigates to what extent both sentence-level and discourse-level cues to grammatical functions can be used to predict the grammatical function of an upcoming NP even before that NP is mentioned. This is done on the basis of the visual world paradigm (e.g., Altman & Kamide, 1999). In this type of experiment, participants' eye-moments to objects and events in visual scenes are tracked while they listen to spoken language. The participants' gaze is assumed to target what is currently being processed. Using this paradigm, previous studies have shown that comprehenders are able to anticipate upcoming information from the sentence context even before it is presented, when the sentence context is restrictive enough.
The aim of study 3 is thus to investigate if comprehenders are able to predict the subject referent of a spoken transitive sentence even before that referent is mentioned, when additional cues to grammatical functions on both the sentence- and discourse level are available.
4.4.2. Materials and methods
Study 3 is an visual word eye-tracking study. Participants listen to short stories while the humans and objects in the stories are visually presented on a computer screen and the gaze of the participants is tracked. Participants' fixations are assumed to reflect what they are currently processing in the stories. The critical dependent measure is thus the proportion of fixations that participants make towards the picture of the subject referent in the final scene of the stories, where the critical OVS sentences are presented. If participants are able to utilize the animacy of the object and the discourse coherence cue in order to predict the upcoming grammatical function during on-line comprehension, object animacy and discourse coherence should result in a higher proportion of fixations towards the subject referent even before the subject NP is mentioned in the story. In the same manner as in Study 2, participants are asked comprehension questions after some of the stories in order to ensure that they stay attentive.
The stories used in Study 3 are identical to those used in Study 2. They differ, however, with respect to the pictures. Since study 3 is a visual word eye-tracking study, picture presentation is a crucial part of the study. Only pictures of the participants and/or of the objects in the stories were presented. These pictures were resized for the visual word paradigm, and presented in the following way in each scene of the story. In the first scene, only the protagonist is presented at the center of the screen. In the second scene, either the antagonists (animate stories) or the inanimate objects (inanimate stories) are randomly assigned to the center-left and the center-right screen position. In the third scene, the protagonist and either the two antagonists (animate stories) or the two objects (inanimate stories) are randomly assigned to the top-centered, bottom-left, and bottom-right screen positions. A few stories included a second protagonist picture. In these, both protagonists were randomly assigned to the center-left or center-right screen position in the first scene. In the third and final event scenes, both protagonists were presented together with either the antagonists (animate stories) or the inanimate objects (inanimate stories). These four pictures were randomly assigned to the top-left, top-right, bottom-left and bottom-right screen positions.
In the first scene, the protagonist picture(s) are presented with a 500 ms picture preview time, that is, the picture is shown 500 ms before the sentence starts. In the second and third scenes, on the other hand, a picture preview time of 2000 ms is used. This should give participants enough time to familiarize themselves with the pictures before the sentence starts (Huettig & McQueen 2007; Huettig & Guerra 2019).
As in Study 2, Study 3 also contained filler stories where the critical scenes are described with SVO sentences with lexical subjects is and pronominal objects, in order to ensure that an initial lexical NP not always entails OVS word order. Comprehension questions following some of the stories were again used to ensure that participants stayed attentive to the story content throughout the experiment. As in Study 2, a total of 160 stories were used (16 (item) × 2 (set) × 4 (condition) = 128 critical stories; 16 (item) × 2 (set) = 32 filler stories).
A total of 48 native Swedish speakers (30 female) performed the experiment. Their age had a median of 30.5 years and ranged from 18 to 68 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent.
4.4.3. Results and discussion
Strikingly, it is only in the incoherent control contexts where Animacy significantly affects fixations. In these contexts, participants show more fixations towards the subject referent in inanimate events directly following the picture presentation. Most importantly, participants also show increased fixations towards the subject referent following the presentation of the verb. This indicates that they predict the upcoming subject referent at this time point. In the coherent contexts, on the other hand, fixations hardly differ as a function of animacy. This indicates that comprehenders do not benefit from animacy as a cue to grammatical functions in coherent discourse contexts where a particular grammatical function assignment can be predicted from the discourse context.
5. Summary
This project has provided a significant contribution to our understanding of spoken language processing of grammatical functions in Swedish transitive sentences, both in terms of decoding during comprehension, and with respect to encoding during production. More specifically, it has provided further insights into how different types of linguistic cues-both semantic as well as morphosyntactic-are utilized during on-line processing.
The project has provided new insights about the underlying cognitive processes that utilize such cues during on-line comprehension. In particular, it provides robust evidence for the notion that on-line grammatical function assignment during comprehension is expectation-based: the contribution of linguistic cues during incremental grammatical function assignment originate from implicit expectations that are based on the distribution of cues in previous language input.
The project has also provided new insights about how linguistic cues on the sentence level are utilized in natural discourse contexts where other cues to grammatical functions are available on the discourse level. During production, speakers balance their use of sentence-level cues to grammatical functions in a communicatively efficient manner, even though there are additional cues on the discourse-level available. In other words, the choices speakers make during grammatical encoding is influenced by a trade-off between the motivation to avoid production costs, and a motivation to be informative enough for the listener. Although previous studies have provided indirect evidence for this account (e.g., Hörberg, 2018; Kurumada & Jaeger 2015), this is the first study to provide more direct, experimental evidence, and to also take additional information in the discourse context into account. During comprehension, listeners can utilize sentence-level linguistic cues, such as animacy, in order to predict the grammatical function of an upcoming NP even before that NP is mentioned. These findings provide strong support for the notion that on-line grammatical function assignment is rapid and incremental, in the sense that listeners make tentative grammatical function assignments on basis of the information currently available in the input, even before the full sentence has been presented (e.g., Hörberg et al. 2013, Hörberg & Jaeger, 2021). However, these cues are less informative in discourse contexts which provide additional information about grammatical functions on the discourse-level (such as contrastive contexts).
Taken together, these findings illustrate an important discrepancy between production and comprehension in informative discourse contexts: even though sentence-level cues are uninformative about grammatical functions in discourse contexts from the perspective of comprehenders, speakers are still sensitive to their availability, and balance their utterances accordingly.
6. Project outreach and dissemination
This project has so far resulted in two publications in top-tier, international journals, four presentations at some of the most prestigious international psycholinguistics conferences and two national conference presentations, as well as one invited talk at the Goethe University in Frankfurt and one at the Centre for Linguistic Theory and Studies in Probability (CLASP), Gothenburg University. It has involved international collaborations with researchers from the University of Rochester, Rochester, USA, from the Technische Universität Kaiserslautern, Kaiserslautern, Germany, from the George Washington University, Washington, USA and from Goethe University in Frankfurt, Germany. Below is a list of publications, conference presentations and talks.
6.1. Journal articles
Hörberg, T. & Sjons, J. 2022. Speakers balance their use of cues to grammatical functions in informative discourse contexts. Language, Cognition and Neuroscience. https://doi.org/10.1080/23273798.2022.2102667.
Hörberg, T. & Jaeger, F. 2021. A rational model of incremental argument interpretation: the comprehension of Swedish transitive clauses. Frontiers in Psychology, 12:674202 (special issue on rational approaches in language sciences). https://doi.org/10.3389/fpsyg.2021.674202
6.2. Conference presentations
Hörberg, T. & Jaeger, F. 2021. Bayesian surprise predicts incremental processing of grammatical functions. Presentation at the 34th Annual CUNY Conference on Human Sentence Processing, on-line. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-194176
Hörberg, T. 2020. Expectation-based encoding of grammatical functions in spoken discourse contexts. Poster presentation at The 26th Architectures and Mechanisms for Language Processing Conference (AMLaP), on-line, University of Potsdam. Available at https://osf.io/dkt85/files/
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Presentation at Rational Approaches in Language Sciences (RAILS) 2019, Saarland University, Saarbrücken, Germany. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185242.
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Presentation at The Swedish Workshop on Data Science (SweDS) 2019, Royal Institute of Technology (KTH), Stockholm, Sweden. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185240.
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Poster presentation at Discourse Expectations: Theoretical, Experimental, and Computational perspectives (DETEC) 2019, Leibniz-Centre General Linguistics (ZAS), Berlin, Germany. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185238.
Hörberg, T. 2019. Expectation-based processing of grammatical functions in Swedish. Poster presentation at the Lund Symposium on Cognition, Communication and Learning, Skissernas Museum, Lund, Sweden. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-172933.
6.3. Colloquia and work-in-progress seminars
Hörberg, T. 2019. Expectation-based processing of grammatical functions in Swedish. Invited talk at the CLASP seminar, Centre for Linguistic Theory and Studies in Probability (CLASP), Dept. of Philosophy, Linguistics and Theory of Science, University of Gothenburg, Gothenburg.
Hörberg, T. 2021. Is the encoding of grammatical functions during spoken language production expectation-based? Invited talk at the Psycholinguistics Colloquium, Dept. Linguistics, of Goethe University Frankfurt, Frankfurt, Germany (on-line).
Hörberg, T. 2019. Language processing in natural discourse - exploring the interplay between language production and comprehension of grammatical functions in short story contexts. Dept. of Linguistics, SU.
References
Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs restricting the domain of subsequent reference. Cognition, (73), 247–264.
Bornkessel, I., and Schlesewsky, M. (2006). The extended argument dependency model: A neurocognitive approach to sentence comprehension across languages. Psychol. Rev. 113, 787–821. doi:10.1037/0033-295X.113.4.787.
Hale, J. (2001). “A Probabilistic Earley Parser as a Psycholinguistic Model,” in Proceedings of NAACL, 159–166.
Hartsuiker, R. J., & Westenberg, C. (2000). Word order priming in written and spoken sentence production. Cognition, 75(2), B27–B39. https://doi.org/10.1016/S0010-0277(99)00080-3
Hörberg, T. (2016). Probabilistic and Prominence-driven Incremental Argument Interpretation in Swedish [PhD thesis, Stockholm University]. https://su.diva-portal.org/smash/get/diva2:924838/FULLTEXT01.pdf
Hörberg, T. (2018). Functional motivations behind direct object fronting in written Swedish: A corpus-distributional account. Glossa: A Journal of General Linguistics, 3(1), 81. https://doi.org/10.5334/gjgl.502
Hörberg, T., Koptjevskaja-Tamm, M., and Kallioinen, P. (2013). The neurophysiological correlate to grammatical function reanalysis in Swedish. Lang. Cogn. Process. 28, 388–416. doi:10.1080/01690965.2011.651345.
Huettig, F., & Guerra, E. (2019). Effects of speech rate, preview time of visual context, and participant instructions reveal strong limits on prediction in language processing. Brain Research, 1706, 196–208. https://doi.org/10.1016/j.brainres.2018.11.013
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. https://doi.org/10.1016/j.jml.2007.02.001
Kurumada, C., & Jaeger, T. F. (2015). Communicative efficiency in language production: Optional case-marking in Japanese. Journal of Memory and Language 83, 152–178.
Levy, R. (2008). Expectation-based syntactic comprehension. Cognition 106, 1126–1177. doi:10.1016/j.cognition.2007.05.006.
Nivre, J., and Megyesi, B. (2007). “Bootstrapping a Swedish Treebank Using Cross-Corpus Harmonization and Annotation Projection,” in Proceedings of the 6th International Workshop on Treebanks and Linguistic Theories, eds K. de Smedt, J. Hajic and S. Kübler, 97–102.
Prat-Sala, M., & Branigan, H. P. (2000). Discourse Constraints on Syntactic Processing in Language Production: A Cross-Linguistic Study in English and Spanish,. Journal of Memory and Language, 42(2), 168–182. https://doi.org/10.1006/jmla.1999.2668
The project was initiated at the start of 2019 and was finished june 2022. The covid pandemic had a strong impact on data collection and thus affected the time line of the project, and the plans needed to change accordingly. Despite of this, the project has broaden our theoretical understanding of language processing and provided novel and groundbreaking results that have contributed to the international research front. The project has resulted in international collaborations and several international disseminations in high-profile journals and conferences.
This summary is structured as follows. In section 2, ("Research aim") a short introduction of the aim of the research project and it's research questions are presented. In Section 3 ("The impact of the covid pandemic"), the ramifications of the covid pandemic on the project is presented. In section 4 ("Individual studies"), an overview of the development, methodological setup and results (if available) of the individual studies are presented. Section 5 ("Summary") summarizes the findings of the studies taken together. In Section 5 ("Project outreach and dissemination"), the international disseminations and collaborations that have resulted from the project are summarized.
2. Research aim
The project investigates spoken language processing of transitive sentences in Swedish such as as (i) "Flickan retar pojken" (‘The girl teases the boy’). It investigates how the grammatical functions of subject, "Flickan" (‘the girl’), and direct object, "pojken" (‘the boy’), are linguistically encoded during production, on the one hand, and propositionally decoded during comprehension, on the other. Grammatical functions allow us to communicate how participants or items are related to events or states. Our understanding of how we process them is therefore crucial for our knowledge about how we understand and produce language more generally.
Much research has been done on how various information types (i.e., linguistic cues) affect production and comprehension of grammatical functions in isolated sentences. Less is known, however, about the underlying processes that utilize such linguistic cues to grammatical functions during on-line comprehension. Linguistic cues could either have a privileged role in language processing, or their effects could originate from implicit expectations based on their distributions in previous language input. It is also unclear how sentence-level linguistic cues to grammatical functions are utilized in natural discourse-pragmatic contexts, such as short stories, where other discourse-level cues are available. The aim of this project is therefore to investigate how the processing of grammatical functions in Swedish is influenced by the interplay of linguistic cues, and how this interplay is affected by the availability of a natural discourse context. The project is concerned with the following research questions:
• Is the processing of linguistic cues to grammatical function expectation-based, i.e., is it based on the distribution of cues in previous language input?
• Do speakers balance their use of sentence-level cues to grammatical functions in a communicatively efficient manner even when discourse level cues are available in the discourse contexts?
• Can processing difficulties due to cue conflicts during the comprehension of grammatical functions be avoided or mitigated if a natural discourse context is provided?
• Can comprehenders predict unexpected grammatical function assignments during language comprehension if a natural discourse context is provided?
The project consists of a expectation-based, Bayesian model of incremental grammatical function assignment (study 0), a sentence production experiment (study 1), an event-related-brain potentials (ERP) experiment (study 2), and a visual world eye-tracking study (study 3).
3. The impact of the covid pandemic
The work on the stimulus materials for studies 1-3 took longer than anticipated. The stimulus materials were finished at the beginning of the second year of the project, 2020 (it was very hard to anticipate how long this work would take). There was therefore no time to pilot the EEG and the eye-tracking study during 2019.
When the covid pandemic broke out in the beginning of 2020, data collection had to be postponed further as the phonetics lab was closed. A research assistant who would collect the data was hired at the beginning of 2020, but his contract was terminated and postponed until the start of the fall semester of 2020. Data collection for study 1 was therefore finished at the end of october 2020. Data collection for study 2 stopped in the end of january 2021, and data collection for study 3 was finalized in the beginning of july 2021.
Because of this delay, the project plans needed to be adapted. In april 2020, in the midst of the pandemic, work on a expectation-based, Bayesian computational model of incremental argument interpretation therefore begun. This work, henceforth referred to as "Study 0", was conduction in collaboration with professor Florian Jaeger at the department of Brain and Cognitive Sciences at Rochester University, NY.
At the time of writing, a journal article on Study 0 (i.e., the Bayesian computational model and an empirical validation of the model predictions based on previously collected data) has been published in the academic journal "Frontiers in Psychology". Study 1 has been finished and published in the academic journal "Language, Cognition and Neuroscience". Also data from study 3 has been analyzed and the analysis of data from study 2 has begun.
4. Individual studies
4.1. Study 0: an expectation-based model of incremental grammatical function assignment
4.1.1. Introduction and background
Several studies have shown that a multitude of linguistic cues are utilized during on-line language comprehension, such as during grammatical function assignment. This involves morphosyntactic cues (e.g., case marking, word order and agreement), lexical-semantic cues (e.g., animacy), and verb-semantic cues (e.g., volitionality). Less is known about the cognitive processes that utilize these cues. Linguistic cues could have a privileged role in language processing in the sense that they constitute a special class of information that is utilized by the sentence comprehension system (e.g., Bornkessel-Schlesewsky & Schlesewsky 2006). It could also be that there is nothing special about linguistic cues in this sense. Instead, their value as cues might stem from statistical properties in their distribution in previous language input. On this view, language processing is expectation-based: the language comprehension system utilizes statistical regularites in the input during on-line interpretation. Processing difficulties stem from violations of expections in the incoming language input.
In study 0, I present an incremental, Bayesian, expectation-based model of incremental grammatical function assignment. This model is trained on the distribution of linguistic cues and their interactions in transitive sentences, collected from a corpus of transitive sentences in written Swedish (see Hörberg, 2016; 2018). The model is then evaluated against empirical reading time (RT) data from a self-paced reading study that was collected during my PhD (Hörberg, 2016). This is done by comparing the expectation-based model against a linguistic model, in terms of how well these models are able to explain the RT data.
In the following, I provide an overview of the expectation-based model. I briefly describe the self-paced reading experiment and then finally present the results and the analyses, comparing the expectation-based model against the linguistic model.
4.1.2. An expectation-based model of incremental grammatical function assignment
Comprehenders incrementally update their implicit expectations about the underlying sentence interpretation as new input becomes available (Hale, 2001; Levy, 2008). In expectation-based models, sentence interpretation involves continuously shifting from a prior to a posterior probability distribution over possible interpretations, a process known as "Bayesian belief-updating". The processing cost associated with new input is in part determined by the amount of new information provided by the input—the degree of shift in expectations or beliefs about the interpretation (Levy, 2008). This shift can be quantified in terms of Bayesian surprise, which is equivalent to the Kullback-Leibler (KL) divergence of the posterior distribution with respect to the prior distribution.
The model is concerned with canonical Swedish transitive clauses with subject-object (SO) order, and object-initial sentences with object-verb-subject (OS) order (see Hörberg, 2018). It estimates the Bayesian surprise of encountering the first NP, the verb(s), and the second NP of such clauses, with respect to grammatical function assignment (SO or OS). The model thus captures the change in expectations about whether the first or the second NP is the subject based on the linguistic cues available at constituent Ci (e.g., the second NP, NP2) with respect to the cues available at the previous constituent Ci-1 (e.g., NP1 and the verb).
In order to calculate Bayesian surprise of a constituent, the relevant prior and posterior probability distributions need to be estimated. In the model, this is done on the basis of corpus data. The data consists of 16,552 transitive sentences from the Svensk Trädbank treebank (Nivre & Megyesi, 2007). These sentences are canonical transitive sentences with SVO order, object-initial transitive sentences with OVS order, and adverbial-initial sentences with VSO or VOS order and are annotated for morphosyntactic (e.g., case-marking, auxiliary verbs), syntactic (embedding, verb-initial vs. verb-medial word order), prominence (e.g., animacy, person, givenness, definiteness), and verb semantic cues (e.g., volitionality, sentience). Bayesian surprise is estimated at three sentence regions: NP1, the verb, and NP2, derived from the probability distribution of OS vs. SO order at four different points in the sentence: (i) at the clause onset prior to any sentence input, (ii) after NP1 has been processed, (iii) after NP1 and the verb has been processed, and (iv) after NP1, the verb, and NP2 has been processed. The Bayesian surprise at NP1 is the KL divergence between the distribution of OS vs. SO after NP1 has been processed (ii) and the distribution of OS vs. SO at the clause onset prior to NP1 (i), etc. These distributions of OS vs. OS order at (i)-(iv) were estimated by fitting four separate Bayesian mixed-effects logistic regressions (GLMMs). Each of these four GLMMs included all annotated cues available up to that point in the sentence.
4.1.3. Testing model predictions
The models' predictions were tested against reading times from a self-paced reading experiment conducted by Hörberg (2016). The stimulus materials consist of transitive sentences with 1st or 2nd person personal pronoun subjects that vary with respect to word order (SVO vs. OVS), object animacy (animate vs. inanimate), and verb class (volitional vs. experiences). The experiment also contained filler sentences included to ensure that subjects also occured as nouns. In the experiment, participants read sentences one word at a time, exposing the subsequent word themselves with a button press. Time intervals between button presses are assumed to reflect processing costs. These should correlate with the models' predicted Bayesian surprise. The effects of linguistic cues on RTs show a qualitativey similar pattern to their effect on Bayesian surprise. As confirmed by statistical analyses, Bayesian surprise predicts RTs, showing also a quantitative relationship between Bayesian surprise and RTs.
As a final step, we also compared the expectation-based models' ability to predict RTs to the predictive power of a “linguistic” model, predicting RTs directly from the sentence properties (i.e., word order, animacy, and verb class) directly. This analysis found that our expectation-based model performs equally well as the linguistic model. In other words, Bayesian surprise largely captures the same predictive information about RTs as the individual linguistic cues. This provides strong support for the expectation-based account since our estimate of Bayesian surprise by neccessity contains less information (in terms of fewer degrees of freedom) about RTs than the information that is provided by the linguistic cues directly. This finding provides strong support for the notion that processing costs associated with linguistic cues are mediated through Bayesian surprise, and that grammatical function assignment is expectation-based.
4.2. Study 1: speakers balance cues to grammatical functions in informative discourse contexts
4.2.1. Introduction and background
Study 0 shows that linguistic cues are utilized in a probabilistic manner during on-line comprehension. Comprehenders' processing costs are predicted by the extent to which linguistic cues violate expectations regarding grammatical function assignment. Study 1 investigates whether speakers balance their use of linguistic cues to grammatical functions in a communicatively efficient manner. In other words, study 1 investigates whether speakers avoid redundant cues in order to reduce production costs, but only to the extent that the message provides enough cues for the listener to be able to determine grammatical functions. Although some studies have provided evidence for the notion that grammatical encoding is influenced by communicative efficiency (Kurumada & Jaeger 2015; Hörberg, 2018), this evidence has primarily been indirect (i.e., based on corpus- or sentence recall data). Further, no study has investigated whether speakers balance their use of cues in a communicatively efficient manner in natural discourse contexts, where other discourse-level cues are available. It could be that discourse-level cues are sufficient for interpretation, rendering the need for additional cues redundant. Study 1 thus asks whether speakers balance their use of cues to grammatical functions in order to avoid both redundancies and ambiguities when producing sentences in story contexts. As shown in study 0, animacy (i.e., whether an NP refers to a living thing or not) functions as an important cue to grammatical functions (i.e., subjects are most frequently animate). Study 1 thus investigates whether speakers balance their use of morphosyntactic cues (e.g., word order and case marking) with respect to animacy.
4.2.2. Materials and methods
Study 1 is a picture description task where participants describe transitive events occuring in short stories that are accompanied with cartooned images (similar to Prat-Sala & Branigan, 2000). The stories consist of scenes, presented one scene at a time, that are divided into three parts. Part 1 consists of one scene which introduces the protagonist of the story ("This is Eva") and presents the situational context ("She is the manager of a staffing agency"). Part 2 and 3 contain two separate scenes. In the first scene, either two animate antagonists ("Before lunch, Eva interviewed a man and a woman") or two inanimate objects ("Before lunch, Eva received several invoices and an interim report") are presented. In the second scene, the protagonist interacts with the antagonists or the objects. Importantly, these events involve an opposition in the way the protagonist interacts with the antagonists or objects and thus license the use of a contrastive construction. Here, the opposition in the involvement of the object referents is signaled by fronting of the direct object, resulting in OVS word order (Hörberg, 2018). These events can thus be described with a transitive sentence with either SO ("She liked the man, but did not like the woman" / "She paid the invoices, but forgot the report") or OS word order ("The man she liked, but the woman she did not like" / "The invoices she paid, but the report she forgot"). Part 2 and 3 differ with respect to the animacy of the object referents, and are therefore referred to as the "animate" and the "inanimate" event. Participants are exposed both to the animate and the inanimate event, starting with either the former or the latter. In the first event, participants listen to a SVO sentence or an OVS sentence that describes the event and that functions as a prime sentence. In the second event, participants describe the event depicted in the image themselves. With event animacy (animate vs. inanimate) crossed with prime type (SVO vs. OVS), there are four picture description conditions: Animate / SVO priming, Animate / OVS priming, Inanimate / SVO priming, and Inanimate / OVS priming.
The stories were recorded in the anechoic chamber at the Department of Linguistics at Stockholm University. They were spoken by a 45-year-old female professional speech coach from the Stockholm area. The pictures used in the stories were drawn by a professional artist. These consisted of drawings of 59 individuals who were the protagonists and/or antagonists in the stories, 33 inanimate objects, 32 inanimate events and 32 animate events. Two sets of 16 full stories with an animate and an inanimate event were constructed. Four story versions, corresponding to each of the four conditions, were constructed for each of the stories, resulting in 16 (item) × 2 (set) × 4 (condition) = 128 stories in total.
A total of 65 native Swedish speakers (39 female) performed the experiment. Their age had a median of 27 years and ranged from 18 to 68 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent. Responses that were not in accordance with the instructions was excluded from further analysis.
4.2.3. Predictions
As explained above, the events-to-be-described differ with respect to the animacy of the object referent. If speakers are sensitive to this animacy manipulation in their encoding, events with inanimate object referents should more frequently be described with OVS word order. Further, if speakers are sensitive to the availability of cues to grammatical functions more generally, OVS word order should also be used more frequently with other morphosyntactic cues, and such cues should further be used more frequently in sentences with two animate NPs.
4.2.4. Results and discussion
As confirmed by statistical analyses (not reported), OVS word order is produced more frequently in descriptions of inanimate events. The statistical analyses also show that OVS word order is used more frequently when the direct object is inanimate (independent of event type), when the subject NP is case marked (i.e., consists of personal pronoun), and when other syntactic cues to grammatical functions are available (e.g., such as an adverbial). Additional analyses also found that subject case marking is used more frequently in sentences with animate direct objects, independent of word order. There is, however, no effect of prime type. OVS word order is used more or less equally frequently in the descriptions of both SVO and OVS prime stories.
Taken together, these findings show that speakers balance their use of both semantic and morphosyntactic cues to grammatical functions when producing sentences in a discourse context that is informative about grammatical functions in itself. OVS word order is preferred when other cues to grammatical functions are available. In other words, speakers prefer to use alternative word order patterns such as OVS?which are potentially ambiguous with respect to grammatical functions?only when other cues are available, even though the discourse context provides enough information for the listener. Speakers also seem to balance their use of other cues such as case marking and animacy.
4.3. Study 2: Are processing difficulties during grammatical function assignment mitigated by an informative natural discourse context?
4.3.1. Introduction and background
Previous studies (e.g. Hörberg et al. 2013) and study 0 show that processing difficulties during the comprension of transitive sentences stem from the extent to which linguistic cues violate expectations regarding grammatical function assignment. However, these studies have only investigated processing of sentences presented in isolation, independent of the discourse context. The discourse context can provide additional cues to grammatical functions, rendering the need for cues on the sentence level redundant. Processing might therefore not be affected by the availability of sentence-level cues in coherent and informative discourse contexts.
The aim of study 2 was thus to investigate whether processing difficulties during grammtical function assignment in spoken language comprehension that result from sentence-level cue conflicts are affected by the information available in the discourse context.
4.3.2. Materials and methods
The study is an event-related brain potentials (ERP) study, where participants listen to short stories. In order to ensure that participants stay attentive to the stories, they are asked comprehension questions after some of the stories. The critical dependent measure is the ERP response to the subject pronoun in OVS sentences. In such sentences, the subject pronoun disambiguate the sentence towards OVS, engendering an ERP response (see Hörberg et al. 2013). This response should be affected by the availability of other sentence-level cues (i.e., animacy) and / or discourse-level cues (i.e., discourse coherence, see below).
The stories are highly similar to those of study 1. The main difference is that they only contain one event which either is animate (i.e., "animate stories") or inanimate (i.e.,"inanimate stories"). This event is described with either an SVO or OVS sentence. In the first scene, the protagonist and the story context is presented ("This is Eva. She works as the manager of a staffing agency"). The second scene introduces either two animate antagonists ("Before lunch, she interviewed a man and a woman") or two inanimate objects ("Before lunch, she received several invoices and an interim report"). In the third, critical scene, a contrastive event is presented in which the protagonist interacts either with the animate antagonists (animate stories) or the inanimate objects (inanimate stories). These events are described with a transitive sentence with either SO ("She liked the man, but not the woman" / "She paid the invoices, but forgot the report") or OS ("The man she liked, but the woman she did not like" / "The invoices she paid, but the report she forgot") word order.
In order to control for the influence of discourse context coherence on the processing of the critical sentences, the experiment also contains control stories that are matched to the critical stories. These only differ from the critical stories in the second scene in which the antagonists or objects are introduced. In the control stories, the two antagonists or objects are never introduced, and instead a more general statement is used ("Before lunch, she interviewed some applicants" / "Before lunch, she had a lot to do"). With this context, the upcoming contrastive event of the final scene becomes infelicitous, and should be unexpected. The processing of the potentially ambiguous OVS sentences should therefore not be facilitated by the context in the control stories.
With the critical sentence type (SVO vs. OVS) crossed with story animacy (Animate vs. Inanimate) and discourse context type (Critical vs. Control), this results in eight critical conditions: SVO sentences with inanimate objects, SVO sentences with animate objects, OVS sentences with inanimate objects, and OVS sentences with animate objects, either occurring in critical contexts or in control contexts. The experiment also contained filler stories where the critical scenes are described with SVO sentences in which the subject is lexical (e.g. the woman) and the object is pronominal (e.g., honom). This ensures that participants are exposed to SVO sentences with both pronominal (critical sentences) and lexical subjects (filler sentences), so that an initial lexical NP not always entails OVS word order. The comprehension questions following some of the stories were open-ended questions (e.g. "What does Eva work as?") that were answered by selecting one of two alternatives (e.g., "manager" or "teacher"). These were used to ensure that participants stayed attentive to the story content throughout the experiment. Two sets of 16 full stories were constructed. Four story versions, corresponding to each of the four conditions, were constructed for each of the stories, resulting in 16 (item) × 2 (set) × 4 (condition) = 128 stories in total. Two sets of filler stories were also created and included, resulting in a total of 160 stories.
A total of 47 native Swedish speakers (30 female) performed the experiment. Their age had a median of 29 years and ranged from 18 to 58 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent.
The data from Study 2 is yet to be analyzed.
4.4. Study 3: Comprehenders can predict unexpected grammatical function assignments during language comprehension from animacy
4.4.1. Introduction and background
Study 3 investigates to what extent both sentence-level and discourse-level cues to grammatical functions can be used to predict the grammatical function of an upcoming NP even before that NP is mentioned. This is done on the basis of the visual world paradigm (e.g., Altman & Kamide, 1999). In this type of experiment, participants' eye-moments to objects and events in visual scenes are tracked while they listen to spoken language. The participants' gaze is assumed to target what is currently being processed. Using this paradigm, previous studies have shown that comprehenders are able to anticipate upcoming information from the sentence context even before it is presented, when the sentence context is restrictive enough.
The aim of study 3 is thus to investigate if comprehenders are able to predict the subject referent of a spoken transitive sentence even before that referent is mentioned, when additional cues to grammatical functions on both the sentence- and discourse level are available.
4.4.2. Materials and methods
Study 3 is an visual word eye-tracking study. Participants listen to short stories while the humans and objects in the stories are visually presented on a computer screen and the gaze of the participants is tracked. Participants' fixations are assumed to reflect what they are currently processing in the stories. The critical dependent measure is thus the proportion of fixations that participants make towards the picture of the subject referent in the final scene of the stories, where the critical OVS sentences are presented. If participants are able to utilize the animacy of the object and the discourse coherence cue in order to predict the upcoming grammatical function during on-line comprehension, object animacy and discourse coherence should result in a higher proportion of fixations towards the subject referent even before the subject NP is mentioned in the story. In the same manner as in Study 2, participants are asked comprehension questions after some of the stories in order to ensure that they stay attentive.
The stories used in Study 3 are identical to those used in Study 2. They differ, however, with respect to the pictures. Since study 3 is a visual word eye-tracking study, picture presentation is a crucial part of the study. Only pictures of the participants and/or of the objects in the stories were presented. These pictures were resized for the visual word paradigm, and presented in the following way in each scene of the story. In the first scene, only the protagonist is presented at the center of the screen. In the second scene, either the antagonists (animate stories) or the inanimate objects (inanimate stories) are randomly assigned to the center-left and the center-right screen position. In the third scene, the protagonist and either the two antagonists (animate stories) or the two objects (inanimate stories) are randomly assigned to the top-centered, bottom-left, and bottom-right screen positions. A few stories included a second protagonist picture. In these, both protagonists were randomly assigned to the center-left or center-right screen position in the first scene. In the third and final event scenes, both protagonists were presented together with either the antagonists (animate stories) or the inanimate objects (inanimate stories). These four pictures were randomly assigned to the top-left, top-right, bottom-left and bottom-right screen positions.
In the first scene, the protagonist picture(s) are presented with a 500 ms picture preview time, that is, the picture is shown 500 ms before the sentence starts. In the second and third scenes, on the other hand, a picture preview time of 2000 ms is used. This should give participants enough time to familiarize themselves with the pictures before the sentence starts (Huettig & McQueen 2007; Huettig & Guerra 2019).
As in Study 2, Study 3 also contained filler stories where the critical scenes are described with SVO sentences with lexical subjects is and pronominal objects, in order to ensure that an initial lexical NP not always entails OVS word order. Comprehension questions following some of the stories were again used to ensure that participants stayed attentive to the story content throughout the experiment. As in Study 2, a total of 160 stories were used (16 (item) × 2 (set) × 4 (condition) = 128 critical stories; 16 (item) × 2 (set) = 32 filler stories).
A total of 48 native Swedish speakers (30 female) performed the experiment. Their age had a median of 30.5 years and ranged from 18 to 68 years. They were informed about the experimental procedure and the precautions taken in order to minimise the risk of spread of the Covid-19 virus. They were further told that they could stop at any time without giving a reason. They provided written informed consent.
4.4.3. Results and discussion
Strikingly, it is only in the incoherent control contexts where Animacy significantly affects fixations. In these contexts, participants show more fixations towards the subject referent in inanimate events directly following the picture presentation. Most importantly, participants also show increased fixations towards the subject referent following the presentation of the verb. This indicates that they predict the upcoming subject referent at this time point. In the coherent contexts, on the other hand, fixations hardly differ as a function of animacy. This indicates that comprehenders do not benefit from animacy as a cue to grammatical functions in coherent discourse contexts where a particular grammatical function assignment can be predicted from the discourse context.
5. Summary
This project has provided a significant contribution to our understanding of spoken language processing of grammatical functions in Swedish transitive sentences, both in terms of decoding during comprehension, and with respect to encoding during production. More specifically, it has provided further insights into how different types of linguistic cues-both semantic as well as morphosyntactic-are utilized during on-line processing.
The project has provided new insights about the underlying cognitive processes that utilize such cues during on-line comprehension. In particular, it provides robust evidence for the notion that on-line grammatical function assignment during comprehension is expectation-based: the contribution of linguistic cues during incremental grammatical function assignment originate from implicit expectations that are based on the distribution of cues in previous language input.
The project has also provided new insights about how linguistic cues on the sentence level are utilized in natural discourse contexts where other cues to grammatical functions are available on the discourse level. During production, speakers balance their use of sentence-level cues to grammatical functions in a communicatively efficient manner, even though there are additional cues on the discourse-level available. In other words, the choices speakers make during grammatical encoding is influenced by a trade-off between the motivation to avoid production costs, and a motivation to be informative enough for the listener. Although previous studies have provided indirect evidence for this account (e.g., Hörberg, 2018; Kurumada & Jaeger 2015), this is the first study to provide more direct, experimental evidence, and to also take additional information in the discourse context into account. During comprehension, listeners can utilize sentence-level linguistic cues, such as animacy, in order to predict the grammatical function of an upcoming NP even before that NP is mentioned. These findings provide strong support for the notion that on-line grammatical function assignment is rapid and incremental, in the sense that listeners make tentative grammatical function assignments on basis of the information currently available in the input, even before the full sentence has been presented (e.g., Hörberg et al. 2013, Hörberg & Jaeger, 2021). However, these cues are less informative in discourse contexts which provide additional information about grammatical functions on the discourse-level (such as contrastive contexts).
Taken together, these findings illustrate an important discrepancy between production and comprehension in informative discourse contexts: even though sentence-level cues are uninformative about grammatical functions in discourse contexts from the perspective of comprehenders, speakers are still sensitive to their availability, and balance their utterances accordingly.
6. Project outreach and dissemination
This project has so far resulted in two publications in top-tier, international journals, four presentations at some of the most prestigious international psycholinguistics conferences and two national conference presentations, as well as one invited talk at the Goethe University in Frankfurt and one at the Centre for Linguistic Theory and Studies in Probability (CLASP), Gothenburg University. It has involved international collaborations with researchers from the University of Rochester, Rochester, USA, from the Technische Universität Kaiserslautern, Kaiserslautern, Germany, from the George Washington University, Washington, USA and from Goethe University in Frankfurt, Germany. Below is a list of publications, conference presentations and talks.
6.1. Journal articles
Hörberg, T. & Sjons, J. 2022. Speakers balance their use of cues to grammatical functions in informative discourse contexts. Language, Cognition and Neuroscience. https://doi.org/10.1080/23273798.2022.2102667.
Hörberg, T. & Jaeger, F. 2021. A rational model of incremental argument interpretation: the comprehension of Swedish transitive clauses. Frontiers in Psychology, 12:674202 (special issue on rational approaches in language sciences). https://doi.org/10.3389/fpsyg.2021.674202
6.2. Conference presentations
Hörberg, T. & Jaeger, F. 2021. Bayesian surprise predicts incremental processing of grammatical functions. Presentation at the 34th Annual CUNY Conference on Human Sentence Processing, on-line. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-194176
Hörberg, T. 2020. Expectation-based encoding of grammatical functions in spoken discourse contexts. Poster presentation at The 26th Architectures and Mechanisms for Language Processing Conference (AMLaP), on-line, University of Potsdam. Available at https://osf.io/dkt85/files/
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Presentation at Rational Approaches in Language Sciences (RAILS) 2019, Saarland University, Saarbrücken, Germany. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185242.
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Presentation at The Swedish Workshop on Data Science (SweDS) 2019, Royal Institute of Technology (KTH), Stockholm, Sweden. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185240.
Hörberg, T. 2019. The Processing of Grammatical Functions in Swedish is Expectation-based. Poster presentation at Discourse Expectations: Theoretical, Experimental, and Computational perspectives (DETEC) 2019, Leibniz-Centre General Linguistics (ZAS), Berlin, Germany. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-185238.
Hörberg, T. 2019. Expectation-based processing of grammatical functions in Swedish. Poster presentation at the Lund Symposium on Cognition, Communication and Learning, Skissernas Museum, Lund, Sweden. Available at http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-172933.
6.3. Colloquia and work-in-progress seminars
Hörberg, T. 2019. Expectation-based processing of grammatical functions in Swedish. Invited talk at the CLASP seminar, Centre for Linguistic Theory and Studies in Probability (CLASP), Dept. of Philosophy, Linguistics and Theory of Science, University of Gothenburg, Gothenburg.
Hörberg, T. 2021. Is the encoding of grammatical functions during spoken language production expectation-based? Invited talk at the Psycholinguistics Colloquium, Dept. Linguistics, of Goethe University Frankfurt, Frankfurt, Germany (on-line).
Hörberg, T. 2019. Language processing in natural discourse - exploring the interplay between language production and comprehension of grammatical functions in short story contexts. Dept. of Linguistics, SU.
References
Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs restricting the domain of subsequent reference. Cognition, (73), 247–264.
Bornkessel, I., and Schlesewsky, M. (2006). The extended argument dependency model: A neurocognitive approach to sentence comprehension across languages. Psychol. Rev. 113, 787–821. doi:10.1037/0033-295X.113.4.787.
Hale, J. (2001). “A Probabilistic Earley Parser as a Psycholinguistic Model,” in Proceedings of NAACL, 159–166.
Hartsuiker, R. J., & Westenberg, C. (2000). Word order priming in written and spoken sentence production. Cognition, 75(2), B27–B39. https://doi.org/10.1016/S0010-0277(99)00080-3
Hörberg, T. (2016). Probabilistic and Prominence-driven Incremental Argument Interpretation in Swedish [PhD thesis, Stockholm University]. https://su.diva-portal.org/smash/get/diva2:924838/FULLTEXT01.pdf
Hörberg, T. (2018). Functional motivations behind direct object fronting in written Swedish: A corpus-distributional account. Glossa: A Journal of General Linguistics, 3(1), 81. https://doi.org/10.5334/gjgl.502
Hörberg, T., Koptjevskaja-Tamm, M., and Kallioinen, P. (2013). The neurophysiological correlate to grammatical function reanalysis in Swedish. Lang. Cogn. Process. 28, 388–416. doi:10.1080/01690965.2011.651345.
Huettig, F., & Guerra, E. (2019). Effects of speech rate, preview time of visual context, and participant instructions reveal strong limits on prediction in language processing. Brain Research, 1706, 196–208. https://doi.org/10.1016/j.brainres.2018.11.013
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. https://doi.org/10.1016/j.jml.2007.02.001
Kurumada, C., & Jaeger, T. F. (2015). Communicative efficiency in language production: Optional case-marking in Japanese. Journal of Memory and Language 83, 152–178.
Levy, R. (2008). Expectation-based syntactic comprehension. Cognition 106, 1126–1177. doi:10.1016/j.cognition.2007.05.006.
Nivre, J., and Megyesi, B. (2007). “Bootstrapping a Swedish Treebank Using Cross-Corpus Harmonization and Annotation Projection,” in Proceedings of the 6th International Workshop on Treebanks and Linguistic Theories, eds K. de Smedt, J. Hajic and S. Kübler, 97–102.
Prat-Sala, M., & Branigan, H. P. (2000). Discourse Constraints on Syntactic Processing in Language Production: A Cross-Linguistic Study in English and Spanish,. Journal of Memory and Language, 42(2), 168–182. https://doi.org/10.1006/jmla.1999.2668