publications

2026

1. CMCL 2026

   Predicting Sentence Acceptability Judgments in Multimodal Contexts

   Hyewon Jang, Nikolai Ilinykh, Sharid Loaiciga, Jey Han Lau, Shalom Lappin

   In Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics (CMCL), 2026

   Abs

   Previous work has examined the capacity of deep neural networks (DNNs), particularly transformers, to predict human sentence acceptability judgments, both independently of context, and in document contexts. We consider the effect of prior exposure to visual images (i.e., visual context) on these judgments for humans and large language models (LLMs). Our results suggest that, in contrast to textual context, visual images appear to have little if any impact on human acceptability ratings. However, LLMs display the compression effect seen in previous work on human judgments in document contexts. Different sorts of LLMs are able to predict human acceptability judgments to a high degree of accuracy, but in general, their performance is slightly better when visual contexts are removed. Moreover, the distribution of LLM judgments varies among models, with Qwen resembling human patterns, and others diverging from them. LLM-generated predictions on sentence acceptability are highly correlated with their normalised log probabilities in general. However, the correlations decrease when visual contexts are present, suggesting that a higher gap exists between the internal representations of LLMs and their generated predictions in the presence of visual contexts. Our experimental work suggests interesting points of similarity and divergence between human and LLM processing of sentences in multimodal contexts.

2. IWSDS 2026

   When social robots see our sketches: evaluating human perception of a robot and a VLM model performance in a drawing task

   Viktoria Paraskevi Daniilidou, Nikolai Ilinykh, Vladislav Maraev

   In Proceedings of the 16th International Workshop on Spoken Dialogue System Technology, Feb 2026

   Abs Bib

   We introduce a multimodal framework for interactive drawing in a robot-assisted second language learning scenario. In this scenario, humans are asked to draw objects and spatial relations between them, while a social robot that uses a vision-language model (VLM) to analyse whether the drawings are correct. The correctness decision that is passed to the human is coming from a Wizard-of-Oz (WoZ) setup. Therefore, we use it to indirectly evaluate the quality of VLM predictions. We show that the task is very challenging for a VLM and approaching evaluation of VLM performance is important: focusing on the correctness of prediction of certain features (objects, relations) provides a different evaluation picture from when the model is evaluated on prediction of the content of the image as a whole. We also examine how the appearance of the social agent and the type of feedback influence perception of the agent by the participants through a questionnaire. The comparison of verbal feedback, generated by the large language models, against simple pattern-based feedback did not show any significant effects whereas the robot’s appearance change indicated significant difference in user ratings concerning naturalness of the agent and its social presence.

   @inproceedings{daniilidou-etal-2026-social,
     title = {When social robots see our sketches: evaluating human perception of a robot and a VLM model performance in a drawing task},
     author = {Daniilidou, Viktoria Paraskevi and Ilinykh, Nikolai and Maraev, Vladislav},
     editor = {Riccardi, Giuseppe and Mousavi, Seyed Mahed and Torres, Maria Ines and Yoshino, Koichiro and Callejas, Zoraida and Chowdhury, Shammur Absar and Chen, Yun-Nung and Bechet, Frederic and Gustafson, Joakim and Damnati, G{\'e}raldine and Papangelis, Alex and D{'}Haro, Luis Fernando and Mendon{\c{c}}a, John and Bernardi, Raffaella and Hakkani-Tur, Dilek and Di Fabbrizio, Giuseppe {''}Pino{''} and Kawahara, Tatsuya and Alam, Firoj and Tur, Gokhan and Johnston, Michael},
     booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
     month = feb,
     year = {2026},
     address = {Trento, Italy},
     publisher = {Association for Computational Linguistics},
     pages = {239--252},
     url = {https://aclanthology.org/2026.iwsds-1.25/},
   }

2025

1. INLG 2025

   Surprisal reveals diversity gaps in image captioning and different scorers change the story

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 18th International Natural Language Generation Conference, Feb 2025

   Abs

   We quantify linguistic diversity in image captioning with surprisal variance – the spread of token-level negative log-probabilities within a caption set. On the MSCOCO test set, we compare five state-of-the-art vision-and-language LLMs, decoded with greedy and nucleus sampling, to human captions. Measured with a caption-trained n-gram LM, humans display roughly twice the surprisal variance of models, but rescoring the same captions with a general-language model reverses the pattern. Our analysis introduces the surprisal-based diversity metric for image captioning. We show that relying on a single scorer can completely invert conclusions, thus, robust diversity evaluation must report surprisal under several scorers.

2. GEM 2025

   Coreference as an indicator of context scope in multimodal narrative

   Nikolai Ilinykh, Shalom Lappin, Asad B. Sayeed, Sharid Loaiciga

   In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), Feb 2025

   Abs

   We demonstrate that large multimodal language models differ substantially from humans in the distribution of coreferential expressions in a visual storytelling task. We introduce a number of metrics to quantify the characteristics of coreferential patterns in both human- and machine-written texts. Humans distribute coreferential expressions in a way that maintains consistency across texts and images, interleaving references to different entities in a highly varied way. Machines are less able to track mixed references, despite achieving perceived improvements in generation quality. Materials, metrics, and code for our study are available at https://github.com/GU-CLASP/coreference-context-scope.

3. SemDial 2025

   Surprised to Kill: Quantifying LLM Uncertainty in Morally-Charged Triadic Dialogues

   Vanessa Vanzan, Nikolai Ilinykh, Erik Lagerstedt, Amy Han Qiu, Vladislav Maraev, Christine Howes

   In Proceedings of the 29th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts, Sep 2025

   Abs Bib

   Multi-party dialogues on ethically and socially challenging (morally charged) topics pose a challenge for large language models (LLMs) trained on massive text corpora. Nevertheless, LLMs can illuminate features of interaction in such dialogues and serve as evaluation proxies. We propose using LLM surprisal as an indicator of points in dialogue which address or relate to the discussion of social norms on a corpus of triadic text conversations from the Balloon Task, in which three participants collaboratively resolve a moral dilemma. We hypothesise that (1) turns featuring indirect reference and implicit moral justification will exhibit higher surprisal than turns with direct reference or explicit justification, and (2) including dialogue-act or reference-type annotations in the prompt will reduce model uncertainty with the help of retrieval-augmented generation. By presenting our planned experiments, we aim to inform the design of socially aware dialogue systems able to reliably interpret nuanced ethical discourse.

   @inproceedings{vanzan-etal-2025-surprised,
     title = {Surprised to Kill: Quantifying LLM Uncertainty in Morally-Charged Triadic Dialogues},
     author = {Vanzan, Vanessa and Ilinykh, Nikolai and Lagerstedt, Erik and Qiu, Amy Han and Maraev, Vladislav and Howes, Christine},
     booktitle = {Proceedings of the 29th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts},
     month = sep,
     year = {2025},
     address = {Bielefeld, Germany},
     publisher = {SEMDIAL},
     pages = {219--221},
     url = {http://semdial.org/anthology/Z25-Vanzan_semdial_4425.pdf},
   }

4. RESOURCEFUL 2025

   “I Need More Context and an English Translation”: Analysing How LLMs Identify Personal Information in Komi, Polish, and English

   Nikolai Ilinykh Maria Irena Szawerna

   In Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025), Mar 2025

   Abs

   Automatic identification of personal information (PI) is particularly difficult for languages with limited linguistic resources. Recently, large language models (LLMs) have been applied to various tasks involving low-resourced languages, but their capability to process PI in such contexts remains under-explored. In this paper we provide a qualitative analysis of the outputs from three LLMs prompted to identify PI in texts written in Komi (Permyak and Zyrian), Polish, and English. Our analysis highlights challenges in using pre-trained LLMs for PI identification in both low- and medium-resourced languages. It also motivates the need to develop LLMs that understand the differences in how PI is expressed across languages with varying levels of availability of linguistic resources.

2024

1. Computational Models of Language and Vision: Studies of Neural Models as Learners of Multi-modal Knowledge

   Nikolai Ilinykh

   University of Gothenburg, Jun 2024

   Abs

   This thesis develops and evaluates computational models that generate natural language descriptions of visual content. We build and examine models of language and vision to gain a deeper understanding of how they reflect the relationship between the two modalities. This understanding is crucial for performing computational tasks. The first part of the thesis introduces three studies that inspect the role of self-attention in three different self-attention blocks of the object relation transformer model. We examine attention heatmaps to understand how the model connects different words, objects, and relations within the tasks of image captioning and image paragraph generation. We connect our interpretation of what the model learns in self-attention weights with insights from theories about human cognition, visual perception, and spatial language. The three studies in the second part of the thesis investigate how representations of images and texts can be applied and learned in task-specific models for image paragraph generation, embodied question answering, and variation in human object naming. The last two studies in the third part examine properties of human-generated texts that multi-modal models are expected to acquire in image paragraph generation as well as perceptual category description and interpretation tasks. We analyse discourse structure in image paragraphs produced with different decoding methods. We also inspect whether models of perceptual categories can abstract from visual representations and use this knowledge to generate descriptions that exhibit discriminativity levels important for the task. We show how automatic measures for evaluating text generation behave in a comparison of model-generated and human-generated image descriptions. This thesis presents several contributions. We illustrate that, under specific modelling conditions, self-attention can capture information about the relationship between objects and words. Our results emphasise that the specifics of the task determine the manner and context in which different modalities are processed, as well as the degree to which each modality contributes to the task. We demonstrate that while favoured by automatic evaluation metrics in different tasks, machine-generated image descriptions lack the discourse complexity and discriminative power that are often important for generating better, human-like image descriptions.

2. SLTC 2024

   How do images help coreference? A case study on the multimodal Tell-me-more dataset

   Nikolai Ilinykh Sharid Loáiciga

   In Swedish Language Technology Conference (SLTC) 2024 - Extended Abstracts, Nov 2024

   Bib HTML

   @inproceedings{ilinykh-loaiciga-2024-coreference,
     title = {How do images help coreference? A case study on the multimodal Tell-me-more dataset},
     author = {Ilinykh, Nikolai and Lo{\'a}iciga, Sharid},
     booktitle = {Swedish Language Technology Conference (SLTC) 2024 - Extended Abstracts},
     month = nov,
     year = {2024},
     address = {Linköping, Sweden},
     url = {https://sltc2024.github.io/abstracts/ilinykh.pdf},
   }

3. SemDial 2024

   Dialogue with LLaVA: does it “understand” the pragmatics of the MeetUp task?

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 28th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts, Sep 2024

   Bib HTML

   @inproceedings{ilinykh-dobnik-2024-dialogue,
     title = {Dialogue with LLaVA: does it ``understand'' the pragmatics of the MeetUp task?},
     author = {Ilinykh, Nikolai and Dobnik, Simon},
     booktitle = {Proceedings of the 28th Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts},
     month = sep,
     year = {2024},
     address = {Trento, Italy},
     publisher = {SEMDIAL},
     url = {http://semdial.org/anthology/Z24-Ilinykh_semdial_0046.pdf},
   }

4. CORE 2024

   What to refer to and when? Reference and re-reference in two language-and-vision tasks

   Simon Dobnik, Nikolai. Ilinykh, Aram Karimi

   In CORE Project Workshop: Unpacking Efficient Communication: The Roles of Cognitive Bias and Extralinguistic Context in Referring Expression Choice - Poster Abstracts, Apr 2024

   Bib HTML

   @inproceedings{dobnik-etal-2024-what-to-refer,
     title = {What to refer to and when? Reference and re-reference in two language-and-vision tasks},
     author = {Dobnik, Simon and Ilinykh, Nikolai. and Karimi, Aram},
     booktitle = {CORE Project Workshop: Unpacking Efficient Communication: The Roles of Cognitive Bias and Extralinguistic Context in Referring Expression Choice - Poster Abstracts},
     month = apr,
     year = {2024},
     address = {Barcelona, Spain},
     url = {http://semdial.org/anthology/Z24-Ilinykh_semdial_0046.pdf},
   }

2023

1. EMNLP 2023

   Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions

   Bill Noble Nikolai Ilinykh

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Dec 2023

   Abs DOI

   Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level. Moreover, such descriptions can be used by other speakers to learn provisional representations of those concepts. Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field. The problem is also highly relevant to the field of representation learning in multi-modal NLP. In this paper, we introduce a framework for testing category-level perceptual grounding in multi-modal language models. In particular, we train separate neural networks to **generate** and **interpret** descriptions of visual categories. We measure the *communicative success* of the two models with the zero-shot classification performance of the interpretation model, which we argue is an indicator of perceptual grounding. Using this framework, we compare the performance of *prototype*- and *exemplar*-based representations. Finally, we show that communicative success exposes performance issues in the generation model, not captured by traditional intrinsic NLG evaluation metrics, and argue that these issues stem from a failure to properly ground language in vision at the category level.

2. INLG 2023

   The VDG Challenge: Response Generation and Evaluation in Collaborative Visual Dialogue

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 16th International Natural Language Generation Conference: Generation Challenges, Sep 2023

   Abs

   We propose the VDG Challenge: a shared task that addresses and benchmarks the task of utterance generation in collaborative visual dialogue. The task features two challenging datasets, an evaluation protocol and a tentative schedule. Our shared task will allow researchers to unravel problems of modelling multi-modal interaction and fit of the existing approaches in the NLP and NLG communities.

3. LiMo 2023

   Context matters: evaluation of target and context features on variation of object naming

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, Sep 2023
4. MM-NLG 2023

   Are Language-and-Vision Transformers Sensitive to Discourse? A Case Study of ViLBERT

   Ekaterina Voloshina, Nikolai Ilinykh, Simon Dobnik

   In Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), Sep 2023

   Abs

   Language-and-vision models have shown good performance in tasks such as image-caption matching and caption generation. However, it is challenging for such models to generate pragmatically correct captions, which adequately reflect what is happening in one image or several images. It is crucial to evaluate this behaviour to understand underlying reasons behind it. Here we explore to what extent contextual language-and-vision models are sensitive to different discourse, both textual and visual. In particular, we employ one of the multi-modal transformers (ViLBERT) and test if it can match descriptions and images, differentiating them from distractors of different degree of similarity that are sampled from different visual and textual contexts. We place our evaluation in the multi-sentence and multi-image setup, where images and sentences are expected to form a single narrative structure. We show that the model can distinguish different situations but it is not sensitive to differences within one narrative structure. We also show that performance depends on the task itself, for example, what modality remains unchanged in non-matching pairs or how similar non-matching pairs are to original pairs.

5. RESOURCEFUL 2023

   Vector Norms as an Approximation of Syntactic Complexity

   Adam Ek Nikolai Ilinykh

   In Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023), May 2023

   Abs

   Internal representations in transformer models can encode useful linguistic knowledge about syntax. Such knowledge could help optimise the data annotation process. However, identifying and extracting such representations from big language models is challenging. In this paper we evaluate two multilingual transformers for the presence of knowledge about the syntactic complexity of sentences and examine different vector norms. We provide a fine-grained evaluation of different norms in different layers and for different languages. Our results suggest that no single part in the models would be the primary source for the knowledge of syntactic complexity. But some norms show a higher degree of sensitivity to syntactic complexity, depending on the language and model used.

2022

1. CLASP 2022

   In Search of Meaning and Its Representations for Computational Linguistics

   Simon Dobnik, Robin Cooper, Adam Ek, Bill Noble, Staffan Larsson, Nikolai Ilinykh, Vladislav Maraev, Vidya Somashekarappa

   In Proceedings of the 2022 CLASP Conference on (Dis)embodiment, Sep 2022

   Abs

   In this paper we examine different meaning representations that are commonly used in different natural language applications today and discuss their limits, both in terms of the aspects of the natural language meaning they are modelling and in terms of the aspects of the application for which they are used.

2. SemDial 2022

   What to refer to and when? Reference and re-reference in two language-and-vision tasks

   Simon Dobnik, Nikolai Ilinykh, Aram Karimi

   In Proceedings of the 26th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, Aug 2022

   Bib HTML

   @inproceedings{dobnik-etal-2022-refer,
     title = {What to refer to and when? Reference and re-reference in two language-and-vision tasks},
     author = {Dobnik, Simon and Ilinykh, Nikolai and Karimi, Aram},
     booktitle = {Proceedings of the 26th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers},
     month = aug,
     year = {2022},
     address = {Dublin, Ireland},
     publisher = {SEMDIAL},
     url = {http://semdial.org/anthology/Z22-Dobnik_semdial_0017.pdf},
   }

3. Findings ACL 2022

   Attention as Grounding: Exploring Textual and Cross-Modal Attention on Entities and Relations in Language-and-Vision Transformer

   Nikolai Ilinykh Simon Dobnik

   In Findings of the Association for Computational Linguistics: ACL 2022, May 2022

   Abs DOI

   We explore how a multi-modal transformer trained for generation of longer image descriptions learns syntactic and semantic representations about entities and relations grounded in objects at the level of masked self-attention (text generation) and cross-modal attention (information fusion). We observe that cross-attention learns the visual grounding of noun phrases into objects and high-level semantic information about spatial relations, while text-to-text attention captures low-level syntactic knowledge between words. This concludes that language models in a multi-modal task learn different semantic information about objects and relations cross-modally and uni-modally (text-only). Our code is available here: \urlhttps://github.com/GU-CLASP/attention-as-grounding.

4. GEM 2022

   Do Decoding Algorithms Capture Discourse Structure in Multi-Modal Tasks? A Case Study of Image Paragraph Generation

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Dec 2022

   Abs DOI

   This paper describes insights into how different inference algorithms structure discourse in image paragraphs. We train a multi-modal transformer and compare 11 variations of decoding algorithms. We propose to evaluate image paragraphs not only with standard automatic metrics, but also with a more extensive, “under the hood” analysis of the discourse formed by sentences. Our results show that while decoding algorithms can be unfaithful to the reference texts, they still generate grounded descriptions, but they also lack understanding of the discourse structure and differ from humans in terms of attentional structure over images.

5. SLTC 2022

   Hallucinate or ground: how general or specific are object descriptions generated by a vision-and-language transformer?

   Nikolai Ilinykh Simon Dobnik

   In Swedish Language Technology Conference (SLTC) 2022 - Extended Abstracts, Nov 2022

   Bib HTML

   @inproceedings{ilinykh-dobnik-2022-hallucinate,
     title = {Hallucinate or ground: how general or specific are object descriptions generated by a vision-and-language transformer?},
     author = {Ilinykh, Nikolai and Dobnik, Simon},
     booktitle = {Swedish Language Technology Conference (SLTC) 2022 - Extended Abstracts},
     month = nov,
     year = {2022},
     address = {Stockholm, Sweden},
     url = {https://sltc2024.github.io/abstracts/ilinykh.pdf},
   }

6. SLTC 2022

   Towards a computational model of reference and re-reference in visual scenes

   Nikolai Ilinykh, Simon Dobnik, Aram Karimi

   In Swedish Language Technology Conference (SLTC) 2022 - Extended Abstracts, Nov 2022

   Bib HTML

   @inproceedings{ilinykh-dobnik-2022-hallucinatf,
     title = {Towards a computational model of reference and re-reference in visual scenes},
     author = {Ilinykh, Nikolai and Dobnik, Simon and Karimi, Aram},
     booktitle = {Swedish Language Technology Conference (SLTC) 2022 - Extended Abstracts},
     month = nov,
     year = {2022},
     address = {Stockholm, Sweden},
     url = {https://sltc2024.github.io/abstracts/ilinykh.pdf},
   }

7. INLG 2022

   Look and Answer the Question: On the Role of Vision in Embodied Question Answering

   Nikolai Ilinykh, Yasmeen Emampoor, Simon Dobnik

   In Proceedings of the 15th International Conference on Natural Language Generation, Jul 2022

   Abs DOI

8. PVLAM 2022

   Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces

   Nikolai Ilinykh, Rafal Černiavski, Eva Elžbieta Sventickaitė, Viktorija Buzaitė, Simon Dobnik

   In Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind, Jun 2022

   Abs

   We investigate how different augmentation techniques on both textual and visual representations affect the performance of the face description generation model. Specifically, we provide the model with either original images, sketches of faces, facial composites or distorted images. In addition, on the language side, we experiment with different methods to augment the original dataset with paraphrased captions, which are semantically equivalent to the original ones, but differ in terms of their form. We also examine if augmenting the dataset with descriptions from a different domain (e.g., image captions of real-world images) has an effect on the performance of the models. We train models on different combinations of visual and linguistic features and perform both (i) automatic evaluation of generated captions and (ii) examination of how useful different visual features are for the task of facial feature classification. Our results show that although original images encode the best possible representation for the task, the model trained on sketches can still perform relatively well. We also observe that augmenting the dataset with descriptions from a different domain can boost performance of the model. We conclude that face description generation systems are more susceptible to language rather than vision data augmentation. Overall, we demonstrate that face caption generation models display a strong imbalance in the utilisation of language and vision modalities, indicating a lack of proper information fusion. We also describe ethical implications of our study and argue that future work on human face description generation should create better, more representative datasets.

2021

1. Frontiers AI 2021

   What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

   Nikolai Ilinykh Simon Dobnik

   Frontiers in Artificial Intelligence, Jun 2021

   Abs DOI

   Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.

2020

1. INLG 2020

   

   When an Image Tells a Story: The Role of Visual and Semantic Information for Generating Paragraph Descriptions

   Nikolai Ilinykh Simon Dobnik

   In Proceedings of the 13th International Conference on Natural Language Generation, Dec 2020

   Abs DOI Bib HTML Code

   Generating multi-sentence image descriptions is a challenging task, which requires a good model to produce coherent and accurate paragraphs, describing salient objects in the image. We argue that multiple sources of information are beneficial when describing visual scenes with long sequences. These include (i) perceptual information and (ii) semantic (language) information about how to describe what is in the image. We also compare the effects of using two different pooling mechanisms on either a single modality or their combination. We demonstrate that the model which utilises both visual and language inputs can be used to generate accurate and diverse paragraphs when combined with a particular pooling mechanism. The results of our automatic and human evaluation show that learning to embed semantic information along with visual stimuli into the paragraph generation model is not trivial, raising a variety of proposals for future experiments.

   @inproceedings{ilinykh-dobnik-2020-image,
     title = {When an Image Tells a Story: The Role of Visual and Semantic Information for Generating Paragraph Descriptions},
     author = {Ilinykh, Nikolai and Dobnik, Simon},
     editor = {Davis, Brian and Graham, Yvette and Kelleher, John and Sripada, Yaji},
     booktitle = {Proceedings of the 13th International Conference on Natural Language Generation},
     month = dec,
     year = {2020},
     address = {Dublin, Ireland},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.inlg-1.40/},
     doi = {10.18653/v1/2020.inlg-1.40},
     pages = {338--348},
   }

2. WebNLG+ 2020

   

   A General Benchmarking Framework for Text Generation

   Diego Moussallem, Paramjot Kaur, Thiago Ferreira, Chris Lee, Anastasia Shimorina, Felix Conrads, Michael Röder, René Speck, Claire Gardent, Simon Mille, Nikolai Ilinykh, Axel-Cyrille Ngonga Ngomo

   In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), Dec 2020

   Abs Bib HTML Code

   The RDF-to-text task has recently gained substantial attention due to the continuous growth of RDF knowledge graphs in number and size. Recent studies have focused on systematically comparing RDF-to-text approaches on benchmarking datasets such as WebNLG. Although some evaluation tools have already been proposed for text generation, none of the existing solutions abides by the Findability, Accessibility, Interoperability, and Reusability (FAIR) principles and involves RDF data for the knowledge extraction task. In this paper, we present BENG, a FAIR benchmarking platform for Natural Language Generation (NLG) and Knowledge Extraction systems with focus on RDF data. BENG builds upon the successful benchmarking platform GERBIL, is opensource and is publicly available along with the data it contains.

   @inproceedings{moussallem-etal-2020-general,
     title = {A General Benchmarking Framework for Text Generation},
     author = {Moussallem, Diego and Kaur, Paramjot and Ferreira, Thiago and van der Lee, Chris and Shimorina, Anastasia and Conrads, Felix and R{\"o}der, Michael and Speck, Ren{\'e} and Gardent, Claire and Mille, Simon and Ilinykh, Nikolai and Ngonga Ngomo, Axel-Cyrille},
     editor = {Castro Ferreira, Thiago and Gardent, Claire and Ilinykh, Nikolai and van der Lee, Chris and Mille, Simon and Moussallem, Diego and Shimorina, Anastasia},
     booktitle = {Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)},
     month = dec,
     year = {2020},
     address = {Dublin, Ireland (Virtual)},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.webnlg-1.3/},
     pages = {27--33},
   }

3. WebNLG+ 2020

   

   The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)

   Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris Lee, Simon Mille, Diego Moussallem, Anastasia Shimorina

   In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), Dec 2020

   Abs Bib HTML Code Website

   WebNLG+ offers two challenges: (i) mapping sets of RDF triples to English or Russian text (generation) and (ii) converting English or Russian text to sets of RDF triples (semantic parsing). Compared to the eponymous WebNLG challenge, WebNLG+ provides an extended dataset that enable the training, evaluation, and comparison of microplanners and semantic parsers. In this paper, we present the results of the generation and semantic parsing task for both English and Russian and provide a brief description of the participating systems.

   @inproceedings{castro-ferreira-etal-2020-2020,
     title = {The 2020 Bilingual, Bi-Directional {W}eb{NLG}+ Shared Task: Overview and Evaluation Results ({W}eb{NLG}+ 2020)},
     author = {Castro Ferreira, Thiago and Gardent, Claire and Ilinykh, Nikolai and van der Lee, Chris and Mille, Simon and Moussallem, Diego and Shimorina, Anastasia},
     editor = {Castro Ferreira, Thiago and Gardent, Claire and Ilinykh, Nikolai and van der Lee, Chris and Mille, Simon and Moussallem, Diego and Shimorina, Anastasia},
     booktitle = {Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)},
     month = dec,
     year = {2020},
     address = {Dublin, Ireland (Virtual)},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.webnlg-1.7/},
     pages = {55--76},
   }

2019

1. SemDial 2019

   Meet Up! A Corpus of Joint Activity Dialogues in a Visual Environment

   Nikolai Ilinykh, Sina Zarrieß, David Schlangen

   In Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, Sep 2019

   Bib HTML

   @inproceedings{ilinykh-etal-2019-meet,
     title = {Meet Up! A Corpus of Joint Activity Dialogues in a Visual Environment},
     author = {Ilinykh, Nikolai and Zarrie{\ss}, Sina and Schlangen, David},
     booktitle = {Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue - Full Papers},
     month = sep,
     year = {2019},
     address = {London, United Kingdom},
     publisher = {SEMDIAL},
     url = {http://semdial.org/anthology/Z19-Ilinykh_semdial_0006.pdf},
   }

2. INLG 2019

   

   Tell Me More: A Dataset of Visual Scene Description Sequences

   Nikolai Ilinykh, Sina Zarrieß, David Schlangen

   In Proceedings of the 12th International Conference on Natural Language Generation, Oct 2019

   Abs DOI Bib HTML Code

   We present a dataset consisting of what we call image description sequences, which are multi-sentence descriptions of the contents of an image. These descriptions were collected in a pseudo-interactive setting, where the describer was told to describe the given image to a listener who needs to identify the image within a set of images, and who successively asks for more information. As we show, this setup produced nicely structured data that, we think, will be useful for learning models capable of planning and realising such description discourses.

   @inproceedings{ilinykh-etal-2019-tell,
     title = {Tell Me More: A Dataset of Visual Scene Description Sequences},
     author = {Ilinykh, Nikolai and Zarrie{\ss}, Sina and Schlangen, David},
     editor = {van Deemter, Kees and Lin, Chenghua and Takamura, Hiroya},
     booktitle = {Proceedings of the 12th International Conference on Natural Language Generation},
     month = oct,
     year = {2019},
     address = {Tokyo, Japan},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/W19-8621/},
     doi = {10.18653/v1/W19-8621},
     pages = {152--157},
   }

2018

1. SemDial 2018

   slurk – A Lightweight Interaction Server for Dialogue Experiments and Data Collection

   David Schlangen, Tim Diekmann, Nikolai Ilinykh, Sina Zarrieß

   In Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts, Nov 2018

   Bib HTML

   @inproceedings{schlangen-etal-2018-slurk,
     title = {slurk -- A Lightweight Interaction Server for Dialogue Experiments and Data Collection},
     author = {Schlangen, David and Diekmann, Tim and Ilinykh, Nikolai and Zarrie{\ss}, Sina},
     booktitle = {Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts},
     month = nov,
     year = {2018},
     address = {Aix-en-Provence, France},
     publisher = {SEMDIAL},
     url = {http://semdial.org/anthology/Z18-Schlangen_semdial_0036.pdf},
   }

2. SemDial 2018

   MeetUp! A Task for Modelling Visual Dialogue

   David Schlangen, Nikolai Ilinykh, Sina Zarrieß

   In Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts, Nov 2018

   Bib HTML

   @inproceedings{schlangen-etal-2018-meetup,
     title = {MeetUp! A Task for Modelling Visual Dialogue},
     author = {Schlangen, David and Ilinykh, Nikolai and Zarrie{\ss}, Sina},
     booktitle = {Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue - Poster Abstracts},
     month = nov,
     year = {2018},
     address = {Aix-en-Provence, France},
     publisher = {SEMDIAL},
     url = {http://semdial.org/anthology/Z18-Schlangen_semdial_0037.pdf},
   }

3. INLG 2018

   

   The Task Matters: Comparing Image Captioning and Task-Based Dialogical Image Description

   Nikolai Ilinykh, Sina Zarrieß, David Schlangen

   In Proceedings of the 11th International Conference on Natural Language Generation, Nov 2018

   Abs DOI Bib HTML

   Image captioning models are typically trained on data that is collected from people who are asked to describe an image, without being given any further task context. As we argue here, this context independence is likely to cause problems for transferring to task settings in which image description is bound by task demands. We demonstrate that careful design of data collection is required to obtain image descriptions which are contextually bounded to a particular meta-level task. As a task, we use MeetUp!, a text-based communication game where two players have the goal of finding each other in a visual environment. To reach this goal, the players need to describe images representing their current location. We analyse a dataset from this domain and show that the nature of image descriptions found in MeetUp! is diverse, dynamic and rich with phenomena that are not present in descriptions obtained through a simple image captioning task, which we ran for comparison.

   @inproceedings{ilinykh-etal-2018-task,
     title = {The Task Matters: Comparing Image Captioning and Task-Based Dialogical Image Description},
     author = {Ilinykh, Nikolai and Zarrie{\ss}, Sina and Schlangen, David},
     editor = {Krahmer, Emiel and Gatt, Albert and Goudbeek, Martijn},
     booktitle = {Proceedings of the 11th International Conference on Natural Language Generation},
     month = nov,
     year = {2018},
     address = {Tilburg University, The Netherlands},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/W18-6547/},
     doi = {10.18653/v1/W18-6547},
     pages = {397--402},
   }

4. SiVL 2018

   Visual Dialogue Needs Symmetry, Goals, and Dynamics: The Example of the MeetUp Task

   David Schlangen, Nikolai Ilinykh, Sina Zarrieß

   In Proceedings of the 1st Workshop on Shortcomings in Vision and Language (SiVL) at ECCV 2018 - Poster Abstracts, Sep 2018

   Bib HTML

   @inproceedings{schlangen-etal-2018-visualdialogue,
     title = {Visual Dialogue Needs Symmetry, Goals, and Dynamics: The Example of the MeetUp Task},
     author = {Schlangen, David and Ilinykh, Nikolai and Zarrie{\ss}, Sina},
     booktitle = {Proceedings of the 1st Workshop on Shortcomings in Vision and Language (SiVL) at ECCV 2018 - Poster Abstracts},
     month = sep,
     year = {2018},
     address = {Munich, Germany},
     url = {https://clp.ling.uni-potsdam.de/publications/Schlangen-2018-2.pdf},
   }