Publications
2025
V2Dial: Unification of Video and Visual Dialog via Multimodal Experts
Adnen Abdessaied, Anna Rohrbach, Marcus Rohrbach, Andreas Bulling
Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
AbstractLinksBibTeXProject
We present V2Dial - a novel expert-based model specifically geared towards simultaneously handling image and video input data for multimodal conversational tasks. Current multimodal models primarily focus on simpler tasks (e.g., VQA, VideoQA, video-text retrieval) and often neglect the more challenging conversational counterparts, such as video and visual/image dialog. Moreover, works on both conversational tasks evolved separately from each other despite their apparent similarities limiting their applicability potential. To this end, we propose to unify both tasks using a single model that for the first time jointly learns the spatial and temporal features of images and videos by routing them through dedicated experts and aligns them using matching and contrastive learning techniques. Furthermore, we systemically study the domain shift between the two tasks by investigating whether and to what extent these seemingly related tasks can mutually benefit from their respective training data. Extensive evaluations on the widely used video and visual dialog datasets of AVSD and VisDial show that our model achieves new state-of-the-art results across four benchmarks both in zero-shot and fine-tuning settings.
@inproceedings{abdessaied25_cvpr,
title = {V<sup>2</sup>Dial: Unification of Video and Visual Dialog via Multimodal Experts},
author = {Adnen Abdessaied and Anna Rohrbach and Marcus Rohrbach and Andreas Bulling},
year = {2025},
booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
doi = {10.1109/CVPR52734.2025.00807},
}
2024
Multi-Modal Video Dialog State Tracking in the Wild
Adnen Abdessaied, Lei Shi, Andreas Bulling
Proc. 18th European Conference on Computer Vision (ECCV), pp. 1--25, 2024.
AbstractLinksBibTeXProject
We present MST-MIXER – a novel video dialog model operating over a generic multi-modal state tracking scheme. Current models that claim to perform multi-modal state tracking fall short of two major aspects: (1) They either track only one modality (mostly the visual input) or (2) they target synthetic datasets that do not reflect the complexity of real-world in the wild scenarios. Our model addresses these two limitations in an attempt to close this crucial research gap. Specifically, MST-MIXER first tracks the most important constituents of each input modality. Then, it predicts the missing underlying structure of the selected constituents of each modality by learning local latent graphs using a novel multi-modal graph structure learning method. Subsequently, the learned local graphs and features are parsed together to form a global graph operating on the mix of all modalities which further refines its structure and node embeddings. Finally, the fine-grained graph node features are used to enhance the hidden states of the backbone Vision-Language Model (VLM). MST-MIXER achieves new state-of-the-art results on five challenging benchmarks.
@inproceedings{abdessaied24_eccv,
title = {Multi-Modal Video Dialog State Tracking in the Wild},
author = {Adnen Abdessaied and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. 18th European Conference on Computer Vision (ECCV)},
pages = {1--25},
doi = {10.1007/978-3-031-72998-0_20},
}
OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog
Adnen Abdessaied, Manuel von Hochmeister, Andreas Bulling
Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 1--11, 2024.
AbstractLinksBibTeXProject
We present the Object Language Video Transformer (OLViT) – a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.
@inproceedings{abdessaied24_coling,
title = {OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog},
author = {Adnen Abdessaied and Manuel von Hochmeister and Andreas Bulling},
year = {2024},
booktitle = {Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)},
pages = {1--11},
}
VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs
Adnen Abdessaied, Lei Shi, Andreas Bulling
Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5805--5814, 2024.
AbstractLinksBibTeXProject
We propose VD-GR – a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs). Prior works mainly focused on one class of models at the expense of the other, thus missing out on the opportunity of combining their respective benefits. At the core of VD-GR is a novel integration mechanism that alternates between spatial-temporal multi-modal GNNs and BERT layers, and that covers three distinct contributions: First, we use multi-modal GNNs to process the features of each modality (image, question, and dialog history) and exploit their local structures before performing
BERT global attention. Second, we propose hub-nodes that link to all other nodes within one modality graph, allowing the model to propagate information from one GNN (modality) to the other in a cascaded manner. Third, we augment the BERT hidden states with fine-grained multi-modal GNN features before passing them to the next VD-GR layer. Evaluations on VisDial v1.0, VisDial v0.9, VisDialConv, and VisPro show that VD-GR achieves new state-of-the-art results across all four datasets
@inproceedings{abdessaied24_wacv,
title = {VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs},
author = {Adnen Abdessaied and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
pages = {5805--5814},
doi = {10.1109/WACV57701.2024.00570},
}
Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition
Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, Andreas Bulling
Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1--16, 2024.
AbstractLinksBibTeXProject
Recent work on dialogue-based collaborative plan acquisition (CPA) has suggested that Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge. Although ToM was claimed to be important for effective collaboration, its real impact on this novel task remains under-explored. By representing plans as graphs and by exploiting task-specific constraints we show that, as performance on CPA nearly doubles when predicting one's own missing knowledge, the improvements due to ToM modelling diminish. This phenomenon persists even when evaluating existing baseline methods. To better understand the relevance of ToM for CPA, we report a principled performance comparison of models with and without ToM features. Results across different models and ablations consistently suggest that learned ToM features are indeed more likely to reflect latent patterns in the data with no perceivable link to ToM. This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.
@inproceedings{bortoletto24_acl,
title = {Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition},
author = {Matteo Bortoletto and Constantin Ruhdorfer and Adnen Abdessaied and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL)},
pages = {1--16},
doi = {10.18653/v1/2024.acl-long.266},
}
2022
Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA
Adnen Abdessaied, Ekta Sood, Andreas Bulling
Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP), pp. 1--12, 2022.
AbstractLinksBibTeXProject
[Equal contribution by the first two authors.] We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA). Our model combines two original contributions: A multimodal fast-learning feature fusion (FLF) block and a mechanism that uses self-attended language features to separately guide neural attention on both static and dynamic visual features extracted from individual video frames and short video clips. When trained from scratch, VLCN achieves competitive results with the state of the art on both MSVD-QA and MSRVTT-QA with 38.06% and 36.01% test accuracies, respectively. Through an ablation study, we further show that FLF improves generalization across different VideoQA datasets and performance for question types that are notoriously challenging in current datasets, such as long questions that require deeper reasoning as well as questions with rare answers.
@inproceedings{abdessaied22_repl4NLP,
title = {Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA},
author = {Adnen Abdessaied and Ekta Sood and Andreas Bulling},
year = {2022},
booktitle = {Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP)},
pages = {1--12},
}
Neuro-Symbolic Visual Dialog
Adnen Abdessaied, Mihai Bâce, Andreas Bulling
Proc. 29th Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 1--11, 2022.
AbstractLinksBibTeXProject
We propose Neuro-Symbolic Visual Dialog (NSVD) —the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog —a relative improvement of more than 10% over the state of the art —while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.
@inproceedings{abdessaied22_coling,
title = {Neuro-Symbolic Visual Dialog},
author = {Adnen Abdessaied and Mihai Bâce and Andreas Bulling},
year = {2022},
booktitle = {Proc. 29th Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)},
pages = {1--11},
}
