Publications
2026
The Yōkai Learning Environment: Tracking Beliefs Over Space and Time
Constantin Ruhdorfer, Matteo Bortoletto, Johannes Forkel, Jakob Nicolaus Foerster, Andreas Bulling
Proc. Reinforcement Learning Conference (RLC), 2026.
AbstractLinksBibTeXProject
The ability to cooperate with unknown partners is a central challenge in cooperative AI and widely studied in the form of zero-shot coordination (ZSC) which evaluates an algorithm by measuring the performance of independently trained agents when paired. The Hanabi Learning Environment (HLE) has become the dominant benchmark for ZSC, but recent work has achieved near-perfect inter-seed cross-play performance, limiting its ability to track algorithmic progress. We introduce the Yokai Learning Environment (YLE) - an open-source multi-agent RL benchmark in which effective collaboration requires building common ground by tracking and updating beliefs over moving cards, reasoning under ambiguous hints, and deciding when to terminate the game based on inferred shared knowledge - features absent in the HLE, where beliefs are tied to hand slots and hints are truthful by rule. We evaluate the leading ZSC methods, including High-Entropy IPPO, Other-Play, and Off-Belief Learning, which achieve near-perfect inter-seed cross-play in the HLE, and show that in the YLE they exhibit persistent SP–XP gaps, degraded early-ending calibration, and weaker belief representations in cross-play, indicating failure to maintain consistent internal models with unseen partners. Methods that perform best in the HLE do not perform best in the YLE, indicating that progress measured on a single benchmark may not generalise. Together, these results establish YLE as a challenging new ZSC benchmark.
@inproceedings{ruhdorfer26_rlc,
title = {The Yōkai Learning Environment: Tracking Beliefs Over Space and Time},
author = {Ruhdorfer, Constantin and Bortoletto, Matteo and Forkel, Johannes and Foerster, Jakob Nicolaus and Bulling, Andreas},
year = {2026},
booktitle = {Proc. Reinforcement Learning Conference (RLC)},
}
Unsupervised Partner Design Enables Robust Ad-hoc Teamwork
Constantin Ruhdorfer, Matteo Bortoletto, Victor Oei, Anna Penzkofer, Andreas Bulling
Proc. International Conference on Machine Learning (ICML), 2026.
AbstractLinksBibTeXProject Spotlight
We introduce Unsupervised Partner Design (UPD), a population-free multi-agent reinforcement learning method for robust ad-hoc teamwork. UPD generates training partners on-the-fly and selects them adaptively based on a learnability criterion, removing the need for pre-trained partner populations or manual parameter tuning. We show that this simple mechanism enables effective partner diversity and can be extended to joint partner-environment selection when a procedural level generator is available. Across Level-Based Foraging, Overcooked-AI, and the Overcooked Generalisation Challenge, UPD consistently achieves strong performance compared to both population-based and population-free baselines. In a human-AI user study, agents trained with UPD achieve higher returns and are rated as more adaptive, more human-like, and less frustrating than all evaluated baseline methods.
@inproceedings{ruhdorfer26_icml,
title = {Unsupervised Partner Design Enables Robust Ad-hoc Teamwork},
author = {Ruhdorfer, Constantin and Bortoletto, Matteo and Oei, Victor and Penzkofer, Anna and Bulling, Andreas},
year = {2026},
booktitle = {Proc. International Conference on Machine Learning (ICML)},
shorttitle = {{UPD}},
}
ProToM: Promoting Prosocial Behaviour via Theory of Mind-Informed Feedback
Matteo Bortoletto, Yichao Zhou, Lance Ying, Tianmin Shu, Andreas Bulling
Findings of the Association for Computational Linguistics (ACL), 2026.
AbstractBibTeXProject
While humans are inherently social creatures, the challenge of identifying when and how to assist and collaborate with others - particularly when pursuing independent goals - can hinder cooperation. To address this challenge, we aim to develop an AI system that provides useful feedback to promote prosocial behaviour - actions that benefit others, even when not directly aligned with one's own goals. We introduce ProToM, a Theory of Mind-informed facilitator that promotes prosocial actions in multi-agent systems by providing targeted, context-sensitive feedback to individual agents. ProToM first infers agents' goals using Bayesian inverse planning, then selects feedback to communicate by maximising expected utility, conditioned on the inferred goal distribution. We evaluate our approach against baselines in two multi-agent environments: Doors, Keys, and Gems, as well as Overcooked. Our results suggest that state-of-the-art large language and reasoning models fall short of communicating feedback that is both contextually grounded and well-timed - leading to higher communication overhead and lower success rates. In contrast, ProToM provides targeted and helpful feedback, achieving a higher success rate, shorter task completion times, and is consistently preferred by human users.
@inproceedings{bortoletto26_acl,
title = {{ProToM}: Promoting Prosocial Behaviour via Theory of Mind-Informed Feedback},
author = {Matteo Bortoletto and Zhou, Yichao and Ying, Lance and Shu, Tianmin and Andreas Bulling},
year = {2026},
booktitle = {Findings of the Association for Computational Linguistics (ACL)},
}
2025
ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions
Matteo Bortoletto, Constantin Ruhdorfer, Andreas Bulling
Proc. Empirical Methods in Natural Language Processing (EMNLP), 2025.
AbstractLinksBibTeXProject
Most existing Theory of Mind (ToM) benchmarks for foundation models rely on variations of the Sally-Anne test, offering only a very limited perspective on ToM and neglecting the complexity of human social interactions. To address this gap, we propose ToM-SSI: a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics. While current ToM benchmarks are limited to text-only or dyadic interactions, ToM-SSI is multimodal and includes group interactions of up to four agents that communicate and move in situated environments. This unique design allows us to study, for the first time, mixed cooperative-obstructive settings and reasoning about multiple agents' mental state in parallel, thus capturing a wider range of social cognition than existing benchmarks. Our evaluations reveal that current models' performance is still severely limited, especially in these new tasks, highlighting critical gaps for future research.
@inproceedings{bortoletto25_emnlp,
title = {ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions},
author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Bulling, Andreas},
year = {2025},
booktitle = {Proc. Empirical Methods in Natural Language Processing (EMNLP)},
doi = {10.18653/v1/2025.emnlp-main.1642},
}
Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models
Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling
Findings of Empirical Methods in Natural Language Processing (EMNLP), 2025.
AbstractLinksBibTeXProject
Despite growing interest in Theory of Mind (ToM) tasks for evaluating language models (LMs), little is known about how LMs internally represent mental states of self and others. Understanding these internal mechanisms is critical - not only to move beyond surface-level performance, but also for model alignment and safety, where subtle misattributions of mental states may go undetected in generated outputs. In this work, we present the first systematic investigation of belief representations in LMs by probing models across different scales, training regimens, and prompts - using control tasks to rule out confounds. Our experiments provide evidence that both model size and fine-tuning substantially improve LMs' internal representations of others' beliefs, which are structured - not mere by-products of spurious correlations - yet brittle to prompt variations. Crucially, we show that these representations can be strengthened: targeted edits to model activations can correct wrong ToM inferences.
@inproceedings{bortoletto25_femnlp,
title = {Brittle {{Minds}}, {{Fixable Activations}}: {{Understanding Belief Representations}} in {{Language Models}}},
author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Shi, Lei and Bulling, Andreas},
year = {2025},
booktitle = {Findings of Empirical Methods in Natural Language Processing (EMNLP)},
doi = {10.18653/v1/2025.findings-emnlp.1226},
shorttitle = {Brittle {{Minds}}, {{Fixable Activations}}},
}
The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design
Constantin Ruhdorfer, Matteo Bortoletto, Anna Penzkofer, Andreas Bulling
Transactions on Machine Learning Research (TMLR), pp. 1-25, 2025.
AbstractLinksBibTeXProject
We introduce the Overcooked Generalisation Challenge (OGC) - a new benchmark for evaluating reinforcement learning (RL) agents on their ability to cooperate with unknown partners in unfamiliar environments. Existing work typically evaluated cooperative RL only in their training environment or with their training partners, thus seriously limiting our ability to understand agents' generalisation capacity - an essential requirement for future collaboration with humans. The OGC extends Overcooked-AI to support dual curriculum design (DCD). It is fully GPU-accelerated, open-source, and integrated into the minimax DCD benchmark suite. Compared to prior DCD benchmarks, where designers manipulate only minimal elements of the environment, OGC introduces a significantly richer design space: full kitchen layouts with multiple objects that require the designer to account for interaction dynamics between agents. We evaluate state-of-the-art DCD algorithms alongside scalable neural architectures and find that current methods fail to produce agents that generalise effectively to novel layouts and unfamiliar partners. Our results indicate that both agents and curriculum designers struggle with the joint challenge of partner and environment generalisation. These findings establish OGC as a demanding testbed for cooperative generalisation and highlight key directions for future research.
@article{ruhdorfer25_tmlr,
title = {The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design},
author = {Constantin Ruhdorfer and Matteo Bortoletto and Anna Penzkofer and Andreas Bulling},
year = {2025},
journal = {Transactions on Machine Learning Research (TMLR)},
pages = {1-25},
url = {https://openreview.net/forum?id=K2KtcMlW6j},
}
2024
Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions
Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling
Proc. 27th European Conference on Artificial Intelligence (ECAI), pp. 866--873, 2024.
AbstractLinksBibTeXProject
We propose MToMnet - a Theory of Mind (ToM) neural network for predicting beliefs and their dynamics during human social interactions from multimodal input. ToM is key for effective nonverbal human communication and collaboration, yet, existing methods for belief modelling have not included explicit ToM modelling or have typically been limited to one or two modalities. MToMnet encodes contextual cues (scene videos and object locations) and integrates them with person-specific cues (human gaze and body language) in a separate MindNet for each person. Inspired by prior research on social cognition and computational ToM, we propose three different MToMnet variants: two involving fusion of latent representations and one involving re-ranking of classification scores. We evaluate our approach on two challenging real-world datasets, one focusing on belief prediction, while the other examining belief dynamics prediction. Our results demonstrate that MToMnet surpasses existing methods by a large margin while at the same time requiring a significantly smaller number of parameters. Taken together, our method opens up a highly promising direction for future work on artificial intelligent systems that can robustly predict human beliefs from their non-verbal behaviour and, as such, more effectively collaborate with humans.
@inproceedings{bortoletto24_ecai,
title = {Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions},
author = {Matteo Bortoletto and Constantin Ruhdorfer and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. 27th European Conference on Artificial Intelligence (ECAI)},
pages = {866--873},
doi = {10.3233/FAIA240573},
}
Benchmarking Mental State Representations in Language Models
Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling
Proc. ICML 2024 Workshop on Mechanistic Interpretability, pp. 1--21, 2024.
AbstractLinksBibTeXProject
While numerous works have assessed the generative performance of language models (LMs) on tasks requiring Theory of Mind reasoning, research into the models' internal representation of mental states remains limited. Recent work has used probing to demonstrate that LMs can represent beliefs of themselves and others. However, these claims are accompanied by limited evaluation, making it difficult to assess how mental state representations are affected by model design and training choices. We report an extensive benchmark with various LM types with different model sizes, fine-tuning approaches, and prompt designs to study the robustness of mental state representations and memorisation issues within the probes. Our results show that the quality of models' internal representations of the beliefs of others increases with model size and, more crucially, with fine-tuning. We are the first to study how prompt variations impact probing performance on theory of mind tasks. We demonstrate that models' representations are sensitive to prompt variations, even when such variations should be beneficial. Finally, we complement previous activation editing experiments on Theory of Mind tasks and show that it is possible to improve models' reasoning performance by steering their activations without the need to train any probe.
@inproceedings{bortoletto24_icmlw,
title = {Benchmarking Mental State Representations in Language Models},
author = {Matteo Bortoletto and Constantin Ruhdorfer and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. ICML 2024 Workshop on Mechanistic Interpretability},
pages = {1--21},
url = {https://openreview.net/forum?id=yEwEVoH9Be},
}
Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition
Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, Andreas Bulling
Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1--16, 2024.
AbstractLinksBibTeXProject
Recent work on dialogue-based collaborative plan acquisition (CPA) has suggested that Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge. Although ToM was claimed to be important for effective collaboration, its real impact on this novel task remains under-explored. By representing plans as graphs and by exploiting task-specific constraints we show that, as performance on CPA nearly doubles when predicting one's own missing knowledge, the improvements due to ToM modelling diminish. This phenomenon persists even when evaluating existing baseline methods. To better understand the relevance of ToM for CPA, we report a principled performance comparison of models with and without ToM features. Results across different models and ablations consistently suggest that learned ToM features are indeed more likely to reflect latent patterns in the data with no perceivable link to ToM. This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.
@inproceedings{bortoletto24_acl,
title = {Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition},
author = {Matteo Bortoletto and Constantin Ruhdorfer and Adnen Abdessaied and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL)},
pages = {1--16},
doi = {10.18653/v1/2024.acl-long.266},
}
Neural Reasoning About Agents’ Goals, Preferences, and Actions
Matteo Bortoletto, Lei Shi, Andreas Bulling
Proc. 38th AAAI Conference on Artificial Intelligence (AAAI), pp. 456--464, 2024.
AbstractLinksBibTeXProject
We propose the Intuitive Reasoning Network (IRENE) – a novel neural model for intuitive psychological reasoning about agents’ goals, preferences, and actions that can generalise previous experiences to new situations. IRENE combines a graph neural network for learning agent and world state representations with a transformer to encode the task context. When evaluated on the challenging Baby Intuitions Benchmark, IRENE achieves new state-of-the-art performance on three out of its five tasks – with up to 48.9 % improvement. In contrast to existing methods, IRENE is able to bind preferences to specific agents, to better distinguish between rational and irrational agents, and to better understand the role of blocking obstacles. We also investigate, for the first time, the influence of the training tasks on test performance. Our analyses demonstrate the effectiveness of IRENE in combining prior knowledge gained during training for unseen evaluation tasks.
@inproceedings{bortoletto24_aaai,
title = {Neural Reasoning About Agents’ Goals, Preferences, and Actions},
author = {Matteo Bortoletto and Lei Shi and Andreas Bulling},
year = {2024},
booktitle = {Proc. 38th AAAI Conference on Artificial Intelligence (AAAI)},
volume = {38},
number = {1},
pages = {456--464},
doi = {10.1609/aaai.v38i1.27800},
}
2023
Exploring Natural Language Processing Methods for Interactive Behaviour Modelling
Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling
Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1--22, 2023.
AbstractLinksBibTeXProject Best Student Paper Nomination
Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.
@inproceedings{zhang23_interact,
title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling},
author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas},
year = {2023},
booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)},
pages = {1--22},
publisher = {Springer},
}
Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention
Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling
Proc. Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639--3646, 2023.
AbstractLinksBibTeXProject
We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.
@inproceedings{sood23_cogsci,
title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention},
author = {Ekta Sood and Lei Shi and Matteo Bortoletto and Yao Wang and Philipp Müller and Andreas Bulling},
year = {2023},
booktitle = {Proc. Annual Meeting of the Cognitive Science Society (CogSci)},
pages = {3639--3646},
}
