Two papers accepted at EMNLP

We are pleased to announce that the following two papers were accepted for publication at the Conference on Empirical Methods in Natural Language Processing (EMNLP).

Congratulations to the authors!

1. ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions

   Matteo Bortoletto, Constantin Ruhdorfer, Andreas Bulling

   Proc. Empirical Methods in Natural Language Processing (EMNLP), 2025.

   Abstract Links BibTeX Project

   Most existing Theory of Mind (ToM) benchmarks for foundation models rely on variations of the Sally-Anne test, offering only a very limited perspective on ToM and neglecting the complexity of human social interactions. To address this gap, we propose ToM-SSI: a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics. While current ToM benchmarks are limited to text-only or dyadic interactions, ToM-SSI is multimodal and includes group interactions of up to four agents that communicate and move in situated environments. This unique design allows us to study, for the first time, mixed cooperative-obstructive settings and reasoning about multiple agents’ mental state in parallel, thus capturing a wider range of social cognition than existing benchmarks. Our evaluations reveal that current models’ performance is still severely limited, especially in these new tasks, highlighting critical gaps for future research.

   doi:

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/tom-ssi

   @inproceedings{bortoletto25_emnlp, title = {ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions}, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Bulling, Andreas}, booktitle = {Proc. Empirical Methods in Natural Language Processing (EMNLP)}, year = {2025}, doi = {} }
2. 

   Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models

   Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling

   Findings of Empirical Methods in Natural Language Processing (EMNLP), 2025.

   Abstract Links BibTeX Project

   Despite growing interest in Theory of Mind (ToM) tasks for evaluating language models (LMs), little is known about how LMs internally represent mental states of self and others. Understanding these internal mechanisms is critical - not only to move beyond surface-level performance, but also for model alignment and safety, where subtle misattributions of mental states may go undetected in generated outputs. In this work, we present the first systematic investigation of belief representations in LMs by probing models across different scales, training regimens, and prompts - using control tasks to rule out confounds. Our experiments provide evidence that both model size and fine-tuning substantially improve LMs’ internal representations of others’ beliefs, which are structured - not mere by-products of spurious correlations - yet brittle to prompt variations. Crucially, we show that these representations can be strengthened: targeted edits to model activations can correct wrong ToM inferences.

   doi: 10.48550/arXiv.2406.17513

   Paper: bortoletto25_femnlp.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/mental-states-in-LMs

   @inproceedings{bortoletto25_femnlp, title = {Brittle {{Minds}}, {{Fixable Activations}}: {{Understanding Belief Representations}} in {{Language Models}}}, shorttitle = {Brittle {{Minds}}, {{Fixable Activations}}}, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Shi, Lei and Bulling, Andreas}, booktitle = {Findings of Empirical Methods in Natural Language Processing (EMNLP)}, year = {2025}, doi = {10.48550/arXiv.2406.17513} }

Here are some related news you might like to read next:

• New paper accepted in TMLR
• New paper accepted at IJCAI Workshop
• New paper accepted at UIST
• New paper accepted at GD
• New paper accepted in IMWUT