Continual ToM: Benchmarking Belief Tracking in Language Models
Matteo Zaramella, Fabian Kögel, Victor Oei, Andreas Bulling
Proc. Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2027.
Abstract
Current benchmarks for evaluating theory of mind (ToM) in language models (LMs) are fundamentally limited by their retrospective nature, granting models access to the full narrative to predict beliefs. This does not capture the demands of real-world interactive settings, where agents must establish and track beliefs dynamically as information unfolds. To bridge this gap, we introduce Event-by-Event (EbE) prompting, a simple yet effective evaluation protocol that assesses continual belief tracking on existing benchmarks by providing their narrative incrementally. Applying EbE to four widely used ToM benchmarks, we conduct extensive evaluations on state-of-the-art open-weight models and two closed-source models. Despite the simplicity of our approach, we find that ToM benchmark scores drop substantially across model families, scales, and thinking modes, revealing critical weaknesses in continual belief tracking that retrospective evaluation misses.
Links
BibTeX
@inproceedings{zaramella27_eacl,
title = {Continual {{ToM}}: {{Benchmarking Belief Tracking}} in {{Language Models}}},
author = {Zaramella, Matteo and Kögel, Fabian and Oei, Victor and Bulling, Andreas},
year = {2027},
booktitle = {Proc. Conference of the European Chapter of the Association for Computational Linguistics (EACL)},
}