The mouse is a pervasive input device used for a wide range of interactive applications. However, computational modelling of mouse behaviour typically requires time-consuming design and extraction of handcrafted features, or approaches that are application-specific. We instead propose Mouse2Vec – a novel self-supervised method designed to learn semantic representations of mouse behaviour that are reusable across users and applications. Mouse2Vec uses a Transformer-based encoder-decoder architecture, which is specifically geared for mouse data: During pretraining, the encoder learns an embedding of input mouse trajectories while the decoder reconstructs the input and simultaneously detects mouse click events. We show that the representations learned by our method can identify interpretable mouse behaviour clusters and retrieve similar mouse trajectories. We also demonstrate on three sample downstream tasks that the representations can be practically used to augment mouse data for training supervised methods and serve as an effective feature extractor.

@inproceedings{zhang24_chi,
  title = {Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour},
  author = {Guanhua Zhang and Zhiming Hu and Mihai B{\^a}ce and Andreas Bulling},
  year = {2024},
  booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
  pages = {1--17},
  doi = {10.1145/3613904.3642141},
}

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting G. Zhang. Z. Hu was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - EXC 2075 – 390740016. M. Bâce was funded by the Swiss National Science Foundation (SNSF) through a Postdoc. Mobility Fellowship (grant number 214434) while at the University of Stuttgart. We acknowledge the support by the Stuttgart Center for Simulation Science (SimTech). A. Bulling was funded by the European Research Council (ERC; grant agreement 801708). We would like to thank Yuxuan Cheng for her technical support, and thank anonymous reviewers for their helpful feedback.