Work featured in 'Max Planck Forschung'
Our work was featured as a focus topic in the current "Max Planck Forschung" magazine (local link).
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation
Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41 (1), pp. 162-175, 2019.
Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze dataset, which contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance, while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves on the state of the art by 22% (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.
Paper: zhang19_pami.pdf
@article{zhang19_pami,
title = {MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation},
author = {Xucong Zhang and Yusuke Sugano and Mario Fritz and Andreas Bulling},
year = {2019},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
volume = {41},
number = {1},
pages = {162-175},
doi = {10.1109/TPAMI.2017.2778103},
}
InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation
Marc Tonsen, Julian Steil, Yusuke Sugano, Andreas Bulling
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1 (3), pp. 1--21, 2017.
AbstractLinksBibTeXProject Distinguished Paper Award
Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5×5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.
doi: 10.1145/3130971
Paper: tonsen17_imwut.pdf
@article{tonsen17_imwut,
title = {InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation},
author = {Marc Tonsen and Julian Steil and Yusuke Sugano and Andreas Bulling},
year = {2017},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
volume = {1},
number = {3},
pages = {1--21},
doi = {10.1145/3130971},
}
Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors
Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling
Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 1--13, 2018.
AbstractLinksBibTeXProject Best Paper Award
Visual attention is highly fragmented during mobile interactions but the erratic nature of attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened. We instead study attention forecasting – the challenging task of predicting users’ gaze behavior (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate that our method can forecast bidirectional attention shifts and whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.
Paper: steil18_mobilehci.pdf
@inproceedings{steil18_mobilehci,
title = {Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors},
author = {Julian Steil and Philipp M{\"{u}}ller and Yusuke Sugano and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)},
pages = {1--13},
doi = {10.1145/3229434.3229439},
}
Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour
Philipp Müller, Michael Xuelin Huang, Xucong Zhang, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--10, 2018.
Eye contact is one of the most important non-verbal social cues and fundamental to human interactions. However, detecting eye contact without specialized eye tracking equipment poses significant challenges, particularly for multiple people in real-world settings. We present a novel method to robustly detect eye contact in natural three- and four-person interactions using off-the-shelf ambient cameras. Our method exploits that, during conversations, people tend to look at the person who is currently speaking. Harnessing the correlation between people's gaze and speaking behaviour therefore allows our method to automatically acquire training data during deployment and adaptively train eye contact detectors for each target user. We empirically evaluate the performance of our method on a recent dataset of natural group interactions and demonstrate that it achieves a relative improvement over the state-of-the-art method of more than 60%, and also improves over a head pose based baseline.
Paper: mueller18_etra.pdf
@inproceedings{mueller18_etra,
title = {Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour},
author = {Philipp M{\"{u}}ller and Michael Xuelin Huang and Xucong Zhang and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--10},
doi = {10.1145/3204493.3204549},
}
Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior
Philipp Müller, Michael Xuelin Huang, Andreas Bulling
Proc. ACM International Conference on Intelligent User Interfaces (IUI), pp. 153-164, 2018.
Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.
Paper: mueller18_iui.pdf
@inproceedings{mueller18_iui,
title = {Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior},
author = {Philipp M{\"{u}}ller and Michael Xuelin Huang and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Conference on Intelligent User Interfaces (IUI)},
pages = {153-164},
doi = {10.1145/3172944.3172969},
}
Emergent Leadership Detection Across Datasets
Philipp Müller, Andreas Bulling
Proc. ACM International Conference on Multimodal Interaction (ICMI), pp. 274-278, 2019.
Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on single datasets – an unrealistic assumption for real-world applications in which systems are required to also work in settings unseen at training time. It therefore remains unclear whether current methods for emergent leadership detection generalise to similar but new settings and to which extent. To overcome this limitation, we are the first to study a cross-dataset evaluation setting for the emergent leadership detection task. We provide evaluations for within- and cross-dataset prediction using two current datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the robustness of commonly used feature channels and online prediction in the cross-dataset setting. Our evaluations show that using pose and eye contact based features, cross-dataset prediction is possible with an accuracy of 0.68, as such providing another important piece of the puzzle towards real-world emergent leadership detection.
Paper: mueller19_icmi.pdf
@inproceedings{mueller19_icmi,
title = {Emergent Leadership Detection Across Datasets},
author = {Philipp M{\"{u}}ller and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM International Conference on Multimodal Interaction (ICMI)},
pages = {274-278},
doi = {10.1145/3340555.3353721},
}
Eye movements during everyday behavior predict personality traits
Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, Andreas Bulling
Frontiers in Human Neuroscience, 12, pp. 1--8, 2018.
Besides allowing us to perceive our surroundings, eye movements are also a window into our mind and a rich source of information on who we are, how we feel, and what we do. Here we show that eye movements during an everyday task predict aspects of our personality. We tracked eye movements of 42 participants while they ran an errand on a university campus and subsequently assessed their personality traits using well-established questionnaires. Using a state-of-the-art machine learning method and a rich set of features encoding different eye movement characteristics, we were able to reliably predict four of the Big Five personality traits (neuroticism, extraversion, agreeableness, conscientiousness) as well as perceptual curiosity only from eye movements. Further analysis revealed new relations between previously neglected eye movement characteristics and personality. Our findings demonstrate a considerable influence of personality on everyday eye movement control, thereby complementing earlier studies in laboratory settings. Improving automatic recognition and interpretation of human social signals is an important endeavor, enabling innovative design of human–computer systems capable of sensing spontaneous natural user behavior to facilitate efficient interaction and personalization.
Paper: hoppe18_fhns.pdf
@article{hoppe18_fhns,
title = {Eye movements during everyday behavior predict personality traits},
author = {Sabrina Hoppe and Tobias Loetscher and Stephanie Morey and Andreas Bulling},
year = {2018},
journal = {Frontiers in Human Neuroscience},
volume = {12},
pages = {1--8},
doi = {10.3389/fnhum.2018.00105},
}
PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features
Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--10, 2019.
AbstractLinksBibTeXProject Best Video Award
Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking but the first-person camera required to map a user's gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker's first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users' eye movements. To open the shutter without visual input, PrivacEye detects changes in users' eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.
Paper: steil19_etra.pdf
Supplementary: steil19_etra_sup.pdf
@inproceedings{steil19_etra,
title = {PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features},
author = {Julian Steil and Marion Koelle and Wilko Heuten and Susanne Boll and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--10},
doi = {10.1145/3314111.3319913},
video = {https://www.youtube.com/watch?v=Gy61255F8T8},
}
Privacy-Aware Eye Tracking Using Differential Privacy
Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--9, 2019.
AbstractLinksBibTeXProject Best Paper Award
With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users' privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.
Paper: steil19_etra_2.pdf
@inproceedings{steil19_etra_2,
title = {Privacy-Aware Eye Tracking Using Differential Privacy},
author = {Julian Steil and Inken Hagestedt and Michael Xuelin Huang and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--9},
doi = {10.1145/3314111.3319915},
}
Here are some related news you might like to read next:
- Andreas Bulling among the top 2% of most cited researchers in their field
- Andreas Bulling among the top 2% of most cited researchers in their field
- Andreas Bulling among the top 2% HCI researchers in the world
- Work featured by the Max Planck Society
- DFG proposal accepted
