Publications
2019
PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features
Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--10, 2019.
AbstractLinksBibTeXProject Best Video Award
Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking but the first-person camera required to map a user's gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker's first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users' eye movements. To open the shutter without visual input, PrivacEye detects changes in users' eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.
@inproceedings{steil19_etra,
title = {PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features},
author = {Julian Steil and Marion Koelle and Wilko Heuten and Susanne Boll and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--10},
doi = {10.1145/3314111.3319913},
video = {https://www.youtube.com/watch?v=Gy61255F8T8},
}
Privacy-Aware Eye Tracking Using Differential Privacy
Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--9, 2019.
AbstractLinksBibTeXProject Best Paper Award
With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users' privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.
@inproceedings{steil19_etra_2,
title = {Privacy-Aware Eye Tracking Using Differential Privacy},
author = {Julian Steil and Inken Hagestedt and Michael Xuelin Huang and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--9},
doi = {10.1145/3314111.3319915},
}
InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation
Julian Steil, Marc Tonsen, Yusuke Sugano, Andreas Bulling
ACM SIGMOBILE Mobile Computing and Communications Review, 23 (2), pp. 30-34, 2019.
AbstractLinksBibTeXProject
Despite their potential for a range of exciting new applications, mobile eye trackers suffer from several fundamental usability problems. InvisibleEye is an innovative approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames, as well as appearance-based gaze estimation to directly estimate gaze from the eye images. Through evaluation on three large-scale, increasingly realistic datasets, we show that InvisibleEyes can achieve a person-specific gaze estimation accuracy of up to 2.04° using three camera pairs with a resolution of only 3x3 pixels.
@article{steil19_sigmobile,
title = {InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation},
author = {Steil, Julian and Tonsen, Marc and Sugano, Yusuke and Bulling, Andreas},
year = {2019},
journal = {ACM SIGMOBILE Mobile Computing and Communications Review},
volume = {23},
number = {2},
pages = {30-34},
}
2018
Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets
Julian Steil, Michael Xuelin Huang, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--9, 2018.
AbstractLinksBibTeXProject
Fixations are widely analysed in human vision, gaze-based interaction, and experimental psychology research. However, robust fixation detection in mobile settings is profoundly challenging given the prevalence of user and gaze target motion. These movements feign a shift in gaze estimates in the frame of reference defined by the eye tracker's scene camera. To address this challenge, we present a novel fixation detection method for head-mounted eye trackers. Our method exploits that, independent of user or gaze target motion, target appearance remains about the same during a fixation. It extracts image information from small regions around the current gaze position and analyses the appearance similarity of these gaze patches across video frames to detect fixations. We evaluate our method using fine-grained fixation annotations on a five-participant indoor dataset (MPIIEgoFixation) with more than 2,300 fixations in total. Our method outperforms commonly used velocity- and dispersion-based algorithms, which highlights its significant potential to analyse scene image information for eye movement detection.
@inproceedings{steil18_etra,
title = {Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets},
author = {Julian Steil and Michael Xuelin Huang and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--9},
doi = {10.1145/3204493.3204538},
}
Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors
Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling
Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 1--13, 2018.
AbstractLinksBibTeXProject Best Paper Award
Visual attention is highly fragmented during mobile interactions but the erratic nature of attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened. We instead study attention forecasting – the challenging task of predicting users’ gaze behavior (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate that our method can forecast bidirectional attention shifts and whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.
@inproceedings{steil18_mobilehci,
title = {Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors},
author = {Julian Steil and Philipp M{\"{u}}ller and Yusuke Sugano and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)},
pages = {1--13},
doi = {10.1145/3229434.3229439},
}
2017
InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation
Marc Tonsen, Julian Steil, Yusuke Sugano, Andreas Bulling
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1 (3), pp. 1--21, 2017.
AbstractLinksBibTeXProject Distinguished Paper Award
Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5×5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.
@article{tonsen17_imwut,
title = {InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation},
author = {Marc Tonsen and Julian Steil and Yusuke Sugano and Andreas Bulling},
year = {2017},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)},
volume = {1},
number = {3},
pages = {1--21},
doi = {10.1145/3130971},
}
2016
3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers
Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 197-200, 2016.
AbstractLinksBibTeXProject
3D gaze information is important for scene-centric attention analysis, but accurate estimation and analysis of 3D gaze in real-world environments remains challenging. We present a novel 3D gaze estimation method for monocular head-mounted eye trackers. In contrast to previous work, our method does not aim to infer 3D eye- ball poses, but directly maps 2D pupil positions to 3D gaze directions in scene camera coordinate space. We first provide a detailed discussion of the 3D gaze estimation task and summarize different methods, including our own. We then evaluate the performance of different 3D gaze estimation approaches using both simulated and real data. Through experimental validation, we demonstrate the effectiveness of our method in reducing parallax error, and we identify research challenges for the design of 3D calibration procedures.
@inproceedings{mansouryar16_etra,
title = {3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers},
author = {Mohsen Mansouryar and Julian Steil and Yusuke Sugano and Andreas Bulling},
year = {2016},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {197-200},
doi = {10.1145/2857491.2857530},
}
2015
Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models
Julian Steil, Andreas Bulling
Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 75-85, 2015.
AbstractLinksBibTeXProject
Human visual behaviour has significant potential for activity recognition and computational behaviour analysis, but previous works focused on supervised methods and recognition of predefined activity classes based on short-term eye movement recordings. We propose a fully unsupervised method to discover users' everyday activities from their long-term visual behaviour. Our method combines a bag-of-words representation of visual behaviour that encodes saccades, fixations, and blinks with a latent Dirichlet allocation (LDA) topic model. We further propose different methods to encode saccades for their use in the topic model. We evaluate our method on a novel long-term gaze dataset that contains full-day recordings of natural visual behaviour of 10 participants (more than 80 hours in total). We also provide annotations for eight sample activity classes (outdoor, social interaction, focused work, travel, reading, computer work, watching media, eating) and periods with no specific activity. We show the ability of our method to discover these activities with performance competitive with that of previously published supervised methods.
@inproceedings{steil15_ubicomp,
title = {Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models},
author = {Steil, Julian and Bulling, Andreas},
year = {2015},
booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)},
pages = {75-85},
doi = {10.1145/2750858.2807520},
}
