2019

Journal Articles

1. 

   MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(1), pp. 162-175, 2019.

   Abstract Links BibTeX Project

   Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze dataset, which contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance, while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves on the state of the art by 22% (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.

   doi:10.1109/TPAMI.2017.2778103

   zhang19_pami.pdf

   @article{zhang19_pami, title = {MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, year = {2019}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)}, doi = {10.1109/TPAMI.2017.2778103}, pages = {162-175}, volume = {41}, number = {1} }
2. 

   Classifying Attention Types with Thermal Imaging and Eye Tracking

   Yomna Abdelrahman, Anam Ahmad Khan, Joshua Newn, Eduardo Velloso, Sherine Ashraf Safwat, James Bailey, Andreas Bulling, Frank Vetere, Albrecht Schmidt

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 3(3), pp. 69:1-69:27, 2019.

   Abstract Links BibTeX Project

   Despite the importance of attention in user performance, current methods for attention classification do not allow to discriminate between different attention types. We propose a novel method that combines thermal imaging and eye tracking to unobtrusively classify four types of attention: sustained, alternating, selective, and divided. We collected a data set in which we stimulate these four attention types in a user study (N=22) using combinations of audio and visual stimuli while measuring users’ facial temperature and eye movement. Using a Logistic Regression on features extracted from both sensing technologies, we can classify the four attention types with high AUC scores up to 75.7% for the user independent-condition independent, 87% for the user-independent-condition dependent, and 77.4% for the user-dependent prediction. Our findings not only demonstrate the potential of thermal imaging and eye tracking for unobtrusive classification of different attention types but also pave the way for novel applications for attentive user interfaces and attention-aware computing.

   doi:10.1145/3351227

   abdelrahman19_imwut.pdf

   @article{abdelrahman19_imwut, author = {Abdelrahman, Yomna and Khan, Anam Ahmad and Newn, Joshua and Velloso, Eduardo and Safwat, Sherine Ashraf and Bailey, James and Bulling, Andreas and Vetere, Frank and Schmidt, Albrecht}, title = {Classifying Attention Types with Thermal Imaging and Eye Tracking}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, year = {2019}, volume = {3}, number = {3}, pages = {69:1-69:27}, doi = {10.1145/3351227} }
3. 

   InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation

   Julian Steil, Marc Tonsen, Yusuke Sugano, Andreas Bulling

   ACM SIGMOBILE Mobile Computing and Communications Review, 23(2), pp. 30-34, 2019.

   Abstract Links BibTeX Project

   Despite their potential for a range of exciting new applications, mobile eye trackers suffer from several fundamental usability problems. InvisibleEye is an innovative approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames, as well as appearance-based gaze estimation to directly estimate gaze from the eye images. Through evaluation on three large-scale, increasingly realistic datasets, we show that InvisibleEyes can achieve a person-specific gaze estimation accuracy of up to 2.04° using three camera pairs with a resolution of only 3x3 pixels.

   steil19_sigmobile.pdf

   @article{steil19_sigmobile, author = {Steil, Julian and Tonsen, Marc and Sugano, Yusuke and Bulling, Andreas}, title = {InvisibleEye: Fully Embedded Mobile Eye Tracking Using Appearance-Based Gaze Estimation}, journal = {ACM SIGMOBILE Mobile Computing and Communications Review}, year = {2019}, volume = {23}, number = {2}, pages = {30-34} }

Conference Papers

1. 

   PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features

   Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), 2019.

   Abstract Links BibTeX Project Best video award

   Eyewear devices, such as augmented reality displays, increasingly integrate eye tracking but the first-person camera required to map a user’s gaze to the visual scene can pose a significant threat to user and bystander privacy. We present PrivacEye, a method to detect privacy-sensitive everyday situations and automatically enable and disable the eye tracker’s first-person camera using a mechanical shutter. To close the shutter in privacy-sensitive situations, the method uses a deep representation of the first-person video combined with rich features that encode users’ eye movements. To open the shutter without visual input, PrivacEye detects changes in users’ eye movements alone to gauge changes in the "privacy level" of the current situation. We evaluate our method on a first-person video dataset recorded in daily life situations of 17 participants, annotated by themselves for privacy sensitivity, and show that our method is effective in preserving privacy in this challenging setting.

   doi:10.1145/3314111.3319913

   steil19_etra.pdf

   @inproceedings{steil19_etra, title = {PrivacEye: Privacy-Preserving Head-Mounted Eye Tracking Using Egocentric Scene Image and Eye Movement Features}, author = {Steil, Julian and Koelle, Marion and Heuten, Wilko and Boll, Susanne and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319913} }
2. 

   KnuckleTouch: Enabling Knuckle Gestures on Capacitive Touchscreens using Deep Learning

   Robin Schweigert, Jan Leusmann, Simon Hagenmayer, Maximilian Weiß, Huy Viet Le, Sven Mayer, Andreas Bulling

   Proc. Mensch und Computer, 2019.

   Abstract Links BibTeX Project

   While mobile devices have become essential for social communication and have paved the way for work on the go, their interactive capabilities are still limited to simple touch input. A promising enhancement for touch interaction is knuckle input but recognizing knuckle gestures robustly and accurately remains challenging. We present a method to differentiate between 17 finger and knuckle gestures based on a long short-term memory (LSTM) machine learning model. Furthermore, we introduce an open source approach that is ready-to-deploy on commodity touch-based devices. The model was trained on a new dataset that we collected in a mobile interaction study with 18 participants. We show that our method can achieve an accuracy of 86.8% on recognizing one of the 17 gestures and an accuracy of 94.6% to differentiate between finger and knuckle. In our evaluation study, we validate our models and found that the LSTM gestures recognizing archived an accuracy of 88.6%. We show that KnuckleTouch can be used to improve the input expressiveness and to provide shortcuts to frequently used functions.

   doi:10.1145/3340764.3340767

   schweigert19_muc.pdf

   @inproceedings{schweigert19_muc, title = {KnuckleTouch: Enabling Knuckle Gestures on Capacitive Touchscreens using Deep Learning}, author = {Schweigert, Robin and Leusmann, Jan and Hagenmayer, Simon and Weiß, Maximilian and Le, Huy Viet and Mayer, Sven and Bulling, Andreas}, year = {2019}, booktitle = {Proc. Mensch und Computer}, doi = {10.1145/3340764.3340767} }
3. 

   Emergent Leadership Detection Across Datasets

   Philipp Müller, Andreas Bulling

   Proc. International Conference on Multimodal Interaction (ICMI), 2019.

   Abstract Links BibTeX Project

   Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on single datasets – an unrealistic assumption for real-world applications in which systems are required to also work in settings unseen at training time. It therefore remains unclear whether current methods for emergent leadership detection generalise to similar but new settings and to which extent. To overcome this limitation, we are the first to study a cross-dataset evaluation setting for the emergent leadership detection task. We provide evaluations for within- and cross-dataset prediction using two current datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the robustness of commonly used feature channels and online prediction in the cross-dataset setting. Our evaluations show that using pose and eye contact based features, cross-dataset prediction is possible with an accuracy of 0.68, as such providing another important piece of the puzzle towards real-world emergent leadership detection.

   doi:10.1145/3340555.3353721

   mueller19_icmi.pdf

   @inproceedings{mueller19_icmi, title = {Emergent Leadership Detection Across Datasets}, author = {M{\"{u}}ller, Philipp and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Conference on Multimodal Interaction (ICMI)}, doi = {10.1145/3340555.3353721} }
4. 

   Reducing Calibration Drift in Mobile Eye Trackers by Exploiting Mobile Phone Usage

   Philipp Müller, Daniel Buschek, Michael Xuelin Huang, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), 2019.

   Abstract Links BibTeX Project

   Automatic saliency-based recalibration is promising for addressing calibration drift in mobile eye trackers but existing bottom-up saliency methods neglect user’s goal-directed visual attention in natural behaviour. By inspecting real-life recordings of egocentric eye tracker cameras, we reveal that users are likely to look at their phones once these appear in view. We propose two novel automatic recalibration methods that exploit mobile phone usage: The first builds saliency maps using the phone location in the egocentric view to identify likely gaze locations. The second uses the occurrence of touch events to recalibrate the eye tracker, thereby enabling privacy-preserving recalibration. Through in-depth evaluations on a recent mobile eye tracking dataset (N=17, 65 hours) we show that our approaches outperform a state-of-the-art saliency approach for the automatic recalibration task. As such, our approach improves mobile eye tracking and gaze-based interaction, particularly for long-term use.

   doi:10.1145/3314111.3319918

   mueller19_etra.pdf

   @inproceedings{mueller19_etra, title = {Reducing Calibration Drift in Mobile Eye Trackers by Exploiting Mobile Phone Usage}, author = {M{\"{u}}ller, Philipp and Buschek, Daniel and Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319918} }
5. 

   Moment-to-Moment Detection of Internal Thought during Video Viewing from Eye Vergence Behavior

   Michael Xuelin Huang, Jiajia Li, Grace Ngai, Hong Va Leong, Andreas Bulling

   Proc. ACM Multimedia (MM), 2019.

   Abstract Links BibTeX Project

   Internal thought refers to the process of directing attention away from a primary visual task to internal cognitive processing. Internal thought is a pervasive mental activity and closely related to primary task performance. As such, automatic detection of internal thought has significant potential for user modelling in intelligent interfaces, particularly for e-learning applications. Despite the close link between the eyes and the human mind, only a few studies have investigated vergence behaviour during internal thought and none has studied moment-to-moment detection of internal thought from gaze. While prior studies relied on long-term data analysis and required a large number of gaze characteristics, we describe a novel method that is computationally light-weight and that only requires eye vergence information that is readily available from binocular eye trackers. We further propose a novel paradigm to obtain ground truth internal thought annotations that exploits human blur perception. We evaluate our method for three increasingly challenging detection tasks: (1) during a controlled math-solving task, (2) during natural viewing of lecture videos, and (3) during daily activities, such as coding, browsing, and reading. Results from these evaluations demonstrate the performance and robustness of vergence-based detection of internal thought and, as such, open up new directions for research on interfaces that adapt to shifts of mental attention.

   doi:10.1145/3343031.3350573

   huang19_mm.pdf

   @inproceedings{huang19_mm, title = {Moment-to-Moment Detection of Internal Thought during Video Viewing from Eye Vergence Behavior}, author = {Huang, Michael Xuelin and Li, Jiajia and Ngai, Grace and Leong, Hong Va and Bulling, Andreas}, booktitle = {Proc. ACM Multimedia (MM)}, year = {2019}, doi = {10.1145/3343031.3350573} }
6. 

   SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements

   Michael Xuelin Huang, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), 2019.

   Abstract Links BibTeX Project

   Recent methods to automatically calibrate stationary eye trackers were shown to effectively reduce inherent calibration distortion. However, these methods require additional information, such as mouse clicks or on-screen content. We propose the first method that only requires users’ eye movements to reduce calibration distortion in the background while users naturally look at an interface. Our method exploits that calibration distortion makes straight saccade trajectories appear curved between the saccadic start and end points. We show that this curving effect is systematic and the result of a distorted gaze projection plane. To mitigate calibration distortion, our method undistorts this plane by straightening saccade trajectories using image warping. We show that this approach improves over the common six-point calibration and is promising for reducing distortion. As such, it provides a non-intrusive solution to alleviating accuracy decrease of eye tracker during long-term use.

   doi:0.1145/3317956.3321553

   huang19_etra.pdf

   @inproceedings{huang19_etra, title = {SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements}, author = {Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {0.1145/3317956.3321553} }
7. 

   A Design Space for Gaze Interaction on Head-mounted Displays

   Teresa Hirzle, Jan Gugenheimer, Florian Geiselhart, Andreas Bulling, Enrico Rukzio

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019.

   Abstract Links BibTeX Project

   Augmented and virtual reality (AR/VR) has entered the mass market and, with it, will soon eye tracking as a core technology for next generation head-mounted displays (HMDs). In contrast to existing gaze interfaces, the 3D nature of AR and VR requires estimating a user’s gaze in 3D. While first applications, such as foveated rendering, hint at the compelling potential of combining HMDs and gaze, a systematic analysis is missing. To fill this gap, we present the first design space for gaze interaction on HMDs. Our design space covers human depth perception and technical requirements in two dimensions aiming to identify challenges and opportunities for interaction design. As such, our design space provides a comprehensive overview and serves as an important guideline for researchers and practitioners working on gaze interaction on HMDs. We further demonstrate how our design space is used in practice by presenting two interactive applications: EyeHealth and XRay-Vision.

   doi:10.1145/3290605.3300855

   hirzle19_chi.pdf

   @inproceedings{hirzle19_chi, author = {Hirzle, Teresa and Gugenheimer, Jan and Geiselhart, Florian and Bulling, Andreas and Rukzio, Enrico}, title = {A Design Space for Gaze Interaction on Head-mounted Displays}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2019}, doi = {10.1145/3290605.3300855} }
8. 

   Can Privacy-Aware Lifelogs Alter Our Memories?

   Passant Elagroudy, Florian Mathis, Andreas Bulling, Mohamed Khamis, Diana Irmscher, Albrecht Schmidt

   Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019.

   Abstract Links BibTeX Project

   The abundance of automatically-triggered lifelogging cameras is a privacy threat to bystanders. Countering this by deleting photos limits relevant memory cues and the informative content of lifelogs. An alternative is to obfuscate bystanders, but it is not clear how this impacts the lifelogger’s recall of memories. We report on a study in which we compare viewing 1) unaltered photos, 2) photos with blurred people, and 3) a subset of the photos after deleting private ones, on memory recall. Findings show that obfuscated content helps users recall a lot of content, but it also results in recalling less accurate details, which can sometimes mislead the user. Our work informs the design of privacy- aware lifelogging systems that maximizes recall and steers discussion about ubiquitous technologies that could alter human memories.

   doi:10.1145/3290607.3313052

   elagroudy19_chi.pdf

   @inproceedings{elagroudy19_chi, author = {Elagroudy, Passant and Mathis, Florian and Bulling, Andreas and Khamis, Mohamed and Irmscher, Diana and Schmidt, Albrecht}, title = {Can Privacy-Aware Lifelogs Alter Our Memories?}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2019}, doi = {10.1145/3290607.3313052} }
9. 

   A fast approach to refraction-aware 3D eye-model fitting and gaze prediction

   Kai Dierkes, Moritz Kassner, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), 2019.

   Abstract Links BibTeX Project

   By temporally integrating information about pupil contours extracted from eye images, model-based methods for glint-free gaze estimation can mitigate pupil detection noise. However, current approaches require time-consuming iterative solving of a nonlinear minimization problem to estimate key parameters, such as eyeball position. Based on the method presented by [Swirski and Dodgson 2013], we propose a novel approach to glint-free 3D eye-model fitting and gaze prediction using a single near-eye camera. By recasting model optimization as a least-squares intersection of lines, we make it amenable to a fast non-iterative solution. We further present a method for estimating deterministic refraction-correction functions from synthetic eye images and validate them on both synthetic and real eye images. We demonstrate the robustness of our method in the presence of pupil detection noise and show the benefit of temporal integration of pupil contour information on eyeball position and gaze estimation accuracy.

   doi:10.1145/3314111.3319819

   dierkes19_etra.pdf

   @inproceedings{dierkes19_etra, title = {A fast approach to refraction-aware 3D eye-model fitting and gaze prediction}, author = {Dierkes, Kai and Kassner, Moritz and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319819} }
10. 

    Towards High-Frequency SSVEP-Based Target Discrimination with an Extended Alphanumeric Keyboard

    Sahar Abdelnabi, Michael Xuelin Huang, Andreas Bulling

    Proc. IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2019.

    Abstract Links BibTeX Project

    Despite significant advances in using Steady-State Visually Evoked Potentials (SSVEP) for on-screen target discrimination, existing methods either require intrusive, low- frequency visual stimulation or only support a small number of targets. We propose SSVEPNet: a convolutional long short-term memory (LSTM) recurrent neural network for high-frequency stimulation (≥30Hz) using a large number of visual targets. We evaluate our method for discriminating between 43 targets on an extended alphanumeric virtual keyboard and compare three different frequency assignment strategies. Our experimental results show that SSVEPNet significantly outperforms state-of-the-art correlation-based methods and convolutional neural networks. As such, our work opens up an exciting new direction of research towards a new class of unobtrusive and highly expressive SSVEP-based interfaces for text entry and beyond.

    abdelnabi19_smc.pdf

    @inproceedings{abdelnabi19_smc, author = {Abdelnabi, Sahar and Huang, Michael Xuelin and Bulling, Andreas}, title = {Towards High-Frequency SSVEP-Based Target Discrimination with an Extended Alphanumeric Keyboard}, booktitle = {Proc. IEEE International Conference on Systems, Man, and Cybernetics (SMC)}, year = {2019} }
11. 

    Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications

    Xucong Zhang, Yusuke Sugano, Andreas Bulling

    Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019.

    Abstract Links BibTeX Project

    Appearance-based gaze estimation methods that only require an off-the-shelf camera have significantly improved but they are still not yet widely used in the human-computer interaction (HCI) community. This is partly because it remains unclear how they perform compared to model-based approaches as well as dominant, special-purpose eye tracking equipment. To address this limitation, we evaluate the performance of state-of-the-art appearance-based gaze estimation for interaction scenarios with and without personal calibration, indoors and outdoors, for different sensing distances, as well as for users with and without glasses. We discuss the obtained findings and their implications for the most important gaze-based applications, namely explicit eye input, attentive user interfaces, gaze-based user modelling, and passive eye monitoring. To democratise the use of appearance-based gaze estimation and interaction in HCI, we finally present OpenGaze (www.opengaze.org), the first software toolkit for appearance-based gaze estimation and interaction.

    doi:10.1145/3290605.3300646

    zhang19_chi.pdf

    http://www.opengaze.org/

    @inproceedings{zhang19_chi, author = {Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, title = {Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2019}, doi = {10.1145/3290605.3300646}, code = {http://www.opengaze.org/} }
12. 

    Privacy-Aware Eye Tracking Using Differential Privacy

    Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), 2019.

    Abstract Links BibTeX Project Best paper award

    With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users’ privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.

    doi:10.1145/3314111.3319915

    steil19_etra_2.pdf

    @inproceedings{steil19_etra_2, title = {Privacy-Aware Eye Tracking Using Differential Privacy}, author = {Steil, Julian and Hagestedt, Inken and Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, doi = {10.1145/3314111.3319915} }
13. 

    Predicting Gaze Patterns: Text Saliency for Integration into Machine Learning Tasks

    Ekta Sood

    Proc. International Workshop on Computational Cognition (ComCo), 2019.

    Links BibTeX Project Best poster award

    sood19_comco.pdf

    @inproceedings{sood19_comco, author = {Sood, Ekta}, title = {Predicting Gaze Patterns: Text Saliency for Integration into Machine Learning Tasks}, year = {2019}, booktitle = {Proc. International Workshop on Computational Cognition (ComCo)} }

Book Chapters

1. 

   Pervasive Eye Tracking for Real-World Consumer Behavior Analysis

   Andreas Bulling, Michel Wedel

   Michael Schulte-Mecklenbeck, Anton Kühberger (Eds.): A Handbook of Process Tracing Methods for Decision Research: A Critical Review and User’s Guide, Taylor & Francis, 2019.

   Abstract Links BibTeX Project

   Eye tracking is the computational process of measuring the absolute point of gaze and/or the relative movement of the eyes over time using sensing systems placed in the environment (remote eye tracking) or worn on the head (mobile eye tracking). Eye tracking has a long history as a tool in psychology, human behaviour, and human-computer interaction research, and has also found its way into many commercial applications, such as marketing, web usability, virtual reality, or automotive engineering. Recent advances in mobile eye tracking as well as remote eye tracking using RGB cameras readily integrated into handheld devices and ambient displays pave the way for a whole new class of everyday eye tracking systems that allow researchers and practitioners to better understand and analyse gaze information in real-world settings. In this chapter, we first provide a history of eye tracking as both a measurement tool and research topic. Afterwards, we discuss the considerable potential but also remaining technical challenges for leveraging everyday eye tracking for real-world consumer behaviour analysis and decision making in retail.

   bulling19_tf.pdf

   @inbook{bulling19_tf, author = {Bulling, Andreas and Wedel, Michel}, title = {Pervasive Eye Tracking for Real-World Consumer Behavior Analysis}, booktitle = {A Handbook of Process Tracing Methods for Decision Research: A Critical Review and User's Guide}, year = {2019}, editor = {Schulte-Mecklenbeck, Michael and K{\"{u}}hberger, Anton}, publisher = {Taylor \& Francis} }

Technical Reports

1. 

   Emergent Leadership Detection Across Datasets

   Philipp Müller, Andreas Bulling

   arXiv:1905.02058, 2019.

   Abstract Links BibTeX Project

   Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on single datasets – an unrealistic assumption for real-world applications in which systems are required to also work in settings unseen at training time. It therefore remains unclear whether current methods for emergent leadership detection generalise to similar but new settings and to which extent. To overcome this limitation, we are the first to study a cross-dataset evaluation setting for the emergent leadership detection task. We provide evaluations for within- and cross-dataset prediction using two current datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the robustness of commonly used feature channels (visual focus of attention, body pose, facial action units, speaking activity) and online prediction in the cross-dataset setting. Our evaluations show that using pose and eye contact based features, cross-dataset prediction is possible with an accuracy of 0.68, as such providing another important piece of the puzzle towards emergent leadership detection in the real world.

   mueller19_arxiv.pdf

   https://arxiv.org/abs/1905.02058

   @techreport{mueller19_arxiv, title = {Emergent Leadership Detection Across Datasets}, author = {M{\"{u}}ller, Philipp and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1905.02058} }
2. 

   SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements

   Michael Xuelin Huang, Andreas Bulling

   arXiv:1903.04047, 2019.

   Abstract Links BibTeX Project

   Recent methods to automatically calibrate stationary eye trackers were shown to effectively reduce inherent calibration distortion. However, these methods require additional information, such as mouse clicks or on-screen content. We propose the first method that only requires users’ eye movements to reduce calibration distortion in the background while users naturally look at an interface. Our method exploits that calibration distortion makes straight saccade trajectories appear curved between the saccadic start and end points. We show that this curving effect is systematic and the result of a distorted gaze projection plane. To mitigate calibration distortion, our method undistorts this plane by straightening saccade trajectories using image warping. We show that this approach improves over the common six-point calibration and is promising for reducing distortion. As such, it provides a non-intrusive solution to alleviating accuracy decrease of eye tracker during long-term use.

   huang19_arxiv_2.pdf

   https://arxiv.org/abs/1903.04047

   @techreport{huang19_arxiv_2, title = {SacCalib: Reducing Calibration Distortion for Stationary Eye Trackers Using Saccadic Eye Movements}, author = {Huang, Michael Xuelin and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1903.04047} }
3. 

   Moment-to-Moment Detection of Internal Thought from Eye Vergence Behaviour

   Michael Xuelin Huang, Jiajia Li, Grace Ngai, Hong Va Leong, Andreas Bulling

   arXiv:1901.06572, 2019.

   Abstract Links BibTeX Project

   Internal thought refers to the process of directing attention away from a primary visual task to internal cognitive processing. Internal thought is a pervasive mental activity and closely related to primary task performance. As such, automatic detection of internal thought has significant potential for user modelling in intelligent interfaces, particularly for e-learning applications. Despite the close link between the eyes and the human mind, only a few studies have investigated vergence behaviour during internal thought and none has studied moment-to-moment detection of internal thought from gaze. While prior studies relied on long-term data analysis and required a large number of gaze characteristics, we describe a novel method that is computationally light-weight and that only requires eye vergence information that is readily available from binocular eye trackers. We further propose a novel paradigm to obtain ground truth internal thought annotations that exploits human blur perception. We evaluate our method for three increasingly challenging detection tasks: (1) during a controlled math-solving task, (2) during natural viewing of lecture videos, and (3) during daily activities, such as coding, browsing, and reading. Results from these evaluations demonstrate the performance and robustness of vergence-based detection of internal thought and, as such, open up new directions for research on interfaces that adapt to shifts of mental attention.

   huang19_arxiv.pdf

   https://arxiv.org/abs/1901.06572

   @techreport{huang19_arxiv, title = {Moment-to-Moment Detection of Internal Thought from Eye Vergence Behaviour}, author = {Huang, Michael Xuelin and Li, Jiajia and Ngai, Grace and Leong, Hong Va and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1901.06572} }
4. 

   How far are we from quantifying visual attention in mobile HCI?

   Mihai Bâce, Sander Staal, Andreas Bulling

   arXiv:1907.11106, 2019.

   Abstract Links BibTeX Project

   With an ever-increasing number of mobile devices competing for our attention, quantifying when, how often, or for how long users visually attend to their devices has emerged as a core challenge in mobile human-computer interaction. Encouraged by recent advances in automatic eye contact detection using machine learning and device-integrated cameras, we provide a fundamental investigation into the feasibility of quantifying visual attention during everyday mobile interactions. We identify core challenges and sources of errors associated with sensing attention on mobile devices in the wild, including the impact of face and eye visibility, the importance of robust head pose estimation, and the need for accurate gaze estimation. Based on this analysis, we propose future research directions and discuss how eye contact detection represents the foundation for exciting new applications towards next-generation pervasive attentive user interfaces.

   bace19_arxiv_2.pdf

   https://arxiv.org/abs/1907.11106

   @techreport{bace19_arxiv_2, title = {How far are we from quantifying visual attention in mobile HCI?}, author = {Bâce, Mihai and Staal, Sander and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1907.11106} }
5. 

   Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions

   Mihai Bâce, Sander Staal, Andreas Bulling

   arXiv:1907.11115, 2019.

   Abstract Links BibTeX Project

   Quantification of human attention is key to several tasks in mobile human-computer interaction (HCI), such as predicting user interruptibility, estimating noticeability of user interface content, or measuring user engagement. Previous works to study mobile attentive behaviour required special-purpose eye tracking equipment or constrained users’ mobility. We propose a novel method to sense and analyse visual attention on mobile devices during everyday interactions. We demonstrate the capabilities of our method on the sample task of eye contact detection that has recently attracted increasing research interest in mobile HCI. Our method builds on a state-of-the-art method for unsupervised eye contact detection and extends it to address challenges specific to mobile interactive scenarios. Through evaluation on two current datasets, we demonstrate significant performance improvements for eye contact detection across mobile devices, users, or environmental conditions. Moreover, we discuss how our method enables the calculation of additional attention metrics that, for the first time, enable researchers from different domains to study and quantify attention allocation during mobile interactions in the wild.

   bace19_arxiv.pdf

   https://arxiv.org/abs/1907.11115

   @techreport{bace19_arxiv, title = {Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions}, author = {Bâce, Mihai and Staal, Sander and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1907.11115} }
6. 

   Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications

   Xucong Zhang, Yusuke Sugano, Andreas Bulling

   arXiv:1901.10906, 2019.

   Abstract Links BibTeX Project

   Appearance-based gaze estimation methods that only require an off-the-shelf camera have significantly improved but they are still not yet widely used in the human-computer interaction (HCI) community. This is partly because it remains unclear how they perform compared to model-based approaches as well as dominant, special-purpose eye tracking equipment. To address this limitation, we evaluate the performance of state-of-the-art appearance-based gaze estimation for interaction scenarios with and without personal calibration, indoors and outdoors, for different sensing distances, as well as for users with and without glasses. We discuss the obtained findings and their implications for the most important gaze-based applications, namely explicit eye input, attentive user interfaces, gaze-based user modelling, and passive eye monitoring. To democratise the use of appearance-based gaze estimation and interaction in HCI, we finally present OpenGaze (www.opengaze.org), the first software toolkit for appearance-based gaze estimation and interaction.

   zhang19_arxiv.pdf

   https://arxiv.org/abs/1901.10906

   @techreport{zhang19_arxiv, title = {Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications}, author = {Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, year = {2019}, code = {https://arxiv.org/abs/1901.10906} }

2018

Journal Articles

1. 

   Eye movements during everyday behavior predict personality traits

   Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, Andreas Bulling

   Frontiers in Human Neuroscience, 12, pp. 105:1-105:8, 2018.

   Abstract Links BibTeX Project

   Besides allowing us to perceive our surroundings, eye movements are also a window into our mind and a rich source of information on who we are, how we feel, and what we do. Here we show that eye movements during an everyday task predict aspects of our personality. We tracked eye movements of 42 participants while they ran an errand on a university campus and subsequently assessed their personality traits using well-established questionnaires. Using a state-of-the-art machine learning method and a rich set of features encoding different eye movement characteristics, we were able to reliably predict four of the Big Five personality traits (neuroticism, extraversion, agreeableness, conscientiousness) as well as perceptual curiosity only from eye movements. Further analysis revealed new relations between previously neglected eye movement characteristics and personality. Our findings demonstrate a considerable influence of personality on everyday eye movement control, thereby complementing earlier studies in laboratory settings. Improving automatic recognition and interpretation of human social signals is an important endeavor, enabling innovative design of human–computer systems capable of sensing spontaneous natural user behavior to facilitate efficient interaction and personalization.

   doi:10.3389/fnhum.2018.00105

   hoppe18_fhns.pdf

   @article{hoppe18_fhns, title = {Eye movements during everyday behavior predict personality traits}, author = {Hoppe, Sabrina and Loetscher, Tobias and Morey, Stephanie and Bulling, Andreas}, doi = {10.3389/fnhum.2018.00105}, volume = {12}, pages = {105:1-105:8}, year = {2018}, journal = {Frontiers in Human Neuroscience} }
2. 

   CueAuth: Comparing Touch, Mid-Air Gestures, and Gaze for Cue-based Authentication on Situated Displays

   Mohamed Khamis, Ludwig Trotter, Ville Mäkelä, Emanuel Zezschwitz, Jens Le, Andreas Bulling, Florian Alt

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2(7), pp. 174:1-174:22, 2018.

   Abstract Links BibTeX Project

   Secure authentication on situated displays s (e.g., to access sensitive information or to make purchases) is becoming increasingly important. A promising approach are authentication schemes that employ cues that users respond to while authenticating; these schemes overwhelm observers by requiring them to observe the cue itself as well as users’ response to the cue. Although previous work proposed a variety of modalities, such as gaze and mid-air gestures, to further improve security, an understanding of how they compare with regard to usability and security is still missing as of today. In this paper, we compare modalities for cue-based authentication on situated displays. We provide the first comparison between touch, mid-air gestures, and calibration-free gaze using a state-of-the-art authentication concept. In two user studies (N=37) we found that the choice of touch or gaze presents a clear trade-off between usability and security. For example, while gaze input is more secure, it is also more demanding and requires longer authentication times. Mid-air gestures are slightly slower and more secure than touch but users hesitate using them in public. We conclude with design implications for authentication using touch, mid-air gestures, and gaze and discuss how the choice of modality creates opportunities and challenges for improved authentication in public.

   doi:10.1145/3287052

   khamis18_imwut.pdf

   @article{khamis18_imwut, title = {CueAuth: Comparing Touch, Mid-Air Gestures, and Gaze for Cue-based Authentication on Situated Displays}, author = {Khamis, Mohamed and Trotter, Ludwig and Mäkelä, Ville and von Zezschwitz, Emanuel and Le, Jens and Bulling, Andreas and Alt, Florian}, year = {2018}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, volume = {2}, number = {7}, pages = {174:1-174:22}, doi = {10.1145/3287052} }
3. 

   GazeDirector: Fully Articulated Eye Gaze Redirection in Video

   Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling

   Computer Graphics Forum (CGF), 37(2), pp. 217-225, 2018.

   Abstract Links BibTeX Project Best paper honourable mention award

   We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting. Our method first tracks the eyes by fitting a multi-part eye region model to video frames using analysis-by-synthesis, thereby recovering eye region shape, texture, pose, and gaze simultaneously. It then redirects gaze by 1) warping the eyelids from the original image using a model-derived flow field, and 2) rendering and compositing synthesized 3D eyeballs onto the output image in a photorealistic manner. GazeDirector allows us to change where people are looking without person-specific training data, and with full articulation, i.e. we can precisely specify new gaze directions in 3D. Quantitatively, we evaluate both model-fitting and gaze synthesis, with experiments for gaze estimation and redirection on the Columbia gaze dataset. Qualitatively, we compare GazeDirector against recent work on gaze redirection, showing better results especially for large redirection angles. Finally, we demonstrate gaze redirection on YouTube videos by introducing new 3D gaze targets and by manipulating visual behavior.

   doi:10.1111/cgf.13355

   wood18_cgf.pdf

   @article{wood18_cgf, title = {GazeDirector: Fully Articulated Eye Gaze Redirection in Video}, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Morency, Louis-Philippe and Robinson, Peter and Bulling, Andreas}, year = {2018}, journal = {Computer Graphics Forum (CGF)}, volume = {37}, number = {2}, pages = {217-225}, doi = {10.1111/cgf.13355} }

Conference Papers

1. 

   Hidden Pursuits: Evaluating Gaze-selection via Pursuits when the Stimulus Trajectory is Partially Hidden

   Thomas Mattusch, Mahsa Mirzamohammad, Mohamed Khamis, Andreas Bulling, Florian Alt

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 27:1-27:5, 2018.

   Abstract Links BibTeX Project

   The idea behind gaze interaction using Pursuits is to leverage the human’s smooth pursuit eye movements performed when following moving targets. However, humans can also anticipate where a moving target would reappear if it temporarily hides from their view. In this work, we investigate how well users can select targets using Pursuits in cases where the target’s trajectory is partially invisible (HiddenPursuits): e.g., can users select a moving target that temporarily hides behind another object? Although HiddenPursuits was not studied in the context of interaction before, understanding how well users can perform HiddenPursuits presents numerous opportunities, particularly for small interfaces where a target’s trajectory can cover area outside of the screen. We found that users can still select targets quickly via Pursuits even if their trajectory is up to 50% hidden, and at the expense of longer selection times when the hidden portion is larger. We discuss how gaze-based interfaces can leverage HiddenPursuits for an improved user experience.

   doi:10.1145/3204493.3204569

   mattusch18_etra.pdf

   @inproceedings{mattusch18_etra, author = {Mattusch, Thomas and Mirzamohammad, Mahsa and Khamis, Mohamed and Bulling, Andreas and Alt, Florian}, title = {Hidden Pursuits: Evaluating Gaze-selection via Pursuits when the Stimulus Trajectory is Partially Hidden}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {27:1-27:5}, doi = {10.1145/3204493.3204569} }
2. 

   Error-Aware Gaze-Based Interfaces for Robust Mobile Gaze Interaction

   Michael Barz, Florian Daiber, Daniel Sonntag, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 24:1-24:10, 2018.

   Abstract Links BibTeX Project Best paper award

   Gaze estimation error is unavoidable in head-mounted eye trackers and can severely hamper usability and performance of mobile gaze-based interfaces given that the error varies constantly for different interaction positions. In this work, we explore error-aware gaze-based interfaces that estimate and adapt to gaze estimation error on-the-fly. We implement a sample error-aware user interface for gaze-based selection and different error compensation methods: a naïve approach that increases component size directly proportional to the absolute error, a recent model by Feit et al. (CHI’17) that is based on the 2-dimensional error distribution, and a novel predictive model that shifts gaze by a directional error estimate. We evaluate these models in a 12-participant user study and show that our predictive model outperforms the others significantly in terms of selection rate, particularly for small gaze targets. These results underline both the feasibility and potential of next generation error-aware gaze-based user interfaces.

   doi:10.1145/3204493.3204536

   barz18_etra.pdf

   @inproceedings{barz18_etra, author = {Barz, Michael and Daiber, Florian and Sonntag, Daniel and Bulling, Andreas}, title = {Error-Aware Gaze-Based Interfaces for Robust Mobile Gaze Interaction}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {24:1-24:10}, doi = {10.1145/3204493.3204536} }
3. 

   A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction

   Kai Dierkes, Moritz Kassner, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 9:1-9:9, 2018.

   Abstract Links BibTeX Project

   Model-based methods for glint-free gaze estimation typically infer eye pose using pupil contours extracted from eye images. Existing methods, however, either ignore or require complex hardware setups to deal with refraction effects occurring at the corneal interfaces. In this work we provide a detailed analysis of the effects of refraction in glint-free gaze estimation using a single near-eye camera, based on the method presented by [Swirski et al. 2013]. We demonstrate systematic deviations in inferred eyeball positions and gaze directions with respect to synthetic ground-truth data and show that ignoring corneal refraction can result in angular errors of several degrees. Furthermore, we quantify gaze direction dependent errors in pupil radius estimates. We propose a novel approach to account for corneal refraction in 3D eye model fitting and by analyzing synthetic and real images show that our new method successfully captures refraction effects and helps to overcome the shortcomings of the state of the art approach.

   doi:10.1145/3204493.3204525

   dierkes18_etra.pdf

   @inproceedings{dierkes18_etra, author = {Dierkes, Kai and Kassner, Moritz and Bulling, Andreas}, title = {A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {9:1-9:9}, doi = {10.1145/3204493.3204525} }
4. 

   Towards a Symbiotic Human-Machine Depth Sensor: Exploring 3D Gaze for Object Reconstruction

   Teresa Hirzle, Jan Gugenheimer, Florian Geiselhart, Andreas Bulling, Enrico Rukzio

   Adj. Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 114-116, 2018.

   Abstract Links BibTeX Project

   Eye tracking is expected to become an integral part of future augmented reality (AR) head-mounted displays (HMDs) given that it can easily be integrated into existing hardware and provides a versatile interaction modality. To augment objects in the real world, AR HMDs require a three-dimensional understanding of the scene, which is currently solved using depth cameras. In this work we aim to explore how 3D gaze data can be used to enhance scene understanding for AR HMDs by envisioning a symbiotic human-machine depth camera, fusing depth data with 3D gaze information. We present a first proof of concept, exploring to what extend we are able to recognise what a user is looking at by plotting 3D gaze data. To measure 3D gaze, we implemented a vergence-based algorithm and built an eye tracking setup consisting of a Pupil Labs headset and an OptiTrack motion capture system, allowing us to measure 3D gaze inside a 50x50x50 cm volume. We show first 3D gaze plots of "gazed-at" objects and describe our vision of a symbiotic human-machine depth camera that combines a depth camera and human 3D gaze information.

   doi:10.1145/3266037.3266119

   hirzle18_uist.pdf

   @inproceedings{hirzle18_uist, title = {Towards a Symbiotic Human-Machine Depth Sensor: Exploring 3D Gaze for Object Reconstruction}, author = {Hirzle, Teresa and Gugenheimer, Jan and Geiselhart, Florian and Bulling, Andreas and Rukzio, Enrico}, year = {2018}, pages = {114-116}, doi = {10.1145/3266037.3266119}, booktitle = {Adj. Proc. ACM Symposium on User Interface Software and Technology (UIST)} }
5. 

   Understanding Face and Eye Visibility in Front-Facing Cameras of Smartphones used in the Wild

   Mohamed Khamis, Anita Baier, Niels Henze, Florian Alt, Andreas Bulling

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 280:1-280:12, 2018.

   Abstract Links BibTeX Project

   Commodity mobile devices are now equipped with high-resolution front-facing cameras, paving the way for applications in biometrics, facial expression analysis, or gaze interaction. However, it is unknown how often users hold devices in a way that allows capturing their face or eyes, and how this impacts detection accuracy. We collected 25,726 in-the-wild photos taken from the front-facing camera of smartphones and associated application usage logs. We found that the full face is visible about 29% of the time, and that in most cases the face is only partially visible. We further identified an influence of users’ current activity; for example, when watching videos, the eyes but not the entire face are visible 75% of the time in our dataset. We found that state-of-the-art face detection algorithms perform poorly against photos taken from front-facing cameras. We discuss how these findings impact mobile applications that leverage face and eye detection, and derive practical implications to address state-of-the art’s limitations.

   doi:10.1145/3173574.3173854

   khamis18_chi.pdf

   @inproceedings{khamis18_chi, title = {Understanding Face and Eye Visibility in Front-Facing Cameras of Smartphones used in the Wild}, author = {Khamis, Mohamed and Baier, Anita and Henze, Niels and Alt, Florian and Bulling, Andreas}, year = {2018}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/3173574.3173854}, pages = {280:1-280:12} }
6. 

   Which one is me? Identifying Oneself on Public Displays

   Mohamed Khamis, Christian Becker, Andreas Bulling, Florian Alt

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 287:1-287:12, 2018.

   Abstract Links BibTeX Project Best paper honourable mention award

   While user representations are extensively used on public displays, it remains unclear how well users can recognize their own representation among those of surrounding users. We study the most widely used representations: abstract objects, skeletons, silhouettes and mirrors. In a prestudy (N=12), we identify five strategies that users follow to recognize themselves on public displays. In a second study (N=19), we quantify the users’ recognition time and accuracy with respect to each representation type. Our findings suggest that there is a significant effect of (1) the representation type, (2) the strategies performed by users, and (3) the combination of both on recognition time and accuracy. We discuss the suitability of each representation for different settings and provide specific recommendations as to how user representations should be applied in multi-user scenarios. These recommendations guide practitioners and researchers in selecting the representation that optimizes the most for the deployment’s requirements, and for the user strategies that are feasible in that environment.

   doi:10.1145/3173574.3173861

   khamis18_chi_2.pdf

   @inproceedings{khamis18_chi_2, title = {Which one is me? Identifying Oneself on Public Displays}, author = {Khamis, Mohamed and Becker, Christian and Bulling, Andreas and Alt, Florian}, year = {2018}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/3173574.3173861}, pages = {287:1-287:12} }
7. 

   VRPursuits: Interaction in Virtual Reality using Smooth Pursuit Eye Movements

   Mohamed Khamis, Carl Oechsner, Florian Alt, Andreas Bulling

   Proc. International Conference on Advanced Visual Interfaces (AVI), pp. 18:1-18:8, 2018.

   Abstract Links BibTeX Project

   Gaze-based interaction using smooth pursuit eye movements (Pursuits) is attractive given that it is intuitive and overcomes the Midas touch problem. At the same time, eye tracking is becoming increasingly popular for VR applications. While Pursuits was shown to be effective in several interaction contexts, it was never explored in-depth for VR before. In a user study (N=26), we investigated how parameters that are specific to VR settings influence the performance of Pursuits. We found that Pursuits is robust against different sizes of virtual 3D targets and sizes to them. However, Pursuits’ performance improves when the trajectory size is larger, particularly if the user is walking while interacting. While walking, selecting moving targets via Pursuits is generally feasible albeit less accurate than when stationary. Finally, we discuss the implications of these findings and the potential of smooth pursuits for interaction in VR by demonstrating two sample use cases: 1) gaze-based authentication in VR, and 2) a space meteors shooting game.

   doi:10.1145/3206505.3206522

   khamis18_avi.pdf

   @inproceedings{khamis18_avi, title = {VRPursuits: Interaction in Virtual Reality using Smooth Pursuit Eye Movements}, author = {Khamis, Mohamed and Oechsner, Carl and Alt, Florian and Bulling, Andreas}, year = {2018}, pages = {18:1-18:8}, booktitle = {Proc. International Conference on Advanced Visual Interfaces (AVI)}, doi = {10.1145/3206505.3206522} }
8. 

   The Past, Present, and Future of Gaze-enabled Handheld Mobile Devices: Survey and Lessons Learned

   Mohamed Khamis, Florian Alt, Andreas Bulling

   Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 38:1–38:17, 2018.

   Abstract Links BibTeX Project Best paper honourable mention award

   While first-generation mobile gaze interfaces required special-purpose hardware, recent advances in computational gaze estimation and the availability of sensor-rich and powerful devices is finally fulfilling the promise of pervasive eye tracking and eye-based interaction on off-the-shelf mobile devices. This work provides the first holistic view on the past, present, and future of eye tracking on handheld mobile devices. To this end, we discuss how research developed from building hardware prototypes, to accurate gaze estimation on unmodified smartphones and tablets. We then discuss implications by laying out 1) novel opportunities, which include pervasive advertising and conducting in-the-wild eye tracking studies on handhelds, as well as 2) new challenges that require further research, such as the visibility of the user’s eyes, lighting conditions, and privacy implications. We discuss how these developments shape MobileHCI research in the future, possibly the “next 20 years”, as the overarching theme of MobileHCI 2018 suggests.

   doi:10.1145/3229434.3229452

   khamis18_mobilehci.pdf

   @inproceedings{khamis18_mobilehci, author = {Khamis, Mohamed and Alt, Florian and Bulling, Andreas}, title = {The Past, Present, and Future of Gaze-enabled Handheld Mobile Devices: Survey and Lessons Learned}, booktitle = {Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)}, year = {2018}, doi = {10.1145/3229434.3229452}, pages = {38:1--38:17} }
9. 

   GazeDrone: Mobile Eye-Based Interaction in Public Space Without Augmenting the User

   Mohamed Khamis, Anna Kienle, Florian Alt, Andreas Bulling

   Proc. ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications (DroNet), pp. 66-71, 2018.

   Abstract Links BibTeX Project

   Gaze interaction holds a lot of promise for seamless human-computer interaction. At the same time, current wearable mobile eye trackers require user augmentation that negatively impacts natural user behavior while remote trackers require users to position themselves within a confined tracking range. We present GazeDrone, the first system that combines a camera-equipped aerial drone with a computational method to detect sidelong glances for spontaneous (calibration-free) gaze-based interaction with surrounding pervasive systems (e.g., public displays). GazeDrone does not require augmenting each user with on-body sensors and allows interaction from arbitrary positions, even while moving. We demonstrate that drone-supported gaze interaction is feasible and accurate for certain movement types. It is well-perceived by users, in particular while interacting from a fixed position as well as while moving orthogonally or diagonally to a display. We present design implications and discuss opportunities and challenges for drone-supported gaze interaction in public.

   doi:10.1145/3213526.3213539

   khamis18_dronet.pdf

   @inproceedings{khamis18_dronet, title = {GazeDrone: Mobile Eye-Based Interaction in Public Space Without Augmenting the User}, author = {Khamis, Mohamed and Kienle, Anna and Alt, Florian and Bulling, Andreas}, doi = {10.1145/3213526.3213539}, year = {2018}, booktitle = {Proc. ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications (DroNet)}, pages = {66-71} }
10. 

    A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks

    Arif Khan, Ingmar Steiner, Yusuke Sugano, Andreas Bulling, Ross Macdonald

    Proc. Language Resources and Evaluation Conference (LREC), 2018.

    Abstract Links BibTeX Project

    Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as close to the manual segmentation as possible. This corpus is an effort to capture the human segmentation behavior by recording experts performing a segmentation task. We believe that this data will enable us to highlight the important aspects of manual segmentation, which can be used in automatic segmentation to improve its accuracy.

    khan18_lrec.pdf

    @inproceedings{khan18_lrec, title = {A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks}, author = {Khan, Arif and Steiner, Ingmar and Sugano, Yusuke and Bulling, Andreas and Macdonald, Ross}, year = {2018}, booktitle = {Proc. Language Resources and Evaluation Conference (LREC)} }
11. 

    Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension

    Matthias Blohm, Glorianna Jagfeld, Ekta Sood, Xiang Yu, Ngoc Thang Vu

    Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1, 2018, pp. 108-118, 2018.

    Abstract Links BibTeX Project

    We propose a machine reading comprehension model based on the compare-aggregate framework with two-staged attention that achieves state-of-the-art results on the MovieQA question answering dataset. To investigate the limitations of our model as well as the behavioral difference between convolutional and recurrent neural networks, we generate adversarial examples to confuse the model and compare to human performance. Furthermore, we assess the generalizability of our model by analyzing its differences to human inference, drawing upon insights from cognitive science.

    doi:10.18653/v1/K18-1011

    sood18_conll.pdf

    https://arxiv.org/abs/1808.08744

    @inproceedings{sood18_conll, author = {Blohm, Matthias and Jagfeld, Glorianna and Sood, Ekta and Yu, Xiang and Vu, Ngoc Thang}, title = {Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension}, booktitle = {Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium, October 31 - November 1, 2018}, doi = {10.18653/v1/K18-1011}, pages = {108-118}, year = {2018}, crossref = {DBLP:conf/conll/2018}, url = {https://www.aclweb.org/anthology/K18-1011/}, code = {https://arxiv.org/abs/1808.08744} }
12. 

    Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior

    Philipp Müller, Michael Xuelin Huang, Andreas Bulling

    Proc. ACM International Conference on Intelligent User Interfaces (IUI), pp. 153-164, 2018.

    Abstract Links BibTeX Project

    Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants’ personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.

    doi:10.1145/3172944.3172969

    mueller18_iui.pdf

    @inproceedings{mueller18_iui, title = {Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behavior}, author = {M{\"{u}}ller, Philipp and Huang, Michael Xuelin and Bulling, Andreas}, year = {2018}, pages = {153-164}, booktitle = {Proc. ACM International Conference on Intelligent User Interfaces (IUI)}, doi = {10.1145/3172944.3172969} }
13. 

    Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour

    Philipp Müller, Michael Xuelin Huang, Xucong Zhang, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 31:1-31:10, 2018.

    Abstract Links BibTeX Project

    Eye contact is one of the most important non-verbal social cues and fundamental to human interactions. However, detecting eye contact without specialized eye tracking equipment poses significant challenges, particularly for multiple people in real-world settings. We present a novel method to robustly detect eye contact in natural three- and four-person interactions using off-the-shelf ambient cameras. Our method exploits that, during conversations, people tend to look at the person who is currently speaking. Harnessing the correlation between people’s gaze and speaking behaviour therefore allows our method to automatically acquire training data during deployment and adaptively train eye contact detectors for each target user. We empirically evaluate the performance of our method on a recent dataset of natural group interactions and demonstrate that it achieves a relative improvement over the state-of-the-art method of more than 60%, and also improves over a head pose based baseline.

    doi:10.1145/3204493.3204549

    mueller18_etra.pdf

    @inproceedings{mueller18_etra, author = {M{\"{u}}ller, Philipp and Huang, Michael Xuelin and Zhang, Xucong and Bulling, Andreas}, title = {Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {31:1-31:10}, doi = {10.1145/3204493.3204549} }
14. 

    Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings

    Seonwook Park, Xucong Zhang, Andreas Bulling, Otmar Hilliges

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 21:1-21:10, 2018.

    Abstract Links BibTeX Project Best presentation award

    Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation.

    doi:10.1145/3204493.3204545

    park18_etra.pdf

    @inproceedings{park18_etra, author = {Park, Seonwook and Zhang, Xucong and Bulling, Andreas and Hilliges, Otmar}, title = {Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {21:1-21:10}, doi = {10.1145/3204493.3204545} }
15. 

    Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets

    Julian Steil, Michael Xuelin Huang, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 23:1-23:9, 2018.

    Abstract Links BibTeX Project

    Fixations are widely analysed in human vision, gaze-based interaction, and experimental psychology research. However, robust fixation detection in mobile settings is profoundly challenging given the prevalence of user and gaze target motion. These movements feign a shift in gaze estimates in the frame of reference defined by the eye tracker’s scene camera. To address this challenge, we present a novel fixation detection method for head-mounted eye trackers. Our method exploits that, independent of user or gaze target motion, target appearance remains about the same during a fixation. It extracts image information from small regions around the current gaze position and analyses the appearance similarity of these gaze patches across video frames to detect fixations. We evaluate our method using fine-grained fixation annotations on a five-participant indoor dataset (MPIIEgoFixation) with more than 2,300 fixations in total. Our method outperforms commonly used velocity- and dispersion-based algorithms, which highlights its significant potential to analyse scene image information for eye movement detection.

    doi:10.1145/3204493.3204538

    steil18_etra.pdf

    @inproceedings{steil18_etra, author = {Steil, Julian and Huang, Michael Xuelin and Bulling, Andreas}, title = {Fixation Detection for Head-Mounted Eye Tracking Based on Visual Similarity of Gaze Targets}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {23:1-23:9}, doi = {10.1145/3204493.3204538} }
16. 

    Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

    Julian Steil, Philipp Müller, Yusuke Sugano, Andreas Bulling

    Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 1:1–1:13, 2018.

    Abstract Links BibTeX Project Best paper award

    Visual attention is highly fragmented during mobile interactions but the erratic nature of attention shifts currently limits attentive user interfaces to adapt after the fact, i.e. after shifts have already happened. We instead study attention forecasting – the challenging task of predicting users’ gaze behavior (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users’ visual scene. We demonstrate that our method can forecast bidirectional attention shifts and whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

    doi:10.1145/3229434.3229439

    steil18_mobilehci.pdf

    @inproceedings{steil18_mobilehci, author = {Steil, Julian and M{\"{u}}ller, Philipp and Sugano, Yusuke and Bulling, Andreas}, title = {Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors}, booktitle = {Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)}, year = {2018}, doi = {10.1145/3229434.3229439}, pages = {1:1--1:13} }
17. 

    Training Person-Specific Gaze Estimators from Interactions with Multiple Devices

    Xucong Zhang, Michael Xuelin Huang, Yusuke Sugano, Andreas Bulling

    Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 624:1-624:12, 2018.

    Abstract Links BibTeX Project

    Learning-based gaze estimation has significant potential to enable attentive user interfaces and gaze-based interaction on the billions of camera-equipped handheld devices and ambient displays. While training accurate person- and device-independent gaze estimators remains challenging, person-specific training is feasible but requires tedious data collection for each target device. To address these limitations, we present the first method to train person-specific gaze estimators across multiple devices. At the core of our method is a single convolutional neural network with shared feature extraction layers and device-specific branches that we train from face images and corresponding on-screen gaze locations. Detailed evaluations on a new dataset of interactions with five common devices (mobile phone, tablet, laptop, desktop computer, smart TV) and three common applications (mobile game, text editing, media center) demonstrate the significant potential of cross-device training. We further explore training with gaze locations derived from natural interactions, such as mouse or touch input.

    doi:10.1145/3173574.3174198

    zhang18_chi.pdf

    @inproceedings{zhang18_chi, title = {Training Person-Specific Gaze Estimators from Interactions with Multiple Devices}, author = {Zhang, Xucong and Huang, Michael Xuelin and Sugano, Yusuke and Bulling, Andreas}, year = {2018}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/3173574.3174198}, pages = {624:1-624:12} }
18. 

    Revisiting Data Normalization for Appearance-Based Gaze Estimation

    Xucong Zhang, Yusuke Sugano, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 12:1-12:9, 2018.

    Abstract Links BibTeX Project

    Appearance-based gaze estimation is promising for unconstrained real-world settings, but the significant variability in head pose and user-camera distance poses significant challenges for training generic gaze estimators. Data normalization was proposed to cancel out this geometric variability by mapping input images and gaze labels to a normalized space. Although used successfully in prior works, the role and importance of data normalization remains unclear. To fill this gap, we study data normalization for the first time using principled evaluations on both simulated and real data. We propose a modification to the current data normalization formulation by removing the scaling factor and show that our new formulation performs significantly better (between 9.5% and 32.7%) in the different evaluation settings. Using images synthesized from a 3D face model, we demonstrate the benefit of data normalization for the efficiency of the model training. Experiments on real-world images confirm the advantages of data normalization in terms of gaze estimation performance.

    doi:10.1145/3204493.3204548

    zhang18_etra.pdf

    @inproceedings{zhang18_etra, author = {Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, title = {Revisiting Data Normalization for Appearance-Based Gaze Estimation}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2018}, pages = {12:1-12:9}, doi = {10.1145/3204493.3204548} }

Technical Reports

1. 

   Privacy-Aware Eye Tracking Using Differential Privacy

   Julian Steil, Inken Hagestedt, Michael Xuelin Huang, Andreas Bulling

   arXiv:1812.08000, 2018.

   Abstract Links BibTeX Project

   With eye tracking being increasingly integrated into virtual and augmented reality (VR/AR) head-mounted displays, preserving users’ privacy is an ever more important, yet under-explored, topic in the eye tracking community. We report a large-scale online survey (N=124) on privacy aspects of eye tracking that provides the first comprehensive account of with whom, for which services, and to which extent users are willing to share their gaze data. Using these insights, we design a privacy-aware VR interface that uses differential privacy, which we evaluate on a new 20-participant dataset for two privacy sensitive tasks: We show that our method can prevent user re-identification and protect gender information while maintaining high performance for gaze-based document type classification. Our results highlight the privacy challenges particular to gaze data and demonstrate that differential privacy is a potential means to address them. Thus, this paper lays important foundations for future research on privacy-aware gaze interfaces.

   steil18_arxiv.pdf

   https://arxiv.org/abs/1812.08000

   @techreport{steil18_arxiv, author = {Steil, Julian and Hagestedt, Inken and Huang, Michael Xuelin and Bulling, Andreas}, title = {Privacy-Aware Eye Tracking Using Differential Privacy}, year = {2018}, code = {https://arxiv.org/abs/1812.08000} }
2. 

   PrivacEye: Privacy-Preserving First-Person Vision Using Image Features and Eye Movement Analysis

   Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, Andreas Bulling

   arXiv:1801.04457, 2018.

   Abstract Links BibTeX Project

   As first-person cameras in head-mounted displays become increasingly prevalent, so does the problem of infringing user and bystander privacy. To address this challenge, we present PrivacEye, a proof-of-concept system that detects privacysensitive everyday situations and automatically enables and disables the first-person camera using a mechanical shutter. To close the shutter, PrivacEye detects sensitive situations from first-person camera videos using an end-to-end deep-learning model. To open the shutter without visual input, PrivacEye uses a separate, smaller eye camera to detect changes in users’ eye movements to gauge changes in the "privacy level" of the current situation. We evaluate PrivacEye on a dataset of first-person videos recorded in the daily life of 17 participants that they annotated with privacy sensitivity levels. We discuss the strengths and weaknesses of our proof-of-concept system based on a quantitative technical evaluation as well as qualitative insights from semi-structured interviews.

   steil18_arxiv_2.pdf

   https://arxiv.org/abs/1801.04457

   @techreport{steil18_arxiv_2, title = {PrivacEye: Privacy-Preserving First-Person Vision Using Image Features and Eye Movement Analysis}, author = {Steil, Julian and Koelle, Marion and Heuten, Wilko and Boll, Susanne and Bulling, Andreas}, year = {2018}, code = {https://arxiv.org/abs/1801.04457} }

2017

Journal Articles

1. 

   EyePACT: Eye-Based Parallax Correction on Touch-Enabled Interactive Displays

   Mohamed Khamis, Daniel Buschek, Tobias Thieron, Florian Alt, Andreas Bulling

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1(4), pp. 146:1-146:18, 2017.

   Abstract Links BibTeX Project

   The parallax effect describes the displacement between the perceived and detected touch locations on a touch-enabled surface. Parallax is a key usability challenge for interactive displays, particularly for those that require thick layers of glass between the screen and the touch surface to protect them from vandalism. To address this challenge, we present EyePACT, a method that compensates for input error caused by parallax on public displays. Our method uses a display-mounted depth camera to detect the user’s 3D eye position in front of the display and the detected touch location to predict the perceived touch location on the surface. We evaluate our method in two user studies in terms of parallax correction performance as well as multi-user support. Our evaluations demonstrate that EyePACT (1) significantly improves accuracy even with varying gap distances between the touch surface and the display, (2) adapts to different levels of parallax by resulting in significantly larger corrections with larger gap distances, and (3) maintains a significantly large distance between two users’ fingers when interacting with the same object. Our results provide implications for the development of future touch-enabled public displays.

   doi:10.1145/3161168

   khamis17_imwut.pdf

   @article{khamis17_imwut, author = {Khamis, Mohamed and Buschek, Daniel and Thieron, Tobias and Alt, Florian and Bulling, Andreas}, title = {EyePACT: Eye-Based Parallax Correction on Touch-Enabled Interactive Displays}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, year = {2017}, volume = {1}, number = {4}, pages = {146:1-146:18}, doi = {10.1145/3161168} }
2. 

   InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation

   Marc Tonsen, Julian Steil, Yusuke Sugano, Andreas Bulling

   Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 1(3), pp. 106:1-106:21, 2017.

   Abstract Links BibTeX Project Distinguished paper award

   Analysis of everyday human gaze behaviour has significant potential for ubiquitous computing, as evidenced by a large body of work in gaze-based human-computer interaction, attentive user interfaces, and eye-based user modelling. However, current mobile eye trackers are still obtrusive, which not only makes them uncomfortable to wear and socially unacceptable in daily life, but also prevents them from being widely adopted in the social and behavioural sciences. To address these challenges we present InvisibleEye, a novel approach for mobile eye tracking that uses millimetre-size RGB cameras that can be fully embedded into normal glasses frames. To compensate for the cameras’ low image resolution of only a few pixels, our approach uses multiple cameras to capture different views of the eye, as well as learning-based gaze estimation to directly regress from eye images to gaze directions. We prototypically implement our system and characterise its performance on three large-scale, increasingly realistic, and thus challenging datasets: 1) eye images synthesised using a recent computer graphics eye region model, 2) real eye images recorded of 17 participants under controlled lighting, and 3) eye images recorded of four participants over the course of four recording sessions in a mobile setting. We show that InvisibleEye achieves a top person-specific gaze estimation accuracy of 1.79° using four cameras with a resolution of only 5×5 pixels. Our evaluations not only demonstrate the feasibility of this novel approach but, more importantly, underline its significant potential for finally realising the vision of invisible mobile eye tracking and pervasive attentive user interfaces.

   doi:10.1145/3130971

   tonsen17_imwut.pdf

   @article{tonsen17_imwut, author = {Tonsen, Marc and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, title = {InvisibleEye: Mobile Eye Tracking Using Multiple Low-Resolution Cameras and Learning-Based Gaze Estimation}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)}, year = {2017}, doi = {10.1145/3130971}, volume = {1}, number = {3}, pages = {106:1-106:21} }
3. 

   Look together: using gaze for assisting co-located collaborative search

   Yanxia Zhang, Ken Pfeuffer, Ming Ki Chong, Jason Alexander, Andreas Bulling, Hans Gellersen

   Personal and Ubiquitous Computing, 21(1), pp. 173-186, 2017.

   Abstract Links BibTeX Project

   Gaze information provides indication of users focus which complements remote collaboration tasks, as distant users can see their partner’s focus. In this paper, we apply gaze for co-located collaboration, where users’ gaze locations are presented on the same display, to help collaboration between partners. We integrated various types of gaze indicators on the user interface of a collaborative search system, and we conducted two user studies to understand how gaze enhances coordination and communication between co-located users. Our results show that gaze indeed enhances co-located collaboration, but with a trade-off between visibility of gaze indicators and user distraction. Users acknowledged that seeing gaze indicators eases communication, because it let them be aware of their partner’s interests and attention. However, users can be reluctant to share their gaze information due to trust and privacy, as gaze potentially divulges their interests.

   doi:10.1007/s00779-016-0969-x

   zhang17_puc.pdf

   @article{zhang17_puc, title = {Look together: using gaze for assisting co-located collaborative search}, author = {Zhang, Yanxia and Pfeuffer, Ken and Chong, Ming Ki and Alexander, Jason and Bulling, Andreas and Gellersen, Hans}, year = {2017}, journal = {Personal and Ubiquitous Computing}, publisher = {Springer}, volume = {21}, number = {1}, pages = {173-186}, doi = {10.1007/s00779-016-0969-x} }

Conference Papers

1. 

   Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery

   Xucong Zhang, Yusuke Sugano, Andreas Bulling

   Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 193-203, 2017.

   Abstract Links BibTeX Project Best paper honourable mention award

   Eye contact is an important non-verbal cue in social signal processing and promising as a measure of overt attention in human-object interactions and attentive user interfaces. However, robust detection of eye contact across different users, gaze targets, camera positions, and illumination conditions is notoriously challenging. We present a novel method for eye contact detection that combines a state-of-the-art appearance-based gaze estimator with a novel approach for unsupervised gaze target discovery, i.e. without the need for tedious and time-consuming manual data annotation. We evaluate our method in two real-world scenarios: detecting eye contact at the workplace, including on the main work display, from cameras mounted to target objects, as well as during everyday social interactions with the wearer of a head-mounted egocentric camera. We empirically evaluate the performance of our method in both scenarios and demonstrate its effectiveness for detecting eye contact independent of target object type and size, camera position, and user and recording environment.

   doi:10.1145/3126594.3126614

   zhang17_uist.pdf

   @inproceedings{zhang17_uist, title = {Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery}, author = {Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, year = {2017}, pages = {193-203}, doi = {10.1145/3126594.3126614}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)} }
2. 

   It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2299-2308, 2017.

   Abstract Links BibTeX Project

   Eye gaze is an important non-verbal cue for human affect analysis. Recent gaze estimation work indicated that information from the full face region can benefit performance. Pushing this idea further, we propose an appearance-based method that, in contrast to a long-standing line of work in computer vision, only takes the full face image as input. Our method encodes the face image using a convolutional neural network with spatial weights applied on the feature maps to flexibly suppress or enhance information in different facial regions. Through extensive evaluation, we show that our full-face method significantly outperforms the state of the art for both 2D and 3D gaze estimation, achieving improvements of up to 14.3% on MPIIGaze and 27.7% on EYEDIAP for person-independent 3D gaze estimation. We further show that this improvement is consistent across different illumination conditions and gaze directions and par- ticularly pronounced for the most challenging extreme head poses.

   doi:10.1109/CVPRW.2017.284

   zhang17_cvprw.pdf

   @inproceedings{zhang17_cvprw, title = {It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)}, year = {2017}, doi = {10.1109/CVPRW.2017.284}, pages = {2299-2308} }
3. 

   They are all after you: Investigating the Viability of a Threat Model that involves Multiple Shoulder Surfers

   Mohamed Khamis, Linda Bandelow, Stina Schick, Dario Casadevall, Andreas Bulling, Florian Alt

   Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM), 2017.

   Abstract Links BibTeX Project Best paper honourable mention award

   Many of the authentication schemes for mobile devices that were proposed lately complicate shoulder surfing by splitting the attacker’s attention into two or more entities. For example, multimodal authentication schemes such as GazeTouchPIN and GazeTouchPass require attackers to observe the user’s gaze input and the touch input performed on the phone’s screen. These schemes have always been evaluated against single observers, while multiple observers could potentially attack these schemes with greater ease, since each of them can focus exclusively on one part of the password. In this work, we study the effectiveness of a novel threat model against authentication schemes that split the attacker’s attention. As a case study, we report on a security evaluation of two state of the art authentication schemes in the case of a team of two observers. Our results show that although multiple observers perform better against these schemes than single observers, multimodal schemes are significantly more secure against multiple observers compared to schemes that employ a single modality. We discuss how this threat model impacts the design of authentication schemes.

   doi:10.1145/3152832.3152851

   khamis17_mum.pdf

   @inproceedings{khamis17_mum, title = {They are all after you: Investigating the Viability of a Threat Model that involves Multiple Shoulder Surfers}, author = {Khamis, Mohamed and Bandelow, Linda and Schick, Stina and Casadevall, Dario and Bulling, Andreas and Alt, Florian}, year = {2017}, doi = {10.1145/3152832.3152851}, booktitle = {Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM)} }
4. 

   GazeTouchPIN: Protecting Sensitive Data on Mobile Devices using Secure Multimodal Authentication

   Mohamed Khamis, Mariam Hassib, Emanuel Zezschwitz, Andreas Bulling, Florian Alt

   Proc. ACM International Conference on Multimodal Interaction (ICMI), pp. 446-450, 2017.

   Abstract Links BibTeX Project

   Although mobile devices provide access to a plethora of sensitive data, most users still only protect them with PINs or patterns, which are vulnerable to side-channel attacks (e.g., shoulder surfing). However, prior research has shown that privacy-aware users are willing to take further steps to protect their private data. We propose GazeTouchPIN, a novel secure authentication scheme for mobile devices that combines gaze and touch input. Our multimodal approach complicates shoulder-surfing attacks by requiring attackers to observe the screen as well as the user’s eyes to find the password. We evaluate the security and usability of GazeTouchPIN in two user studies (N=30). We found that while GazeTouchPIN requires longer entry times, privacy aware users would use it on-demand when feeling observed or when accessing sensitive data. The results show that successful shoulder surfing attack rate drops from 68% to 10.4% when using GazeTouchPIN.

   doi:10.1145/3136755.3136809

   khamis17_icmi.pdf

   @inproceedings{khamis17_icmi, title = {GazeTouchPIN: Protecting Sensitive Data on Mobile Devices using Secure Multimodal Authentication}, author = {Khamis, Mohamed and Hassib, Mariam and von Zezschwitz, Emanuel and Bulling, Andreas and Alt, Florian}, year = {2017}, pages = {446-450}, doi = {10.1145/3136755.3136809}, booktitle = {Proc. ACM International Conference on Multimodal Interaction (ICMI)} }
5. 

   EyeScout: Active Eye Tracking for Position and Movement Independent Gaze Interaction with Large Public Displays

   Mohamed Khamis, Axel Hoesl, Alexander Klimczak, Martin Reiss, Florian Alt, Andreas Bulling

   Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 155-166, 2017.

   Abstract Links BibTeX Project

   While gaze holds a lot of promise for hands-free interaction with public displays, remote eye trackers with their confined tracking box restrict users to a single stationary position in front of the display. We present EyeScout, an active eye tracking system that combines an eye tracker mounted on a rail system with a computational method to automatically detect and align the tracker with the user’s lateral movement. EyeScout addresses key limitations of current gaze-enabled large public displays by offering two novel gaze-interaction modes for a single user: In "Walk then Interact" the user can walk up to an arbitrary position in front of the display and interact, while in "Walk and Interact" the user can interact even while on the move. We report on a user study that shows that EyeScout is well perceived by users, extends a public display’s sweet spot into a sweet line, and reduces gaze interaction kick- off time to 3.5 seconds - a 62% improvement over state of the art solutions. We discuss sample applications that demonstrate how EyeScout can enable position and movement-independent gaze interaction with large public displays.

   doi:10.1145/3126594.3126630

   khamis17_uist.pdf

   @inproceedings{khamis17_uist, title = {EyeScout: Active Eye Tracking for Position and Movement Independent Gaze Interaction with Large Public Displays}, author = {Khamis, Mohamed and Hoesl, Axel and Klimczak, Alexander and Reiss, Martin and Alt, Florian and Bulling, Andreas}, year = {2017}, pages = {155-166}, doi = {10.1145/3126594.3126630}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)} }
6. 

   Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling

   Hosnieh Sattar, Andreas Bulling, Mario Fritz

   Proc. IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 2740-2748, 2017.

   Abstract Links BibTeX Project

   Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction. In contrast to previous work that has focused on individual instances as search target, we propose the first approach to predict categories and attributes of search targets based on gaze data. However, state of the art models for categorical recognition in general require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-based architectures as an attention mechanism - incorporating both spatial and temporal aspects of human gaze behavior. We show that our approach is effective even when the gaze pooling layer is added to an already trained CNN, thus eliminating the need for expensive joint data collection of visual and gaze data. We propose an experimental setup and data set and demonstrate the effectiveness of our method for search target prediction based on gaze behavior. We further study how to integrate temporal and spatial gaze information most effectively, and indicate directions for future research in gaze-based prediction of mental states.

   doi:10.1109/ICCVW.2017.322

   sattar17_iccvw.pdf

   @inproceedings{sattar17_iccvw, title = {Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling}, author = {Sattar, Hosnieh and Bulling, Andreas and Fritz, Mario}, year = {2017}, pages = {2740-2748}, doi = {10.1109/ICCVW.2017.322}, booktitle = {Proc. IEEE International Conference on Computer Vision Workshops (ICCVW)} }
7. 

   GTmoPass: Two-factor Authentication on Public Displays Using GazeTouch passwords and Personal Mobile Devices

   Mohamed Khamis, Regina Hasholzner, Andreas Bulling, Florian Alt

   Proc. ACM International Symposium on Pervasive Displays (PerDis), pp. 8:1-8:9, 2017.

   Abstract Links BibTeX Project

   As public displays continue to deliver increasingly private and personalized content, there is a need to ensure that only the legitimate users can access private information in sensitive contexts. While public displays can adopt similar authentication concepts like those used on public terminals (e.g., ATMs), authentication in public is subject to a number of risks. Namely, adversaries can uncover a user’s password through (1) surfing users, (2) thermal attacks, or (3) smudge attacks. To address this problem we propose GTmoPass, an authentication architecture that enables Multi-factor user authentication on public displays. The first factor is a knowledge-factor: we employ a shoulder-surfing resilient multimodal scheme that combines gaze and touch input for password entry. The second factor is a possession-factor: users utilize their personal mobile devices, on which they enter the password. Credentials are securely transmitted to a server via Bluetooth beacons. We describe the implementation of GTmoPass and report on an evaluation of its usability and security, which shows that although authentication using GTmoPass is slightly slower than traditional methods, it protects against the three aforementioned threats.

   doi:10.1145/3078810.3078815

   khamis17_perdis.pdf

   @inproceedings{khamis17_perdis, title = {GTmoPass: Two-factor Authentication on Public Displays Using GazeTouch passwords and Personal Mobile Devices}, author = {Khamis, Mohamed and Hasholzner, Regina and Bulling, Andreas and Alt, Florian}, doi = {10.1145/3078810.3078815}, year = {2017}, pages = {8:1-8:9}, booktitle = {Proc. ACM International Symposium on Pervasive Displays (PerDis)} }
8. 

   Gaze Embeddings for Zero-Shot Image Classification

   Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling

   Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6412-6421, 2017.

   Abstract Links BibTeX Project Spotlight presentation

   Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.

   doi:10.1109/CVPR.2017.679

   karessli17_cvpr.pdf

   @inproceedings{karessli17_cvpr, title = {Gaze Embeddings for Zero-Shot Image Classification}, author = {Karessli, Nour and Akata, Zeynep and Schiele, Bernt and Bulling, Andreas}, year = {2017}, booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, pages = {6412-6421}, doi = {10.1109/CVPR.2017.679} }
9. 

   Noticeable or Distractive? A Design Space for Gaze-Contingent User Interface Notifications

   Michaela Klauck , Yusuke Sugano, Andreas Bulling

   Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1779-1786, 2017.

   Abstract Links BibTeX Project

   Users are interrupted by an ever-increasing number of notifications, ranging from error messages, over new email or chat alerts, to advertisement pop-ups. We explore gaze-contingent user interfaces notifications that are shown depending on users’ current gaze location. Specifically, we evaluate how different design properties influence notification noticeability and distractiveness. We measure noticeability quantitatively by analyzing participants’ performance in confirming notifications and distractiveness using a questionnaire. Based on a 12-participant user study on a public display, we show that each of these properties affects noticeability and distractiveness differently and that the properties, in turn, allow for fine-grained optimization of notification display. These findings inform the design of future attentive user interfaces that could optimize the trade-off between, for example, the notification importance and the cost of interruption.

   doi:10.1145/3027063.3053085

   klauck17_chi.pdf

   @inproceedings{klauck17_chi, author = {Klauck, Michaela and Sugano, Yusuke and Bulling, Andreas}, title = {Noticeable or Distractive? A Design Space for Gaze-Contingent User Interface Notifications}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2017}, pages = {1779-1786}, doi = {10.1145/3027063.3053085} }
10. 

    EyeMirror: Mobile Calibration-Free Gaze Approximation using Corneal Imaging

    Christian Lander, Sven Gehring, Markus Löchtefeld, Andreas Bulling, Antonio Krüger

    Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM), 2017.

    Abstract Links BibTeX Project

    Gaze is a powerful measure of people’s attracted attention and reveals where we are looking at within our current field of view. Hence, gaze-based interfaces are gaining in importance. However, gaze estimation usually requires extensive hardware and depends on a calibration that has to be renewed regularly. We present EyeMirror, a mobile device for calibration-free gaze approximation on surfaces (e.g. displays). It consists of a head-mounted camera, connected to a wearable mini-computer, capturing the environment reflected on the human cornea. The corneal images are analyzed using natural feature tracking for gaze estimation on surfaces. In two lab studies we compared variations of EyeMirror against established methods for gaze estimation in a display scenario, and investigated the effect of display content (i.e. number of features). EyeMirror achieved 4.03° gaze estimation error, while we found no significant effect of display content.

    doi:10.1145/3152832.3152839

    lander17_mum.pdf

    @inproceedings{lander17_mum, title = {EyeMirror: Mobile Calibration-Free Gaze Approximation using Corneal Imaging}, author = {Lander, Christian and Gehring, Sven and L{\"{o}}chtefeld, Markus and Bulling, Andreas and Kr{\"{u}}ger, Antonio}, year = {2017}, doi = {10.1145/3152832.3152839}, booktitle = {Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM)} }

Technical Reports

1. 

   MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   arXiv:1711.09017, 2017.

   Abstract Links BibTeX Project

   Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze dataset, which contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance, while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves on the state of the art by 22% (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.

   zhang17_arxiv.pdf

   https://arxiv.org/abs/1711.09017

   @techreport{zhang17_arxiv, title = {MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, year = {2017}, code = {https://arxiv.org/abs/1711.09017} }
2. 

   GazeDirector: Fully Articulated Eye Gaze Redirection in Video

   Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling

   arXiv:1704.08763, 2017.

   Abstract Links BibTeX Project

   We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting. Our method first tracks the eyes by fitting a multi-part eye region model to video frames using analysis-by-synthesis, thereby recovering eye region shape, texture, pose, and gaze simultaneously. It then redirects gaze by 1) warping the eyelids from the original image using a model-derived flow field, and 2) rendering and compositing synthesized 3D eyeballs onto the output image in a photorealistic manner. GazeDirector allows us to change where people are looking without person-specific training data, and with full articulation, i.e. we can precisely specify new gaze directions in 3D. Quantitatively, we evaluate both model-fitting and gaze synthesis, with experiments for gaze estimation and redirection on the Columbia gaze dataset. Qualitatively, we compare GazeDirector against recent work on gaze redirection, showing better results especially for large redirection angles. Finally, we demonstrate gaze redirection on YouTube videos by introducing new 3D gaze targets and by manipulating visual behavior.

   wood17_arxiv.pdf

   https://arxiv.org/abs/1704.08763

   @techreport{wood17_arxiv, title = {GazeDirector: Fully Articulated Eye Gaze Redirection in Video}, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Morency, Louis-Philippe and Robinson, Peter and Bulling, Andreas}, year = {2017}, code = {https://arxiv.org/abs/1704.08763} }
3. 

   Visual Decoding of Targets During Visual Search From Human Eye Fixations

   Hosnieh Sattar, Mario Fritz, Andreas Bulling

   arXiv:1706.05993, 2017.

   Abstract Links BibTeX Project

   What does human gaze reveal about a users’ intents and to which extend can these intents be inferred or even visualized? Gaze was proposed as an implicit source of information to predict the target of visual search and, more recently, to predict the object class and attributes of the search target. In this work, we go one step further and investigate the feasibility of combining recent advances in encoding human gaze information using deep convolutional neural networks with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Such visual decoding is challenging for two reasons: 1) the search target only resides in the user’s mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all. We show, for the first time, that visual representations of search targets can indeed be decoded only from human gaze fixations. We propose to first encode fixations into a semantic representation and then decode this representation into an image. We evaluate our method on a recent gaze dataset of 14 participants searching for clothing in image collages and validate the model’s predictions using two human studies. Our results show that 62% (Chance level = 10%) of the time users were able to select the categories of the decoded image right. In our second studies we show the importance of a local gaze encoding for decoding visual search targets of users.

   sattar17_arxiv.pdf

   https://arxiv.org/abs/1706.05993

   @techreport{sattar17_arxiv, title = {Visual Decoding of Targets During Visual Search From Human Eye Fixations}, author = {Sattar, Hosnieh and Fritz, Mario and Bulling, Andreas}, year = {2017}, code = {https://arxiv.org/abs/1706.05993} }
4. 

   A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks

   Arif Khan, Ingmar Steiner, Yusuke Sugano, Andreas Bulling, Ross Macdonald

   arXiv:1712.04798, 2017.

   Abstract Links BibTeX Project

   Phonetic segmentation is the process of splitting speech into distinct phonetic units. Human experts routinely perform this task manually by analyzing auditory and visual cues using analysis software, which is an extremely time-consuming process. Methods exist for automatic segmentation, but these are not always accurate enough. In order to improve automatic segmentation, we need to model it as close to the manual segmentation as possible. This corpus is an effort to capture the human segmentation behavior by recording experts performing a segmentation task. We believe that this data will enable us to highlight the important aspects of manual segmentation, which can be used in automatic segmentation to improve its accuracy.

   khan17_arxiv.pdf

   https://arxiv.org/abs/1712.04798

   @techreport{khan17_arxiv, title = {A Multimodal Corpus of Expert Gaze and Behavior during Phonetic Segmentation Tasks}, author = {Khan, Arif and Steiner, Ingmar and Sugano, Yusuke and Bulling, Andreas and Macdonald, Ross}, year = {2017}, code = {https://arxiv.org/abs/1712.04798} }

2016

Journal Articles

1. 

   Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art

   Wolfgang Fuhl, Marc Tonsen, Andreas Bulling, Enkelejda Kasneci

   Springer Machine Vision and Applications, , pp. 1-14, 2016.

   Abstract Links BibTeX Project

   Robust and accurate detection of the pupil position is a key building block for head-mounted eye tracking and prerequisite for applications on top, such as gaze-based human-computer interaction or attention analysis. Despite a large body of work, detecting the pupil in images recorded under real-world conditions is challenging given significant variability in eye appearance (e.g., illumination, reflections, occlusions, etc.), individual differences in eye physiology, as well as other sources of noise, such as contact lenses or make-up. In this paper we review six state-of-the-art pupil detection methods, namely ElSe, ExCuSe, Pupil Labs, SET, Starburst, and Swirski. We compare their performance on a large-scale dataset consisting of 225,569 annotated eye images taken from four publicly available datasets. Our experimental results show that the algorithm ElSe outperforms other pupil detection methods by a large margin, offering thus robust and accurate pupil positions on challenging everyday eye images.

   doi:10.1007/s00138-016-0776-4

   fuhl16_mvap.pdf

   @article{fuhl16_mvap, title = {Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art}, author = {Fuhl, Wolfgang and Tonsen, Marc and Bulling, Andreas and Kasneci, Enkelejda}, year = {2016}, pages = {1-14}, doi = {10.1007/s00138-016-0776-4}, journal = {Springer Machine Vision and Applications} }
2. 

   EyeWear Computers for Human-Computer Interaction

   Andreas Bulling, Kai Kunze

   ACM Interactions, 23(3), pp. 70-73, 2016.

   Abstract Links BibTeX Project

   Head-worn displays and eye trackers, augmented and virtual reality glasses, egocentric cameras, and other "smart eyewear" have recently emerged as a research platform in fields such as ubiquitous computing, computer vision, and cognitive and social science. While earlier generations of devices were too bulky to be worn regularly, recent technological advances have made eyewear unobtrusive and lightweight, and therefore more suitable for daily use. Given that many human senses are located on the head, smart eyewear provides opportunities for types of interaction that were impossible before now. In this article, we highlight the potential of eyewear computing for HCI, discuss available input and output modalities, and suggest the most promising future directions for eyewear computing research, namely multimodal user modeling, lifelong learning, and large-scale (collective) human-behavior sensing and analysis.

   doi:10.1145/2912886

   bulling16_interactions.pdf

   @article{bulling16_interactions, title = {EyeWear Computers for Human-Computer Interaction}, author = {Bulling, Andreas and Kunze, Kai}, year = {2016}, journal = {ACM Interactions}, volume = {23}, number = {3}, doi = {10.1145/2912886}, pages = {70-73} }
3. 

   Pervasive Attentive User Interfaces

   Andreas Bulling

   IEEE Computer, 49(1), pp. 94-98, 2016.

   Abstract Links BibTeX Project

   As the number of displays we interact with rapidly increases, managing user attention has emerged as a critical challenge for next-generation human−computer interfaces.

   doi:10.1109/MC.2016.32

   bulling16_computer.pdf

   @article{bulling16_computer, title = {Pervasive Attentive User Interfaces}, author = {Bulling, Andreas}, doi = {10.1109/MC.2016.32}, year = {2016}, journal = {IEEE Computer}, volume = {49}, number = {1}, pages = {94-98} }

Conference Papers

1. 

   Attention, please! Comparing Features for Measuring Audience Attention Towards Pervasive Displays

   Florian Alt, Andreas Bulling, Lukas Mecke, Daniel Buschek

   Proc. ACM SIGCHI Conference on Designing Interactive Systems (DIS), pp. 823-828, 2016.

   Abstract Links BibTeX Project

   Measuring audience attention towards pervasive displays is important but accurate measurement in real time remains a significant sensing challenge. Consequently, researchers and practitioners typically use other features, such as face presence, as a proxy. We provide a principled comparison of the performance of six features and their combinations for measuring attention: face presence, movement trajectory, walking speed, shoulder orientation, head pose, and gaze direction. We implemented a prototype that is capable of capturing this rich set of features from video and depth camera data. Using a controlled lab experiment (N=18) we show that as a single feature, face presence is indeed among the most accurate. We further show that accuracy can be increased through a combination of features (+10.3%), knowledge about the audience (+63.8%), as well as user identities (+69.0%). Our findings are valuable for display providers who want to collect data on display effectiveness or build interactive, responsive apps.

   doi:10.1145/2901790.2901897

   alt16_dis.pdf

   @inproceedings{alt16_dis, author = {Alt, Florian and Bulling, Andreas and Mecke, Lukas and Buschek, Daniel}, title = {Attention, please! Comparing Features for Measuring Audience Attention Towards Pervasive Displays}, booktitle = {Proc. ACM SIGCHI Conference on Designing Interactive Systems (DIS)}, year = {2016}, pages = {823-828}, doi = {10.1145/2901790.2901897} }
2. 

   AggreGaze: Collective Estimation of Audience Attention on Public Displays

   Yusuke Sugano, Xucong Zhang, Andreas Bulling

   Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 821-831, 2016.

   Abstract Links BibTeX Project Best paper honourable mention award

   Gaze is frequently explored in public display research given its importance for monitoring and analysing audience attention. However, current gaze-enabled public display interfaces require either special-purpose eye tracking equipment or explicit personal calibration for each individual user. We present AggreGaze, a novel method for estimating spatio-temporal audience attention on public displays. Our method requires only a single off-the-shelf camera attached to the display, does not require any personal calibration, and provides visual attention estimates across the full display. We achieve this by 1) compensating for errors of state-of-the-art appearance-based gaze estimation methods through on-site training data collection, and by 2) aggregating uncalibrated and thus inaccurate gaze estimates of multiple users into joint attention estimates. We propose different visual stimuli for this compensation: a standard 9-point calibration, moving targets, text and visual stimuli embedded into the display content, as well as normal video content. Based on a two-week deployment in a public space, we demonstrate the effectiveness of our method for estimating attention maps that closely resemble ground-truth audience gaze distributions.

   doi:10.1145/2984511.2984536

   sugano16_uist.pdf

   @inproceedings{sugano16_uist, title = {AggreGaze: Collective Estimation of Audience Attention on Public Displays}, author = {Sugano, Yusuke and Zhang, Xucong and Bulling, Andreas}, year = {2016}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, doi = {10.1145/2984511.2984536}, pages = {821-831} }
3. 

   Smooth Eye Movement Interaction Using EOG Glasses

   Murtaza Dhuliawala, Juyoung Lee, Junichi Shimizu, Andreas Bulling, Kai Kunze, Thad Starner, Woontack Woo

   Proc. International Conference on Multimodal Interaction (ICMI), pp. 307-311, 2016.

   Abstract Links BibTeX Project

   Orbits combines a visual display and an eye motion sensor to allow a user to select between options by tracking a cursor with the eyes as the cursor travels in a circular path around each option. Using an off-the-shelf J!NS MEME pair of eyeglasses, we present a pilot study that suggests that the eye movement required for Orbits can be sensed using three electrodes: one in the nose bridge and one in each nose pad. For forced choice binary selection, we achieve a 2.6 bits per second (bps) input rate at 250ms per input. We also introduce Head Orbits, where the user fixates the eyes on a target and moves the head in synchrony with the orbiting target. Measuring only the relative movement of the eyes in relation to the head, this method achieves a maximum rate of 2.0 bps at 500ms per input. Finally, we combine the two techniques together with a gyro to create an interface with a maximum input rate of 5.0 bps.

   doi:10.1145/2993148.2993181

   dhuliawala16_icmi.pdf

   @inproceedings{dhuliawala16_icmi, title = {Smooth Eye Movement Interaction Using EOG Glasses}, author = {Dhuliawala, Murtaza and Lee, Juyoung and Shimizu, Junichi and Bulling, Andreas and Kunze, Kai and Starner, Thad and Woo, Woontack}, year = {2016}, doi = {10.1145/2993148.2993181}, pages = {307-311}, booktitle = {Proc. International Conference on Multimodal Interaction (ICMI)} }
4. 

   On the Verge: Voluntary Convergences for Accurate and Precise Timing of Gaze Input

   Dominik Kirst, Andreas Bulling

   Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1519-1525, 2016.

   Abstract Links BibTeX Project

   The problem of triggering input accurately (with a small temporal offset) and precisely (with high repeatability) at a specific point in time has so far been largely ignored in gaze interaction research. We explore voluntary eye convergences as a novel interaction technique for precise and accurate timing of gaze input and a solution to the "Midas touch" problem, i.e. the accidental triggering of input when looking at an interface. We introduce a novel clock paradigm to study input timing and demonstrate that voluntary convergences are significantly more accurate and precise than common gaze dwelling. Our findings suggest that voluntary convergences are well-suited for applications in which timing of user input is important, thereby complementing existing gaze techniques that focus on speed and spatial precision.

   doi:10.1145/2851581.2892307

   kirst16_chi.pdf

   @inproceedings{kirst16_chi, author = {Kirst, Dominik and Bulling, Andreas}, title = {On the Verge: Voluntary Convergences for Accurate and Precise Timing of Gaze Input}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2016}, pages = {1519-1525}, doi = {10.1145/2851581.2892307} }
5. 

   Xplore-M-Ego: Contextual Media Retrieval Using Natural Language Queries

   Sreyasi Nag Chowdhury, Mateusz Malinowski, Andreas Bulling, Mario Fritz

   Proc. ACM International Conference on Multimedia Retrieval (ICMR), pp. 243-247, 2016.

   Abstract Links BibTeX Project

   The widespread integration of cameras in hand-held and head-worn devices and the ability to share content online enables a large and diverse visual capture of the world that millions of users build up collectively every day. We envision these images as well as associated meta information, such as GPS coordinates and timestamps, to form a collective visual memory that can be queried while automatically taking the ever-changing context of mobile users into account. As a first step towards this vision, in this work we present Xplore-M-Ego: a novel media retrieval system that allows users to query a dynamic database of images using spatio-temporal natural language queries. We evaluate our system using a new dataset of real image queries as well as through a usability study. One key finding is that there is a considerable amount of inter-user variability in the resolution of spatial relations in natural language utterances. We show that our system can cope with this variability using personalisation through an online learning-based retrieval formulation.

   doi:10.1145/2911996.2912044

   chowdhury16_icmr.pdf

   @inproceedings{chowdhury16_icmr, title = {Xplore-M-Ego: Contextual Media Retrieval Using Natural Language Queries}, author = {Chowdhury, Sreyasi Nag and Malinowski, Mateusz and Bulling, Andreas and Fritz, Mario}, year = {2016}, booktitle = {Proc. ACM International Conference on Multimedia Retrieval (ICMR)}, doi = {10.1145/2911996.2912044}, pages = {243-247} }
6. 

   Prediction of Gaze Estimation Error for Error-Aware Gaze-Based Interfaces

   Michael Barz, Florian Daiber, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 275-278, 2016.

   Abstract Links BibTeX Project

   Gaze estimation error is inherent in head-mounted eye trackers and seriously impacts performance, usability, and user experience of gaze-based interfaces. Particularly in mobile settings, this error varies constantly as users move in front and look at different parts of a display. We envision a new class of gaze-based interfaces that are aware of the gaze estimation error and adapt to it in real time. As a first step towards this vision we introduce an error model that is able to predict the gaze estimation error. Our method covers major building blocks of mobile gaze estimation, specifically mapping of pupil positions to scene camera coordinates, marker-based display detection, and mapping of gaze from scene camera to on-screen coordinates. We develop our model through a series of principled measurements of a state-of-the-art head-mounted eye tracker.

   doi:10.1145/2857491.2857493

   barz16_etra.pdf

   @inproceedings{barz16_etra, author = {Barz, Michael and Daiber, Florian and Bulling, Andreas}, title = {Prediction of Gaze Estimation Error for Error-Aware Gaze-Based Interfaces}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {275-278}, doi = {10.1145/2857491.2857493} }
7. 

   3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers

   Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 197-200, 2016.

   Abstract Links BibTeX Project

   3D gaze information is important for scene-centric attention analysis, but accurate estimation and analysis of 3D gaze in real-world environments remains challenging. We present a novel 3D gaze estimation method for monocular head-mounted eye trackers. In contrast to previous work, our method does not aim to infer 3D eye- ball poses, but directly maps 2D pupil positions to 3D gaze directions in scene camera coordinate space. We first provide a detailed discussion of the 3D gaze estimation task and summarize different methods, including our own. We then evaluate the performance of different 3D gaze estimation approaches using both simulated and real data. Through experimental validation, we demonstrate the effectiveness of our method in reducing parallax error, and we identify research challenges for the design of 3D calibration procedures.

   doi:10.1145/2857491.2857530

   mansouryar16_etra.pdf

   @inproceedings{mansouryar16_etra, author = {Mansouryar, Mohsen and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, title = {3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {197-200}, doi = {10.1145/2857491.2857530} }
8. 

   Memorability of Cued-Recall Graphical Passwords with Saliency Masks

   Florian Alt, Mateusz Mikusz, Stefan Schneegass, Andreas Bulling

   Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM), pp. 191-200, 2016.

   Abstract Links BibTeX Project

   Cued-recall graphical passwords have a lot of potential for secure user authentication, particularly if combined with saliency masks to prevent users from selecting weak passwords. Saliency masks exclude those areas of the image that are most likely to lead to hotspots and were shown to significantly improve password security. In this paper we investigate the impact of such saliency masks on the memorability of cued-recall graphical passwords. We first conduct two pre-studies with 52 participants to obtain a set of images with three different image complexities as well as real passwords. Based on a month-long user study with 26 participants we then show that cued-recall graphical passwords defined on a single image with a saliency mask are not more difficult to remember than those without saliency mask, and that the complexity of the password images does not have any influence on password memorability. These results complement prior work on the security of such passwords and underline the potential of saliency masks as both a secure and usable improvement to cued-recall gaze-based graphical passwords.

   doi:10.1145/3012709.3012730

   alt16_mum.pdf

   @inproceedings{alt16_mum, title = {Memorability of Cued-Recall Graphical Passwords with Saliency Masks}, author = {Alt, Florian and Mikusz, Mateusz and Schneegass, Stefan and Bulling, Andreas}, year = {2016}, doi = {10.1145/3012709.3012730}, pages = {191-200}, booktitle = {Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM)} }
9. 

   Concept for Using Eye Tracking in a Head-mounted Display to Adapt Rendering to the User’s Current Visual Field

   Daniel Pohl, Xucong Zhang, Andreas Bulling, Oliver Grau

   Proc. of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST), pp. 323-324, 2016.

   Abstract Links BibTeX Project

   With increasing spatial and temporal resolution in head-mounted displays (HMDs), using eye trackers to adapt rendering to the user is getting important to handle the rendering workload. Besides using methods like foveated rendering, we propose to use the current visual field for rendering, depending on the eye gaze. We use two effects for performance optimizations. First, we noticed a lens defect in HMDs, where depending on the distance of the eye gaze to the center, certain parts of the screen towards the edges are not visible anymore. Second, if the user looks up, he cannot see the lower parts of the screen anymore. For the invisible areas, we propose to skip rendering and to reuse the pixels colors from the previous frame. We provide a calibration routine to measure these two effects. We apply the current visual field to a renderer and get up to 2x speed-ups.

   doi:10.1145/2993369.2996300

   pohl16_vrst.pdf

   @inproceedings{pohl16_vrst, title = {Concept for Using Eye Tracking in a Head-mounted Display to Adapt Rendering to the User's Current Visual Field}, author = {Pohl, Daniel and Zhang, Xucong and Bulling, Andreas and Grau, Oliver}, doi = {10.1145/2993369.2996300}, year = {2016}, booktitle = {Proc. of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST)}, pages = {323-324} }
10. 

    Combining Eye Tracking with Optimizations for Lens Astigmatism in Modern Wide-Angle HMDs

    Daniel Pohl, Xucong Zhang, Andreas Bulling

    Proc. IEEE Virtual Reality (VR), pp. 269-270, 2016.

    Abstract Links BibTeX Project

    Virtual Reality has hit the consumer market with affordable head-mounted displays. When using these, it quickly becomes apparent that the resolution of the built-in display panels still needs to be highly increased. To overcome the resulting higher performance demands, eye tracking can be used for foveated rendering. However, as there are lens distortions in HMDs, there are more possibilities to increase the performance with smarter rendering approaches. We present a new system using optimizations for rendering considering lens astigmatism and combining this with foveated rendering through eye tracking. Depending on the current eye gaze, this delivers a rendering speed-up of up to 20%.

    doi:10.1109/VR.2016.7504757

    pohl16_vr.pdf

    @inproceedings{pohl16_vr, title = {Combining Eye Tracking with Optimizations for Lens Astigmatism in Modern Wide-Angle HMDs}, author = {Pohl, Daniel and Zhang, Xucong and Bulling, Andreas}, year = {2016}, pages = {269-270}, doi = {10.1109/VR.2016.7504757}, booktitle = {Proc. IEEE Virtual Reality (VR)} }
11. 

    EyeVote in the Wild: Do Users bother Correcting System Errors on Public Displays?

    Mohamed Khamis, Ludwig Trotter, Markus Tessman, Christina Dannhart, Andreas Bulling, Florian Alt

    Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM), pp. 57-62, 2016.

    Abstract Links BibTeX Project

    Although recovering from errors is straightforward on most interfaces, public display systems pose very unique design challenges. Namely, public display users interact for very short amounts of times and are believed to abandon the display when interrupted or forced to deviate from the main task. To date, it is not well understood whether public display designers should enable users to correct errors (e.g. by asking users to confirm or giving them a chance correct their input), or aim for faster interaction and rely on other types of feedback to estimate errors. To close this gap, we conducted a field study where we investigated the users willingness to correct their input on public displays. We report on our findings from an in-the-wild deployment of a public gaze-based voting system where we intentionally evoked system errors to see if users correct them. We found that public display users are willing to correct system errors provided that the correction is fast and straightforward. We discuss how our findings influence the choice of interaction methods for public displays; interaction methods that are highly usable but suffer from low accuracy can still be effective if users can "undo" their interactions.

    doi:10.1145/3012709.3012743

    khamis16_mum.pdf

    @inproceedings{khamis16_mum, title = {EyeVote in the Wild: Do Users bother Correcting System Errors on Public Displays?}, author = {Khamis, Mohamed and Trotter, Ludwig and Tessman, Markus and Dannhart, Christina and Bulling, Andreas and Alt, Florian}, year = {2016}, doi = {10.1145/3012709.3012743}, pages = {57-62}, booktitle = {Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM)} }
12. 

    SkullConduct: Biometric User Identification on Eyewear Computers Using Bone Conduction Through the Skull

    Stefan Schneegass, Youssef Oualil, Andreas Bulling

    Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1379-1384, 2016.

    Abstract Links BibTeX Project

    Secure user identification is important for the increasing number of eyewear computers but limited input capabilities pose significant usability challenges for established knowledge-based schemes, such as a passwords or PINs. We present SkullConduct, a biometric system that uses bone conduction of sound through the user’s skull as well as a microphone readily integrated into many of these devices, such as Google Glass. At the core of SkullConduct is a method to analyze the characteristic frequency response created by the user’s skull using a combination of Mel Frequency Cepstral Coefficient (MFCC) features as well as a computationally light-weight 1NN classifier. We report on a controlled experiment with 10 participants that shows that this frequency response is person-specific and stable - even when taking off and putting on the device multiple times - and thus serves as a robust biometric. We show that our method can identify users with 97.0% accuracy and authenticate them with an equal error rate of 6.9%, thereby bringing biometric user identification to eyewear computers equipped with bone conduction technology.

    doi:10.1145/2858036.2858152

    schneegass16_chi.pdf

    @inproceedings{schneegass16_chi, title = {SkullConduct: Biometric User Identification on Eyewear Computers Using Bone Conduction Through the Skull}, author = {Schneegass, Stefan and Oualil, Youssef and Bulling, Andreas}, year = {2016}, pages = {1379-1384}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/2858036.2858152} }
13. 

    A 3D Morphable Eye Region Model for Gaze Estimation

    Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling

    Proc. European Conference on Computer Vision (ECCV), pp. 297-313, 2016.

    Abstract Links BibTeX Project

    Morphable face models are a powerful tool, but have previ- ously failed to model the eye accurately due to complexities in its material and motion. We present a new multi-part model of the eye that includes a morphable model of the facial eye region, as well as an anatomy-based eyeball model. It is the first morphable model that accurately captures eye region shape, since it was built from high-quality head scans. It is also the first to allow independent eyeball movement, since we treat it as a separate part. To showcase our model we present a new method for illumination- and head-pose–invariant gaze estimation from a single RGB image. We fit our model to an image through analysis-by-synthesis, solving for eye region shape, texture, eyeball pose, and illumination simul- taneously. The fitted eyeball pose parameters are then used to estimate gaze direction. Through evaluation on two standard datasets we show that our method generalizes to both webcam and high-quality camera images, and outperforms a state-of-the-art CNN method achieving a gaze estimation accuracy of 9.44° in a challenging user-independent scenario.

    doi:10.1007/978-3-319-46448-0_18

    wood16_eccv.pdf

    @inproceedings{wood16_eccv, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Morency, Louis-Philippe and Robinson, Peter and Bulling, Andreas}, title = {A 3D Morphable Eye Region Model for Gaze Estimation}, booktitle = {Proc. European Conference on Computer Vision (ECCV)}, year = {2016}, pages = {297-313}, doi = {10.1007/978-3-319-46448-0_18} }
14. 

    Gaussian Processes as an Alternative to Polynomial Gaze Estimation Functions

    Laura Sesma-Sanchez, Yanxia Zhang, Andreas Bulling, Hans Gellersen

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 229-232, 2016.

    Abstract Links BibTeX Project

    Interpolation-based methods are widely used for gaze estimation due to their simplicity. In particular, feature-based methods that map the image eye features to gaze, are very popular. The most spread regression function used in this kind of method is the polynomial regression. In this paper, we present an alternative regression function to estimate gaze: the Gaussian regression. We show how the Gaussian processes can better adapt to the non-linear behavior of the eye movement, providing higher gaze estimation accuracies. The Gaussian regression is compared, in a simulated environment, to the polynomial regression, when using the same mapping features, the normalized pupil center-corneal reflection and pupil center-eye corners vectors. This comparison is done for three different screen sizes. The results show that for larger screens, where wider gaze angles are required, i.e., the non-linear behavior of the eye is more present, the outperformance of the Gaussian regression is more evident. Furthermore, we can conclude that, for both types of regressions, the gaze estimation accuracy increases for smaller screens, where the eye movements are more linear.

    doi:10.1145/2857491.2857509

    sesma16_etra.pdf

    @inproceedings{sesma16_etra, author = {Sesma-Sanchez, Laura and Zhang, Yanxia and Bulling, Andreas and Gellersen, Hans}, title = {Gaussian Processes as an Alternative to Polynomial Gaze Estimation Functions}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {229-232}, doi = {10.1145/2857491.2857509} }
15. 

    Solar System: Smooth Pursuit Interactions Using EOG Glasses

    Junichi Shimizu, Juyoung Lee, Murtaza Dhuliawala, Andreas Bulling, Thad Starner, Woontack Woo, Kai Kunze

    Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 369-372, 2016.

    Abstract Links BibTeX Project

    Solar System implements smooth pursuit eye movement interactions on commercial smart glasses using electrooculography. The system requires no calibration and little to no training. We present a prototype implementation, describe initial user tests and show several application scenarios for hands-free eye gaze interactions.

    doi:10.1145/2968219.2971376

    shimizu16_ubicomp.pdf

    @inproceedings{shimizu16_ubicomp, author = {Shimizu, Junichi and Lee, Juyoung and Dhuliawala, Murtaza and Bulling, Andreas and Starner, Thad and Woo, Woontack and Kunze, Kai}, title = {Solar System: Smooth Pursuit Interactions Using EOG Glasses}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2016}, pages = {369-372}, doi = {10.1145/2968219.2971376}, keywords = {eye tracking, gaze interaction, wearable computing} }
16. 

    Three-Point Interaction: Combining Bi-manual Direct Touch with Gaze

    Adalberto Simeone, Andreas Bulling, Jason Alexander, Hans Gellersen

    Proc. International Conference on Advanced Visual Interfaces (AVI), pp. 168-175, 2016.

    Abstract Links BibTeX Project

    The benefits of two-point interaction for tasks that require users to simultaneously manipulate multiple entities or dimensions are widely known. Two-point interaction has become common, e.g., when zooming or pinching using two fingers on a smartphone. We propose a novel interaction technique that implements three-point interaction by augmenting two-finger direct touch with gaze as a third input channel. We evaluate two key characteristics of our technique in two multi-participant user studies. In the first, we used the technique for object selection. In the second, we evaluate it in a 3D matching task that requires simultaneous continuous input from fingers and the eyes. Our results show that in both cases participants learned to interact with three input channels without cognitive or mental overload. Participants’ performance tended towards fast selection times in the first study and exhibited parallel interaction in the second. These results are promising and show that there is scope for additional input channels beyond two-point interaction.

    doi:10.1145/2909132.2909251

    simeone16_avi.pdf

    @inproceedings{simeone16_avi, author = {Simeone, Adalberto and Bulling, Andreas and Alexander, Jason and Gellersen, Hans}, title = {Three-Point Interaction: Combining Bi-manual Direct Touch with Gaze}, booktitle = {Proc. International Conference on Advanced Visual Interfaces (AVI)}, year = {2016}, pages = {168-175}, doi = {10.1145/2909132.2909251} }
17. 

    Challenges and Design Space of Gaze-enabled Public Displays

    Mohamed Khamis, Florian Alt, Andreas Bulling

    Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 1736-1745, 2016.

    Abstract Links BibTeX Project

    Gaze is an attractive modality for public displays, hence the recent years saw an increase in deployments of gaze-enabled public displays. Although gaze has been thoroughly investigated for desktop scenarios, gaze-enabled public displays present new challenges that are unique to this setup. In contrast to desktop settings, public displays (1) cannot afford requiring eye tracker calibration, (2) expect users to interact from different positions, and (3) expect multiple users to interact simultaneously. In this work we discuss these challenges, and explore the design space of gaze-enabled public displays. We conclude by discussing how the current state of research stands wrt. the identified challenges, and highlight directions for future work.

    doi:10.1145/2968219.2968342

    khamis16_petmei.pdf

    @inproceedings{khamis16_petmei, title = {Challenges and Design Space of Gaze-enabled Public Displays}, author = {Khamis, Mohamed and Alt, Florian and Bulling, Andreas}, year = {2016}, pages = {1736-1745}, doi = {10.1145/2968219.2968342}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)} }
18. 

    TextPursuits: Using Text for Pursuits-Based Interaction and Calibration on Public Displays

    Mohamed Khamis, Ozan Saltuk, Alina Hang, Katharina Stolz, Andreas Bulling, Florian Alt

    Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 274-285, 2016.

    Abstract Links BibTeX Project

    Pursuits, a technique that correlates users’ eye movements with moving on-screen targets, was recently introduced for calibration-free interaction with public displays. While prior work used abstract objects or dots as targets, we explore the use of Pursuits with text (read-and-pursue). Given that much of the content on public displays includes text, designers could greatly benefit from users being able to spontaneously interact and implicitly calibrate an eye tracker while simply read- ing text on a display. At the same time, using Pursuits with textual content is challenging given that the eye movements performed while reading interfere with the pursuit movements. We present two systems, EyeVote and Read2Calibrate, that enable spontaneous gaze interaction and implicit calibration by reading text. Results from two user studies (N=37) show that Pursuits with text is feasible and can achieve similar accu- racy as non text-based pursuit approaches. While calibration is less accurate, it integrates smoothly with reading and allows to identify areas of the display the user is looking at.

    doi:10.1145/2971648.2971679

    khamis16_ubicomp.pdf

    @inproceedings{khamis16_ubicomp, author = {Khamis, Mohamed and Saltuk, Ozan and Hang, Alina and Stolz, Katharina and Bulling, Andreas and Alt, Florian}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, title = {TextPursuits: Using Text for Pursuits-Based Interaction and Calibration on Public Displays}, year = {2016}, doi = {10.1145/2971648.2971679}, pages = {274-285} }
19. 

    Learning an appearance-based gaze estimator from one million synthesised images

    Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 131–138, 2016.

    Abstract Links BibTeX Project Emerging investigator award

    Learning-based methods for appearance-based gaze estimation achieve state-of-the-art performance in challenging real-world settings but require large amounts of labelled training data. Learning-by-synthesis was proposed as a promising solution to this problem but current methods are limited with respect to speed, the appearance variability as well as the head pose and gaze angle distribution they can synthesize. We present UnityEyes, a novel method to rapidly synthesize large amounts of variable eye region images as training data. Our method combines a novel generative 3D model of the human eye region with a real-time rendering framework. The model is based on high-resolution 3D face scans and uses real- time approximations for complex eyeball materials and structures as well as novel anatomically inspired procedural geometry methods for eyelid animation. We show that these synthesized images can be used to estimate gaze in difficult in-the-wild scenarios, even for extreme gaze angles or in cases in which the pupil is fully occluded. We also demonstrate competitive gaze estimation results on a benchmark in-the-wild dataset, despite only using a light-weight nearest-neighbor algorithm. We are making our UnityEyes synthesis framework freely available online for the benefit of the research community.

    doi:10.1145/2857491.2857492

    wood16_etra.pdf

    @inproceedings{wood16_etra, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Morency, Louis-Philippe and Robinson, Peter and Bulling, Andreas}, title = {Learning an appearance-based gaze estimator from one million synthesised images}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {131--138}, doi = {10.1145/2857491.2857492} }
20. 

    Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments

    Marc Tonsen, Xucong Zhang, Yusuke Sugano, Andreas Bulling

    Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 139-142, 2016.

    Abstract Links BibTeX Project

    We present labelled pupils in the wild (LPW), a novel dataset of 66 high-quality, high-speed eye region videos for the development and evaluation of pupil detection algorithms. The videos in our dataset were recorded from 22 participants in everyday locations at about 95 FPS using a state-of-the-art dark-pupil head-mounted eye tracker. They cover people of different ethnicities and a diverse set of everyday indoor and outdoor illumination environments, as well as natural gaze direction distributions. The dataset also includes participants wearing glasses, contact lenses, and make-up. We bench- mark five state-of-the-art pupil detection algorithms on our dataset with respect to robustness and accuracy. We further study the influence of image resolution and vision aids as well as recording lo- cation (indoor, outdoor) on pupil detection performance. Our evaluations provide valuable insights into the general pupil detection problem and allow us to identify key challenges for robust pupil detection on head-mounted eye trackers.

    doi:10.1145/2857491.2857520

    tonsen16_etra.pdf

    @inproceedings{tonsen16_etra, author = {Tonsen, Marc and Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, title = {Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2016}, pages = {139-142}, doi = {10.1145/2857491.2857520} }
21. 

    GazeTouchPass: Multimodal Authentication Using Gaze and Touch on Mobile Devices

    Mohamed Khamis, Florian Alt, Mariam Hassib, Emanuel Zezschwitz, Regina Hasholzner, Andreas Bulling

    Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 2156-2164, 2016.

    Abstract Links BibTeX Project

    We propose a multimodal scheme, GazeTouchPass, that combines gaze and touch for shoulder-surfing resistant user authentication on mobile devices. GazeTouchPass allows passwords with multiple switches between input modalities during authentication. This requires attackers to simultaneously observe the device screen and the user’s eyes to find the password. We evaluate the security and usability of GazeTouchPass in two user studies. Our findings show that GazeTouchPass is usable and significantly more secure than single-modal authentication against basic and even advanced shoulder-surfing attacks.

    doi:10.1145/2851581.2892314

    khamis16_chi.pdf

    @inproceedings{khamis16_chi, author = {Khamis, Mohamed and Alt, Florian and Hassib, Mariam and von Zezschwitz, Emanuel and Hasholzner, Regina and Bulling, Andreas}, title = {GazeTouchPass: Multimodal Authentication Using Gaze and Touch on Mobile Devices}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2016}, pages = {2156-2164}, doi = {10.1145/2851581.2892314} }
22. 

    Spatio-Temporal Modeling and Prediction of Visual Attention in Graphical User Interfaces

    Pingmei Xu, Yusuke Sugano, Andreas Bulling

    Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 3299-3310, 2016.

    Abstract Links BibTeX Project Best paper honourable mention award

    We present a computational model to predict users’ spatio-temporal visual attention for WIMP-style (windows, icons, mouse, pointer) graphical user interfaces. Like existing models of bottom-up visual attention in computer vision, our model does not require any eye tracking equipment. Instead, it predicts attention solely using information available to the interface, specifically users’ mouse and keyboard input as well as the UI components they interact with. To study our model in a principled way we further introduce a method to synthesize user interface layouts that are functionally equivalent to real-world interfaces, such as from Gmail, Facebook, or GitHub. We first quantitatively analyze attention allocation and its correlation with user input and UI components using ground-truth gaze, mouse, and keyboard data of 18 participants performing a text editing task. We then show that our model predicts attention maps more accurately than state-of-the-art methods. Our results underline the significant potential of spatio-temporal attention modeling for user interface evaluation, optimization, or even simulation.

    doi:10.1145/2858036.2858479

    xu16_chi.pdf

    @inproceedings{xu16_chi, title = {Spatio-Temporal Modeling and Prediction of Visual Attention in Graphical User Interfaces}, author = {Xu, Pingmei and Sugano, Yusuke and Bulling, Andreas}, year = {2016}, pages = {3299-3310}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, doi = {10.1145/2858036.2858479} }

Book Chapters

1. Proc. 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)

   Paul Lukowicz, Antonio Krüger, Andreas Bulling, Youn-Kyung Lim, Shwetak N. Patel

   Heidelberg, Germany, ACM, 2016.

   BibTeX Project

   @inbook{lukowicz16_ubicomp, title = {Proc. 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2016}, author = {Lukowicz, Paul and Kr{\"{u}}ger, Antonio and Bulling, Andreas and Lim, Youn-Kyung and Patel, Shwetak N.}, isbn = {978-1-4503-4461-6}, location = {Heidelberg, Germany}, publisher = {ACM}, address = {New York, NY, USA}, url = {https://dl.acm.org/citation.cfm?id=2971648} }

Technical Reports

1. 

   Gaze Embeddings for Zero-Shot Image Classification

   Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling

   arXiv:1611.09309, 2016.

   Abstract Links BibTeX Project

   Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.

   karessli16_arxiv.pdf

   https://arxiv.org/abs/1611.09309

   @techreport{karessli16_arxiv, title = {Gaze Embeddings for Zero-Shot Image Classification}, author = {Karessli, Nour and Akata, Zeynep and Schiele, Bernt and Bulling, Andreas}, year = {2016}, code = {https://arxiv.org/abs/1611.09309} }
2. 

   It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   arXiv:1611.08860, 2016.

   Abstract Links BibTeX Project

   Eye gaze is an important non-verbal cue for human affect analysis. Recent gaze estimation work indicated that information from the full face region can benefit performance. Pushing this idea further, we propose an appearance-based method that, in contrast to a long-standing line of work in computer vision, only takes the full face image as input. Our method encodes the face image using a convolutional neural network with spatial weights applied on the feature maps to flexibly suppress or enhance information in different facial regions. Through extensive evaluation, we show that our full-face method significantly outperforms the state of the art for both 2D and 3D gaze estimation, achieving improvements of up to 14.3% on MPIIGaze and 27.7% on EYEDIAP for person-independent 3D gaze estimation. We further show that this improvement is consistent across different illumination conditions and gaze directions and particularly pronounced for the most challenging extreme head poses.

   zhang16_arxiv.pdf

   https://arxiv.org/abs/1611.08860

   @techreport{zhang16_arxiv, title = {It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, year = {2016}, code = {https://arxiv.org/abs/1611.08860} }
3. 

   Contextual Media Retrieval Using Natural Language Queries

   Sreyasi Nag Chowdhury, Mateusz Malinowski, Andreas Bulling, Mario Fritz

   arXiv:1602.04983, 2016.

   Abstract Links BibTeX Project

   The widespread integration of cameras in hand-held and head-worn devices as well as the ability to share content online enables a large and diverse visual capture of the world that millions of users build up collectively every day. We envision these images as well as associated meta information, such as GPS coordinates and timestamps, to form a collective visual memory that can be queried while automatically taking the ever-changing context of mobile users into account. As a first step towards this vision, in this work we present Xplore-M-Ego: a novel media retrieval system that allows users to query a dynamic database of images and videos using spatio-temporal natural language queries. We evaluate our system using a new dataset of real user queries as well as through a usability study. One key finding is that there is a considerable amount of inter-user variability, for example in the resolution of spatial relations in natural language utterances. We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.

   chowdhury16_arxiv.pdf

   https://arxiv.org/abs/1602.04983

   @techreport{chowdhury16_arxiv, title = {Contextual Media Retrieval Using Natural Language Queries}, author = {Chowdhury, Sreyasi Nag and Malinowski, Mateusz and Bulling, Andreas and Fritz, Mario}, year = {2016}, code = {https://arxiv.org/abs/1602.04983} }
4. 

   Seeing with Humans: Gaze-Assisted Neural Image Captioning

   Yusuke Sugano, Andreas Bulling

   arXiv:1608.05203, 2016.

   Abstract Links BibTeX Project

   Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks.

   sugano16_tr.pdf

   https://arxiv.org/abs/1608.05203

   @techreport{sugano16_tr, title = {Seeing with Humans: Gaze-Assisted Neural Image Captioning}, author = {Sugano, Yusuke and Bulling, Andreas}, year = {2016}, code = {https://arxiv.org/abs/1608.05203} }
5. 

   Predicting the Category and Attributes of Mental Pictures Using Deep Gaze Pooling

   Hosnieh Sattar, Andreas Bulling, Mario Fritz

   arXiv:1611.10162, 2016.

   Abstract Links BibTeX Project

   Predicting the target of visual search from eye fixation (gaze) data is a challenging problem with many applications in human-computer interaction. In contrast to previous work that has focused on individual instances as a search target, we propose the first approach to predict categories and attributes of search targets based on gaze data. However, state of the art models for categorical recognition, in general, require large amounts of training data, which is prohibitive for gaze data. To address this challenge, we propose a novel Gaze Pooling Layer that integrates gaze information into CNN-based architectures as an attention mechanism - incorporating both spatial and temporal aspects of human gaze behavior. We show that our approach is effective even when the gaze pooling layer is added to an already trained CNN, thus eliminating the need for expensive joint data collection of visual and gaze data. We propose an experimental setup and data set and demonstrate the effectiveness of our method for search target prediction based on gaze behavior. We further study how to integrate temporal and spatial gaze information most effectively, and indicate directions for future research in the gaze-based prediction of mental states.

   sattar16_arxiv.pdf

   https://arxiv.org/abs/1611.10162

   @techreport{sattar16_arxiv, title = {Predicting the Category and Attributes of Mental Pictures Using Deep Gaze Pooling}, author = {Sattar, Hosnieh and Bulling, Andreas and Fritz, Mario}, year = {2016}, code = {https://arxiv.org/abs/1611.10162} }
6. 

   3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers

   Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling

   arXiv:1601.02644, 2016.

   Abstract Links BibTeX Project

   3D gaze information is important for scene-centric attention analysis but accurate estimation and analysis of 3D gaze in real-world environments remains challenging. We present a novel 3D gaze estimation method for monocular head-mounted eye trackers. In contrast to previous work, our method does not aim to infer 3D eyeball poses but directly maps 2D pupil positions to 3D gaze directions in scene camera coordinate space. We first provide a detailed discussion of the 3D gaze estimation task and summarize different methods, including our own. We then evaluate the performance of different 3D gaze estimation approaches using both simulated and real data. Through experimental validation, we demonstrate the effectiveness of our method in reducing parallax error, and we identify research challenges for the design of 3D calibration procedures.

   mansouryar16_arxiv.pdf

   https://arxiv.org/abs/1601.02644

   @techreport{mansouryar16_arxiv, title = {3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers}, author = {Mansouryar, Mohsen and Steil, Julian and Sugano, Yusuke and Bulling, Andreas}, year = {2016}, code = {https://arxiv.org/abs/1601.02644} }
7. 

   Computational Modelling and Prediction of Gaze Estimation Error for Head-mounted Eye Trackers

   Michael Barz, Andreas Bulling, Florian Daiber

   DFKI Research Reports, 2016.

   Abstract Links BibTeX Project

   Head-mounted eye tracking has significant potential for mobile gaze-based interaction with ambient displays but current interfaces lack information about the tracker’s gaze estimation error. Consequently, current interfaces do not exploit the full potential of gaze input as the inherent estimation error can not be dealt with. The error depends on the physical properties of the display and constantly varies with changes in position and distance of the user to the display. In this work we present a computational model of gaze estimation error for head-mounted eye trackers. Our model covers the full processing pipeline for mobile gaze estimation, namely mapping of pupil positions to scene camera coordinates, marker-based display detection, and display mapping. We build the model based on a series of controlled measurements of a sample state-of-the-art monocular head-mounted eye tracker. Results show that our model can predict gaze estimation error with a root mean squared error of 17.99 px (1.96^circ).

   barz16_techrep.pdf

   @techreport{barz16_techrep, author = {Barz, Michael and Bulling, Andreas and Daiber, Florian}, title = {Computational Modelling and Prediction of Gaze Estimation Error for Head-mounted Eye Trackers}, volume = {1}, year = {2016}, institution = {German Research Center for Artificial Intelligence (DFKI)} }
8. 

   End-to-End Eye Movement Detection Using Convolutional Neural Networks

   Sabrina Hoppe, Andreas Bulling

   arXiv:1609.02452, 2016.

   Abstract Links BibTeX Project

   Common computational methods for automated eye movement detection - i.e. the task of detecting different types of eye movement in a continuous stream of gaze data - are limited in that they either involve thresholding on hand-crafted signal features, require individual detectors each only detecting a single movement, or require pre-segmented data. We propose a novel approach for eye movement detection that only involves learning a single detector end-to-end, i.e. directly from the continuous gaze data stream and simultaneously for different eye movements without any manual feature crafting or segmentation. Our method is based on convolutional neural networks (CNN) that recently demonstrated superior performance in a variety of tasks in computer vision, signal processing, and machine learning. We further introduce a novel multi-participant dataset that contains scripted and free-viewing sequences of ground-truth annotated saccades, fixations, and smooth pursuits. We show that our CNN-based method outperforms state-of-the-art baselines by a large margin on this challenging dataset, thereby underlining the significant potential of this approach for holistic, robust, and accurate eye movement protocol analysis.

   hoppe16_arxiv.pdf

   https://arxiv.org/abs/1609.02452

   @techreport{hoppe16_arxiv, title = {End-to-End Eye Movement Detection Using Convolutional Neural Networks}, author = {Hoppe, Sabrina and Bulling, Andreas}, year = {2016}, code = {https://arxiv.org/abs/1609.02452} }

2015

Journal Articles

1. 

   Walking reduces spatial neglect

   Tobias Loetscher, Celia Chen, Sabrina Hoppe, Andreas Bulling, Sophie Wignall, Churches Owen, Nicole Thomas, Andrew Lee

   , 21(S2), pp. 120-121, 2015.

   Abstract Links BibTeX Project

   Spatial neglect is a common consequence of stroke. Neglect behaviour is typically exacerbated by increased task demands. It was thus anticipated that the addition of a secondary task requiring general attention (walking) would worsen performance on tests of spatial neglect. Here, however, we report a patient in whom neglect was considerably reduced when performing a visual search task while walking.

   loetscher15_ins.pdf

   @article{loetscher15_ins, title = {Walking reduces spatial neglect}, author = {Loetscher, Tobias and Chen, Celia and Hoppe, Sabrina and Bulling, Andreas and Wignall, Sophie and Owen, Churches and Thomas, Nicole and Lee, Andrew}, year = {2015}, volume = {21}, number = {S2}, pages = {120-121}, booktitle = {Journal of the International Neuropsychological Society} }
2. 

   The Feet in HCI: A Survey of Foot-Based Interaction

   Eduardo Velloso, Dominik Schmidt, Jayson Alexander, Hans Gellersen, Andreas Bulling

   ACM Computing Surveys, 48(2), 2015.

   Abstract Links BibTeX Project

   Foot-operated computer interfaces have been studied since the inception of Human-Computer Interaction. Thanks to the miniaturisation and decreasing cost of sensing technology, there is an increasing interest exploring this alternative input modality, but no comprehensive overview of its research landscape. In this survey, we review the literature on interfaces operated by the lower limbs. We investigate the characteristics of users and how they affect the design of such interfaces. Next, we describe and analyse foot-based research prototypes and commercial systems in how they capture input and provide feedback. We then analyse the interactions between users and systems from the perspective of the actions performed in these interactions. Finally, we discuss our findings and use them to identify open questions and directions for future research.

   doi:10.1145/2816455

   velloso15_csur.pdf

   @article{velloso15_csur, author = {Velloso, Eduardo and Schmidt, Dominik and Alexander, Jayson and Gellersen, Hans and Bulling, Andreas}, title = , journal = {ACM Computing Surveys}, year = {2015}, volume = {48}, number = {2}, doi = {10.1145/2816455} }
3. 

   Pursuits: Spontaneous Eye-Based Interaction for Dynamic Interfaces

   Mélodie Vidal, Andreas Bulling, Hans Gellersen

   ACM SIGMOBILE Mobile Computing and Communications Review, 18(4), pp. 8-10, 2015.

   Abstract Links BibTeX Project

   Although gaze is an attractive modality for pervasive interaction, real-world implementation of eye-based interfaces poses significant challenges. In particular, user calibration is tedious and time consuming. Pursuits is an innovative interaction technique that enables truly spontaneous interaction with eye-based interfaces. A user can simply walk up to the screen and readily interact with moving targets. Instead of being based on gaze location, Pursuits correlates eye pursuit movements with objects dynamically moving on the interface.

   doi:10.1145/2721914.2721917

   vidal15_sigmobile.pdf

   @article{vidal15_sigmobile, author = {Vidal, M{\'{e}}lodie and Bulling, Andreas and Gellersen, Hans}, title = {Pursuits: Spontaneous Eye-Based Interaction for Dynamic Interfaces}, journal = {ACM SIGMOBILE Mobile Computing and Communications Review}, volume = {18}, number = {4}, year = {2015}, pages = {8-10}, doi = {10.1145/2721914.2721917} }
4. 

   Introduction to the Special Issue on Activity Recognition for Interaction

   Andreas Bulling, Ulf Blanke, Desney Tan, Jun Rekimoto, Gregory Abowd

   ACM Transactions on Interactive Intelligent Systems (TiiS), 4(16e), 2015.

   Abstract Links BibTeX Project

   This editorial introduction describes the aims and scope of the ACM Transactions on Interactive Intelligent Systems special issue on Activity Recognition for Interaction. It explains why activity recognition is becoming crucial as part of the cycle of interaction between users and computing systems, and it shows how the five articles selected for this special issue reflect this theme.

   doi:10.1145/2694858

   bulling15_tiis.pdf

   @article{bulling15_tiis, author = {Bulling, Andreas and Blanke, Ulf and Tan, Desney and Rekimoto, Jun and Abowd, Gregory}, title = {Introduction to the Special Issue on Activity Recognition for Interaction}, journal = {ACM Transactions on Interactive Intelligent Systems (TiiS)}, volume = {4}, number = {16e}, year = {2015}, doi = {10.1145/2694858} }
5. 

   Eye tracking for public displays in the wild

   Yanxia Zhang, Ming Ki Chong, Jörg Müller, Andreas Bulling, Hans Gellersen

   Springer Personal and Ubiquitous Computing, 19(5), pp. 967-981, 2015.

   Abstract Links BibTeX Project

   In public display contexts, interactions are spontaneous and have to work without preparation. We propose gaze as a modality for such con- texts, as gaze is always at the ready, and a natural indicator of the user’s interest. We present GazeHorizon, a system that demonstrates sponta- neous gaze interaction, enabling users to walk up to a display and navi- gate content using their eyes only. GazeHorizon is extemporaneous and optimised for instantaneous usability by any user without prior configura- tion, calibration or training. The system provides interactive assistance to bootstrap gaze interaction with unaware users, employs a single off-the- shelf web camera and computer vision for person-independent tracking of the horizontal gaze direction, and maps this input to rate-controlled nav- igation of horizontally arranged content. We have evaluated GazeHorizon through a series of field studies, culminating in a four-day deployment in a public environment during which over a hundred passers-by interacted with it, unprompted and unassisted. We realised that since eye move- ments are subtle, users cannot learn gaze interaction from only observing others, and as a results guidance is required.

   doi:10.1007/s00779-015-0866-8

   zhang15_puc.pdf

   @article{zhang15_puc, title = {Eye tracking for public displays in the wild}, author = {Zhang, Yanxia and Chong, Ming Ki and M\"uller, J\"org and Bulling, Andreas and Gellersen, Hans}, year = {2015}, doi = {10.1007/s00779-015-0866-8}, pages = {967-981}, volume = {19}, number = {5}, journal = {Springer Personal and Ubiquitous Computing}, keywords = {Eye tracking; Gaze interaction; Public displays; Scrolling; Calibration-free; In-the-wild study; Deployment} }
6. 

   A study on the natural history of scanning behaviour in patients with visual field defects after stroke

   Tobias Loetscher, Celia Chen, Sophie Wignall, Andreas Bulling, Sabrina Hoppe, Owen Churches, Nicole Thomas

   BMC Neurology, 15(64), 2015.

   Abstract Links BibTeX Project

   A visual field defect (VFD) is a common consequence of stroke with a detrimental effect upon the survivors’ functional ability and quality of life. The identification of effective treatments for VFD is a key priority relating to life post-stroke. Understanding the natural evolution of scanning compensation over time may have important ramifications for the development of efficacious therapies. The study aims to unravel the natural history of visual scanning behaviour in patients with VFD. The assessment of scanning patterns in the acute to chronic stages of stroke will reveal who does and does not learn to compensate for vision loss. Methods/Design Eye-tracking glasses are used to delineate eye movements in a cohort of 100 stroke patients immediately after stroke, and additionally at 6 and 12 months post-stroke. The longitudinal study will assess eye movements in static (sitting) and dynamic (walking) conditions. The primary outcome constitutes the change of lateral eye movements from the acute to chronic stages of stroke. Secondary outcomes include changes of lateral eye movements over time as a function of subgroup characteristics, such as side of VFD, stroke location, stroke severity and cognitive functioning. Discussion The longitudinal comparison of patients who do and do not learn compensatory scanning techniques may reveal important prognostic markers of natural recovery. Importantly, it may also help to determine the most effective treatment window for visual rehabilitation.

   doi:10.1186/s12883-015-0321-5

   loetscher15_neurology.pdf

   @article{loetscher15_neurology, title = {A study on the natural history of scanning behaviour in patients with visual field defects after stroke}, author = {Loetscher, Tobias and Chen, Celia and Wignall, Sophie and Bulling, Andreas and Hoppe, Sabrina and Churches, Owen and Thomas, Nicole}, doi = {10.1186/s12883-015-0321-5}, year = {2015}, journal = {BMC Neurology}, volume = {15}, number = {64} }

Conference Papers

1. 

   Gaze+RST: Integrating Gaze and Multitouch for Remote Rotate-Scale-Translate Tasks

   Jayson Turner, Jason Alexander, Andreas Bulling, Hans Gellersen

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 4179-4188, 2015.

   Abstract Links BibTeX Project

   Our work investigates the use of gaze and multitouch to flu- idly perform rotate-scale translate (RST) tasks on large dis- plays. The work specifically aims to understand if gaze can provide benefit in such a task, how task complexity af- fects performance, and how gaze and multitouch can be com- bined to create an integral input structure suited to the task of RST. We present four techniques that individually strike a different balance between gaze-based and touch-based trans- lation while maintaining concurrent rotation and scaling op- erations. A 16 participant empirical evaluation revealed that three of our four techniques present viable options for this scenario, and that larger distances and rotation/scaling opera- tions can significantly affect a gaze-based translation configu- ration. Furthermore we uncover new insights regarding mul- timodal integrality, finding that gaze and touch can be com- bined into configurations that pertain to integral or separable input structures.

   doi:10.1145/2702123.2702355

   turner15_chi.pdf

   @inproceedings{turner15_chi, author = {Turner, Jayson and Alexander, Jason and Bulling, Andreas and Gellersen, Hans}, title = {Gaze+RST: Integrating Gaze and Multitouch for Remote Rotate-Scale-Translate Tasks}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2015}, pages = {4179-4188}, doi = {10.1145/2702123.2702355} }
2. 

   An Empirical Investigation of Gaze Selection in Mid-Air Gestural 3D Manipulation

   Eduardo Velloso, Jayson Turner, Jason Alexander, Andreas Bulling, Hans Gellersen

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 315-330, 2015.

   Abstract Links BibTeX Project

   In this work, we investigate gaze selection in the context of mid-air hand gestural manipulation of 3D rigid bodies in monoscopic displays. We present the results of a user study with 12 participants in which we compared the performance of Gaze, a Raycasting technique (2D Cursor) and a Virtual Hand technique (3D Cursor) to select objects in two 3D mid-air interaction tasks. Also, we compared selection confirmation times for Gaze selection when selection is followed by manipulation to when it is not. Our results show that gaze selection is faster and more preferred than 2D and 3D mid-air-controlled cursors, and is particularly well suited for tasks in which users constantly switch between several objects during the manipulation. Further, selection confirmation times are longer when selection is followed by manipulation than when it is not.

   doi:10.1007/978-3-319-22668-2_25

   velloso15_interact.pdf

   @inproceedings{velloso15_interact, title = , author = {Velloso, Eduardo and Turner, Jayson and Alexander, Jason and Bulling, Andreas and Gellersen, Hans}, year = {2015}, pages = {315-330}, doi = {10.1007/978-3-319-22668-2_25}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)} }
3. 

   Interactions Under the Desk: A Characterisation of Foot Movements for Input in a Seated Position

   Eduardo Velloso, Jason Alexander, Andreas Bulling, Hans Gellersen

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 384-401, 2015.

   Abstract Links BibTeX Project

   This paper takes a bottom-up approach to characterising foot movements as input for users seated at computing systems. We conducted four user studies to characterise various aspects of foot-based interaction. First, we built unconstrained foot pointing performance models for 16 participants in a seated desktop setting using 1D and 2D ISO 9241-9-compliant Fitts’s Law tasks. Second, we evaluated the effect of the foot and direction in one-direction tasks, finding no effect of the foot used, but a significant effect of the direction in which targets are distributed. Third, we compared the use of one foot against two feet to control two independent variables, finding that while one foot is better suited for tasks with a spatial representation that matches its movement, there is little difference between the two feet techniques when it does not. Fourth, we analysed the overhead caused by introducing a feet-controlled variable in a mouse-based task, finding the feet to be comparable to the scroll wheel. The results of our studies show the feet are an effective method of enhancing our interaction with desktop systems; we use our findings to inform a series of design guidelines for such systems.

   doi:10.1007/978-3-319-22701-6_29

   velloso15_interact_2.pdf

   @inproceedings{velloso15_interact_2, title = {Interactions Under the Desk: A Characterisation of Foot Movements for Input in a Seated Position}, author = {Velloso, Eduardo and Alexander, Jason and Bulling, Andreas and Gellersen, Hans}, year = {2015}, pages = {384-401}, doi = {10.1007/978-3-319-22701-6_29}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)} }
4. 

   The Royal Corgi: Exploring Social Gaze Interaction for Immersive Gameplay

   Mélodie Vidal, Remi Bismuth, Andreas Bulling, Hans Gellersen

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 115-124, 2015.

   Abstract Links BibTeX Project

   The eyes are a rich channel for non-verbal communication in our daily interactions. We propose social gaze interaction as a game mechanic to enhance user interactions with virtual characters. We develop a game from the ground-up in which characters are designed to be reactive to the player’s gaze in social ways, such as getting annoyed when the player seems distracted or changing their dialogue depending on the player’s apparent focus of attention. Results from a qualitative user study provide insights bout how social gaze interaction is intuitive for users, elicits deep feelings of immersion, and highlight the players’ self-consciousness of their own eye movements through their strong reactions to the characters.

   doi:10.1145/2702123.2702163

   vidal15_chi.pdf

   @inproceedings{vidal15_chi, author = {Vidal, M{\'{e}}lodie and Bismuth, Remi and Bulling, Andreas and Gellersen, Hans}, title = , booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2015}, pages = {115-124}, doi = {10.1145/2702123.2702163} }
5. 

   Analyzing Visual Attention During Whole Body Interaction with Public Displays

   Robert Walter, Andreas Bulling, David Lindlbauer, Martin Schüssler, Hans Jörg Müller

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 1263-1267, 2015.

   Abstract Links BibTeX Project

   While whole body interaction can enrich user experience on public displays, it remains unclear how common visualizations of user representations impact users’ ability to perceive content on the display. In this work we use a head-mounted eye tracker to record visual behavior of 25 users interacting with a public display game that uses a silhouette user representation, mirroring the users’ movements. Results from visual attention analysis as well as post-hoc recall and recognition tasks on display contents reveal that visual attention is mostly on users’ silhouette while peripheral screen elements remain largely unattended. In our experiment, content attached to the user representation attracted significantly more attention than other screen contents, while content placed at the top and bottom of the screen attracted significantly less. Screen contents attached to the user representation were also significantly better remembered than those at the top and bottom of the screen.

   doi:10.1145/2750858.280425

   walter15_ubicomp.pdf

   @inproceedings{walter15_ubicomp, author = {Walter, Robert and Bulling, Andreas and Lindlbauer, David and Sch{\"{u}}ssler, Martin and M{\"{u}}ller, Hans J{\"{o}}rg}, title = {Analyzing Visual Attention During Whole Body Interaction with Public Displays}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2015}, doi = {10.1145/2750858.280425}, pages = {1263-1267} }
6. 

   Human Visual Behaviour for Collaborative Human-Machine Interaction

   Andreas Bulling

   Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 903-907, 2015.

   Abstract Links BibTeX Project

   Non-verbal behavioural cues are fundamental to human communication and interaction. Despite significant advances in recent years, state-of-the-art human-machine systems still fall short in sensing, analysing, and fully "understanding" cues naturally expressed in everyday settings. Two of the most important non-verbal cues, as evidenced by a large body of work in experimental psychology and behavioural sciences, are visual (gaze) behaviour and body language. We envision a new class of collaborative human-machine systems that fully exploit the information content available in non-verbal human behaviour in everyday settings through joint analysis of human gaze and physical behaviour.

   doi:10.1145/2800835.2815378

   bulling15_ubicomp.pdf

   @inproceedings{bulling15_ubicomp, title = {Human Visual Behaviour for Collaborative Human-Machine Interaction}, author = {Bulling, Andreas}, doi = {10.1145/2800835.2815378}, pages = {903-907}, year = {2015}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)} }
7. 

   Appearance-based Gaze Estimation in the Wild

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4511-4520, 2015.

   Abstract Links BibTeX Project

   Appearance-based gaze estimation is believed to work well in real-world settings but existing datasets were collected under controlled laboratory conditions and methods were not evaluated across multiple datasets. In this work we study appearance-based gaze estimation in the wild. We present the MPIIGaze dataset that contains 213,659 images we collected from 15 participants during natural everyday laptop use over more than three months. Our dataset is significantly more variable than existing datasets with respect to appearance and illumination. We also present a method for in-the-wild appearance-based gaze estimation using multimodal convolutional neural networks, which significantly outperforms state-of-the art methods in the most challenging cross-dataset evaluation setting. We present an extensive evaluation of several state-of-the-art image-based gaze estimation algorithm on three current datasets, including our own. This evaluation provides clear insights and allows us identify key research challenges of gaze estimation in the wild.

   doi:10.1109/CVPR.2015.7299081

   zhang15_cvpr.pdf

   @inproceedings{zhang15_cvpr, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, title = {Appearance-based Gaze Estimation in the Wild}, booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2015}, pages = {4511-4520}, doi = {10.1109/CVPR.2015.7299081} }
8. 

   Orbits: Enabling Gaze Interaction in Smart Watches using Moving Targets

   Augusto Esteves, Eduardo Velloso, Andreas Bulling, Hans Gellersen

   Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 419-422, 2015.

   Abstract Links BibTeX Project

   In this paper we demonstrate Orbits, a novel gaze interaction technique that accounts for both the reduced size of smart watch displays and the hands-free nature of conventional watches. Orbits combines graphical controls that display one or multiple targets moving on a circular path, with input that is provided by users as they follow any of the targets briefly with their eyes. This gaze input triggers the functionality associated with the followed target – be it answering a call, playing a song or managing multiple notifications.

   doi:10.1145/2800835.2800942

   esteves15_ubicomp.pdf

   @inproceedings{esteves15_ubicomp, title = {Orbits: Enabling Gaze Interaction in Smart Watches using Moving Targets}, author = {Esteves, Augusto and Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans}, year = {2015}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, doi = {10.1145/2800835.2800942}, pages = {419-422} }
9. 

   Orbits: Enabling Gaze Interaction in Smart Watches using Moving Targets

   Augusto Esteves, Eduardo Velloso, Andreas Bulling, Hans Gellersen

   Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 457-466, 2015.

   Abstract Links BibTeX Project Best paper award

   We introduce Orbits, a novel gaze interaction technique that enables hands-free input on smart watches. The technique relies on moving controls to leverage the smooth pursuit movements of the eyes and detect whether and at which control the user is looking at. In Orbits, controls include targets that move in a circular trajectory in the face of the watch, and can be selected by following the desired one for a small amount of time. We conducted two user studies to assess the technique’s recognition and robustness, which demonstrated how Orbits is robust against false positives triggered by natural eye movements and how it presents a hands-free, high accuracy way of interacting with smart watches using off-the-shelf devices. Finally, we developed three example interfaces built with Orbits: a music player, a notifications face plate and a missed call menu. Despite relying on moving controls – very unusual in current HCI interfaces – these were generally well received by participants in a third and final study.

   doi:10.1145/2807442.2807499

   esteves15_uist.pdf

   @inproceedings{esteves15_uist, title = {Orbits: Enabling Gaze Interaction in Smart Watches using Moving Targets}, author = {Esteves, Augusto and Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans}, year = {2015}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, doi = {10.1145/2807442.2807499}, pages = {457-466} }
10. 

    Recognition of Curiosity Using Eye Movement Analysis

    Sabrina Hoppe, Tobias Loetscher, Stephanie Morey, Andreas Bulling

    Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 185-188, 2015.

    Abstract Links BibTeX Project

    Among the different personality traits that guide our behaviour, curiosity is particularly interesting for context-aware assistive systems as it is closely linked to our well-being and the way we learn. This work proposes eye movement analysis for automatic recognition of different levels of curiosity. We present a 26-participant gaze dataset recorded during a real-world shopping task with empirically validated curiosity questionnaires as ground truth. Using a support vector machine classifier and a leave-one-person-out evaluation scheme we can discriminate between two to four classes of standard curiosity scales well above chance. These results are promising and point towards a new class of context-aware systems that take the user’s curiosity into account, thereby enabling new types of interaction and user adaptation.

    doi:10.1145/2800835.2800910

    hoppe15_ubicomp.pdf

    @inproceedings{hoppe15_ubicomp, title = {Recognition of Curiosity Using Eye Movement Analysis}, author = {Hoppe, Sabrina and Loetscher, Tobias and Morey, Stephanie and Bulling, Andreas}, year = {2015}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, doi = {10.1145/2800835.2800910}, pages = {185-188} }
11. 

    GravitySpot: Guiding Users in Front of Public Displays Using On-Screen Visual Cues

    Florian Alt, Andreas Bulling, Gino Gravanis, Daniel Buschek

    Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 47-56, 2015.

    Abstract Links BibTeX Project

    Users tend to position themselves in front of interactive public displays in such a way as to best perceive its content. Currently, this sweet spot is implicitly defined by display properties, content, the input modality, as well as space constraints in front of the display. We present GravitySpot – an approach that makes sweet spots flexible by actively guiding users to arbitrary target positions in front of displays using visual cues. Such guidance is beneficial, for example, if a particular input technology only works at a specific distance or if users should be guided towards a non-crowded area of a large display. In two controlled lab studies (n=29) we evaluate different visual cues based on color, shape, and motion, as well as position-to-cue mapping functions. We show that both the visual cues and mapping functions allow for fine-grained control over positioning speed and accuracy. Findings are complemented by observations from a 3-month real-world deployment.

    doi:10.1145/2807442.2807490

    alt15_uist.pdf

    @inproceedings{alt15_uist, title = {GravitySpot: Guiding Users in Front of Public Displays Using On-Screen Visual Cues}, author = {Alt, Florian and Bulling, Andreas and Gravanis, Gino and Buschek, Daniel}, year = {2015}, pages = {47-56}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, doi = {10.1145/2807442.2807490} }
12. 

    Tackling Challenges of Interactive Public Displays using Gaze

    Mohamed Khamis, Andreas Bulling, Florian Alt

    Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 763-766, 2015.

    Abstract Links BibTeX Project

    Falling hardware prices led to a widespread use of public displays. Common interaction techniques for such displays currently include touch, mid-air, or smartphone-based interaction. While these techniques are well understood from a technical perspective, several remaining challenges hinder the uptake of interactive displays among passersby. In this paper we propose addressing major public display challenges through gaze as a novel interaction modality. We discuss why gaze-based interaction can tackle these challenges effectively and discuss how solutions can be technically realized. Furthermore, we summarize state-of-the-art eye tracking techniques that show particular promise in the area of public displays.

    doi:10.1145/2800835.2807951

    khamis15_ubicomp.pdf

    @inproceedings{khamis15_ubicomp, title = {Tackling Challenges of Interactive Public Displays using Gaze}, author = {Khamis, Mohamed and Bulling, Andreas and Alt, Florian}, year = {2015}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, doi = {10.1145/2800835.2807951}, pages = {763-766} }
13. 

    A Field Study on Spontaneous Gaze-based Interaction with a Public Display using Pursuits

    Mohamed Khamis, Florian Alt, Andreas Bulling

    Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 865-874, 2015.

    Abstract Links BibTeX Project

    Smooth pursuit eye movements were recently introduced as a promising technique for calibration-free and thus spontaneous and natural gaze interaction. While pursuits have been evaluated in controlled laboratory studies, the technique has not yet been evaluated with respect to usability in the wild. We report on a field study in which we deployed a game on a public display where participants used pursuits to select fish moving in linear and circular trajectories at different speeds. The study ran for two days in a busy computer lab resulting in a total of 56 interactions. Results from our study show that linear trajectories are statistically faster to select via pursuits than circular trajectories. We also found that pursuits is well perceived by users who find it fast and responsive.

    doi:10.1145/2800835.2804335

    khamis15_ubicomp_2.pdf

    @inproceedings{khamis15_ubicomp_2, title = {A Field Study on Spontaneous Gaze-based Interaction with a Public Display using Pursuits}, author = {Khamis, Mohamed and Alt, Florian and Bulling, Andreas}, year = {2015}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, doi = {10.1145/2800835.2804335}, pages = {865-874} }
14. 

    Graphical Passwords in the Wild – Understanding How Users Choose Pictures and Passwords in Image-based Authentication Schemes

    Florian Alt, Stefan Schneegass, Alireza Sahami, Mariam Hassib, Andreas Bulling

    Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), pp. 316-322, 2015.

    Abstract Links BibTeX Project

    Common user authentication methods on smartphones, such as lock patterns, PINs, or passwords, impose a trade-off between security and password memorability. Image-based passwords were proposed as a secure and usable alternative. As of today, however, it remains unclear how such schemes are used in the wild. We present the first study to investigate how image-based passwords are used over long periods of time in the real world. Our analyses are based on data from 2318 unique devices collected over more than one year using a custom application released in the Android Play store. We present an in-depth analysis of what kind of images users select, how they define their passwords, and how secure these passwords are. Our findings provide valuable insights into real-world use of image-based passwords and inform the design of future graphical authentication schemes.

    doi:10.1145/2785830.2785882

    alt15_mobilehci.pdf

    @inproceedings{alt15_mobilehci, title = {Graphical Passwords in the Wild -- Understanding How Users Choose Pictures and Passwords in Image-based Authentication Schemes}, author = {Alt, Florian and Schneegass, Stefan and Sahami, Alireza and Hassib, Mariam and Bulling, Andreas}, year = {2015}, pages = {316-322}, booktitle = {Proc. International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI)}, doi = {10.1145/2785830.2785882} }
15. 

    Scene viewing and gaze analysis during phonetic segmentation tasks

    Arif Khan, Ingmar Steiner, Ross Macdonald, Yusuke Sugano, Andreas Bulling

    Proc. European Conference on Eye Movements (ECEM), 2015.

    Links BibTeX Project

    khan15_ecem.pdf

    @inproceedings{khan15_ecem, title = {Scene viewing and gaze analysis during phonetic segmentation tasks}, author = {Khan, Arif and Steiner, Ingmar and Macdonald, Ross and Sugano, Yusuke and Bulling, Andreas}, year = {2015}, booktitle = {Proc. European Conference on Eye Movements (ECEM)} }
16. 

    On the interplay between spontaneous spoken instructions and human visual behaviour in an indoor guidance task

    Nikolina Koleva, Sabrina Hoppe, Mohammed Mehdi Moniri, Maria Staudte, Andreas Bulling

    Proc. Annual Meeting of the Cognitive Science Society (CogSci), 2015.

    Abstract Links BibTeX Project

    We report on an indoor guidance study to explore the inter- play between spontaneous spoken instructions and listeners’ eye movement behaviour. The study involves a remote speaker (the instructor) to verbally guide a listener (the walker) to com- plete nine everyday tasks in different locations inside a room. We collect a multi-modal dataset of 12 pairs of users consist- ing of egocentric videos from the listener’s perspective, their gaze data, and instructors’ verbal instructions. We analyse the impact on instructions and listener gaze when the speaker can see 1) only the egocentric video, 2) the video and the point of gaze, or 3) the video and gaze with artificial noise. Our re- sults show that gaze behaviour varies significantly after (but hardly before) instructions and that speakers give more nega- tive feedback when listener gaze is available. These findings suggest that although speakers use gaze information as an in- dication of what referent the listener is effectively considering, this does not lead listeners to deliberately use their gaze as a pointer even when this is potentially beneficial for the task.

    koleva15_cogsci.pdf

    @inproceedings{koleva15_cogsci, title = {On the interplay between spontaneous spoken instructions and human visual behaviour in an indoor guidance task}, author = {Koleva, Nikolina and Hoppe, Sabrina and Moniri, Mohammed Mehdi and Staudte, Maria and Bulling, Andreas}, year = {2015}, booktitle = {Proc. Annual Meeting of the Cognitive Science Society (CogSci)} }
17. 

    GazeProjector: Accurate Gaze Estimation and Seamless Gaze Interaction Across Multiple Displays

    Christian Lander, Sven Gehring, Antonio Krüger, Sebastian Boring, Andreas Bulling

    Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 395-404, 2015.

    Abstract Links BibTeX Project

    Mobile gaze-based interaction with multiple displays may occur from arbitrary positions and orientations. However, maintaining high gaze estimation accuracy in such situations remains a significant challenge. In this paper, we present GazeProjector, a system that combines (1) natural feature tracking on displays to determine the mobile eye tracker’s position relative to a display with (2) accurate point-of-gaze estimation. GazeProjector allows for seamless gaze estimation and interaction on multiple displays of arbitrary sizes independently of the user’s position and orientation to the display. In a user study with 12 participants we compare GazeProjector to established methods (here: visual on-screen markers and a state-of-the-art video-based motion capture system). We show that our approach is robust to varying head poses, orientations, and distances to the display, while still providing high gaze estimation accuracy across multiple displays without re-calibration for each variation. Our system represents an important step towards the vision of pervasive gaze-based interfaces.

    doi:10.1145/2807442.2807479

    lander15_uist.pdf

    @inproceedings{lander15_uist, title = {GazeProjector: Accurate Gaze Estimation and Seamless Gaze Interaction Across Multiple Displays}, author = {Lander, Christian and Gehring, Sven and Kr{\"{u}}ger, Antonio and Boring, Sebastian and Bulling, Andreas}, year = {2015}, pages = {395-404}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, doi = {10.1145/2807442.2807479} }
18. 

    Emotion recognition from embedded bodily expressions and speech during dyadic interactions

    Philipp Müller, Sikandar Amin, Prateek Verma, Mykhaylo Andriluka, Andreas Bulling

    Proc. International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 663-669, 2015.

    Abstract Links BibTeX Project

    Previous work on emotion recognition from bodily expressions focused on analysing such expressions in isolation, of individuals or in controlled settings, from a single camera view, or required intrusive motion tracking equipment. We study the problem of emotion recognition from bodily expressions and speech during dyadic (person-person) interactions in a real kitchen instrumented with ambient cameras and microphones. We specifically focus on bodily expressions that are embedded in regular interactions and background activities and recorded without human augmentation to increase naturalness of the expressions. We present a human-validated dataset that contains 224 high-resolution, multi-view video clips and audio recordings of emotionally charged interactions between eight couples of actors. The dataset is fully annotated with categorical labels for four basic emotions (anger, happiness, sadness, and surprise) and continuous labels for valence, activation, power, and anticipation provided by five annotators for each actor. We evaluate vision and audio-based emotion recognition using dense trajectories and a standard audio pipeline and provide insights into the importance of different body parts and audio features for emotion recognition.

    doi:10.1109/ACII.2015.7344640

    mueller15_acii.pdf

    @inproceedings{mueller15_acii, title = {Emotion recognition from embedded bodily expressions and speech during dyadic interactions}, author = {M{\"{u}}ller, Philipp and Amin, Sikandar and Verma, Prateek and Andriluka, Mykhaylo and Bulling, Andreas}, year = {2015}, pages = {663-669}, doi = {10.1109/ACII.2015.7344640}, booktitle = {Proc. International Conference on Affective Computing and Intelligent Interaction (ACII)} }
19. 

    Prediction of Search Targets From Fixations in Open-world Settings

    Hosnieh Sattar, Sabine Müller, Mario Fritz, Andreas Bulling

    Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 981-990, 2015.

    Abstract Links BibTeX Project

    Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets. In this work we go beyond the state of the art by studying search target prediction in an open-world setting in which we no longer assume that we have fixation data to train for the search targets. We present a dataset containing fixation data of 18 users searching for natural images from three image categories within synthesised image collages of about 80 images. In a closed-world baseline experiment we show that we can predict the correct target image out of a candidate set of five images. We then present a new problem formulation for search target prediction in the open-world setting that is based on learning compatibilities between fixations and potential targets.

    doi:10.1109/CVPR.2015.7298700

    sattar15_cvpr.pdf

    @inproceedings{sattar15_cvpr, author = {Sattar, Hosnieh and M{\"{u}}ller, Sabine and Fritz, Mario and Bulling, Andreas}, title = {Prediction of Search Targets From Fixations in Open-world Settings}, booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2015}, pages = {981-990}, doi = {10.1109/CVPR.2015.7298700} }
20. 

    Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models

    Julian Steil, Andreas Bulling

    Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 75-85, 2015.

    Abstract Links BibTeX Project

    Human visual behaviour has significant potential for activity recognition and computational behaviour analysis, but previous works focused on supervised methods and recognition of predefined activity classes based on short-term eye movement recordings. We propose a fully unsupervised method to discover users’ everyday activities from their long-term visual behaviour. Our method combines a bag-of-words representation of visual behaviour that encodes saccades, fixations, and blinks with a latent Dirichlet allocation (LDA) topic model. We further propose different methods to encode saccades for their use in the topic model. We evaluate our method on a novel long-term gaze dataset that contains full-day recordings of natural visual behaviour of 10 participants (more than 80 hours in total). We also provide annotations for eight sample activity classes (outdoor, social interaction, focused work, travel, reading, computer work, watching media, eating) and periods with no specific activity. We show the ability of our method to discover these activities with performance competitive with that of previously published supervised methods.

    doi:10.1145/2750858.2807520

    steil15_ubicomp.pdf

    @inproceedings{steil15_ubicomp, author = {Steil, Julian and Bulling, Andreas}, title = {Discovery of Everyday Human Activities From Long-term Visual Behaviour Using Topic Models}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2015}, doi = {10.1145/2750858.2807520}, pages = {75-85} }
21. 

    Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

    Erroll Wood, Tadas Baltrušaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling

    Proc. IEEE International Conference on Computer Vision (ICCV), pp. 3756-3764, 2015.

    Abstract Links BibTeX Project

    Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model’s controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

    doi:10.1109/ICCV.2015.428

    wood15_iccv.pdf

    @inproceedings{wood15_iccv, title = {Rendering of Eyes for Eye-Shape Registration and Gaze Estimation}, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Zhang, Xucong and Sugano, Yusuke and Robinson, Peter and Bulling, Andreas}, doi = {10.1109/ICCV.2015.428}, year = {2015}, pages = {3756-3764}, booktitle = {Proc. IEEE International Conference on Computer Vision (ICCV)} }
22. 

    Self-Calibrating Head-Mounted Eye Trackers Using Egocentric Visual Saliency

    Yusuke Sugano, Andreas Bulling

    Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 363-372, 2015.

    Abstract Links BibTeX Project

    Head-mounted eye tracking has significant potential for gaze-based applications such as life logging, mental health monitoring, or quantified self. However, a neglected challenge for such applications is that drift in the initial person-specific eye tracker calibration, for example caused by physical activity, can severely impact gaze estimation accuracy and, thus, system performance and user experience. We first analyse calibration drift on a new dataset of natural gaze data recorded using synchronised video-based and Electrooculography-based eye trackers of 20 users performing everyday activities in a mobile setting. Based on this analysis we present a method to automatically self-calibrate head-mounted eye trackers based on a computational model of bottom-up visual saliency. Through evaluations on the dataset we show that our method is 1) effective in reducing calibration drift in calibrated eye trackers and 2) given sufficient data, can achieve competitive gaze estimation accuracy to a calibrated eye tracker without any manual calibration.

    doi:10.1145/2807442.2807445

    sugano15_uist.pdf

    @inproceedings{sugano15_uist, title = {Self-Calibrating Head-Mounted Eye Trackers Using Egocentric Visual Saliency}, author = {Sugano, Yusuke and Bulling, Andreas}, year = {2015}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, doi = {10.1145/2807442.2807445}, pages = {363-372} }

Book Chapters

1. Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)

   Peter Kiefer, Yanxia Zhang, Andreas Bulling

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2015.

   Abstract Links BibTeX Project

   Previous work on eye tracking and eye-based human-computer interfaces mainly concentrated on making use of the eyes in traditional desktop settings. With the recent growth of interest in smart eyewear and low-cost mobile eye trackers, gaze-based techniques for mobile computing is becoming increasingly important. PETMEI 2015 focuses on the pervasive eye tracking paradigm as a trailblazer for mobile eye-based interaction and eye-based context-awareness. We want to stimulate and explore the creativity of these communities with respect to the implications, key research challenges, and new applications for pervasive eye tracking in ubiquitous computing. The long-term goal is to create a strong interdisciplinary research community linking these fields together and to establish the workshop as the premier forum for research on pervasive eye tracking.

   doi:10.1145/2800835.2807960

   @inbook{kiefer15_petmei, author = {Kiefer, Peter and Zhang, Yanxia and Bulling, Andreas}, title = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, year = {2015}, doi = {10.1145/2800835.2807960}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)} }

Technical Reports

1. 

   Appearance-Based Gaze Estimation in the Wild

   Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling

   arXiv:1504.02863, 2015.

   Abstract Links BibTeX Project

   Appearance-based gaze estimation is believed to work well in real-world settings, but existing datasets have been collected under controlled laboratory conditions and methods have been not evaluated across multiple datasets. In this work we study appearance-based gaze estimation in the wild. We present the MPIIGaze dataset that contains 213,659 images we collected from 15 participants during natural everyday laptop use over more than three months. Our dataset is significantly more variable than existing ones with respect to appearance and illumination. We also present a method for in-the-wild appearance-based gaze estimation using multimodal convolutional neural networks that significantly outperforms state-of-the art methods in the most challenging cross-dataset evaluation. We present an extensive evaluation of several state-of-the-art image-based gaze estimation algorithms on three current datasets, including our own. This evaluation provides clear insights and allows us to identify key research challenges of gaze estimation in the wild.

   zhang15_arxiv.pdf

   https://arxiv.org/abs/1504.02863

   @techreport{zhang15_arxiv, title = {Appearance-Based Gaze Estimation in the Wild}, author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas}, year = {2015}, code = {https://arxiv.org/abs/1504.02863} }
2. 

   GazeProjector: Location-independent gaze interaction on and across multiple displays

   Christian Lander, Sven Gehring, Antonio Krüger, Sebastian Boring, Andreas Bulling

   DFKI Research Reports, 2015.

   Abstract Links BibTeX Project

   Mobile gaze-based interaction with multiple displays may occur from arbitrary positions and orientations. However, maintaining high gaze estimation accuracy still represents a significant challenge. To address this, we present GazeProjector, a system that combines accurate point-of-gaze estimation with natural feature tracking on displays to determine the mobile eye tracker’s position relative to a display. The detected eye positions are transformed onto that display allowing for gaze-based interaction. This allows for seamless gaze estimation and interaction on (1) multiple displays of arbitrary sizes, (2) independently of the user’s position and orientation to the display. In a user study with 12 participants we compared GazeProjector to existing well- established methods such as visual on-screen markers and a state-of-the-art motion capture system. Our results show that our approach is robust to varying head poses, orientations, and distances to the display, while still providing high gaze estimation accuracy across multiple displays without re-calibration. The system represents an important step towards the vision of pervasive gaze-based interfaces.

   lander15_techrep.pdf

   @techreport{lander15_techrep, author = {Lander, Christian and Gehring, Sven and Kr{\"{u}}ger, Antonio and Boring, Sebastian and Bulling, Andreas}, title = {GazeProjector: Location-independent gaze interaction on and across multiple displays}, volume = {1}, year = {2015}, institution = {German Research Center for Artificial Intelligence (DFKI)} }
3. 

   Prediction of Search Targets From Fixations in Open-world Settings

   Hosnieh Sattar, Sabine Müller, Mario Fritz, Andreas Bulling

   arXiv:1502.05137, 2015.

   Abstract Links BibTeX Project

   Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets. In this work we go beyond the state of the art by studying search target prediction in an open-world setting in which we no longer assume that we have fixation data to train for the search targets. We present a dataset containing fixation data of 18 users searching for natural images from three image categories within synthesised image collages of about 80 images. In a closed-world baseline experiment we show that we can predict the correct target image out of a candidate set of five images. We then present a new problem formulation for search target prediction in the open-world setting that is based on learning compatibilities between fixations and potential targets.

   sattar15_arxiv.pdf

   https://arxiv.org/abs/1502.05137

   @techreport{sattar15_arxiv, author = {Sattar, Hosnieh and M{\"{u}}ller, Sabine and Fritz, Mario and Bulling, Andreas}, title = {Prediction of Search Targets From Fixations in Open-world Settings}, year = {2015}, code = {https://arxiv.org/abs/1502.05137} }
4. 

   GazeDPM: Early Integration of Gaze Information in Deformable Part Models

   Iaroslav Shcherbatyi, Andreas Bulling, Mario Fritz

   arXiv:1505.05753, 2015.

   Abstract Links BibTeX Project

   An increasing number of works explore collaborative human-computer systems in which human gaze is used to enhance computer vision systems. For object detection these efforts were so far restricted to late integration approaches that have inherent limitations, such as increased precision without increase in recall. We propose an early integration approach in a deformable part model, which constitutes a joint formulation over gaze and visual data. We show that our GazeDPM method improves over the state-of-the-art DPM baseline by 4% and a recent method for gaze-supported object detection by 3% on the public POET dataset. Our approach additionally provides introspection of the learnt models, can reveal salient image structures, and allows us to investigate the interplay between gaze attracting and repelling areas, the importance of view-specific models, as well as viewers’ personal biases in gaze patterns. We finally study important practical aspects of our approach, such as the impact of using saliency maps instead of real fixations, the impact of the number of fixations, as well as robustness to gaze estimation error.

   shcherbatyi15_arxiv.pdf

   https://arxiv.org/abs/1505.05753

   @techreport{shcherbatyi15_arxiv, title = {GazeDPM: Early Integration of Gaze Information in Deformable Part Models}, author = {Shcherbatyi, Iaroslav and Bulling, Andreas and Fritz, Mario}, year = {2015}, code = {https://arxiv.org/abs/1505.05753} }
5. 

   Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

   Erroll Wood, Tadas Baltrušaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling

   arXiv:1505.05916, 2015.

   Abstract Links BibTeX Project

   Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model’s controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

   wood15_arxiv.pdf

   https://arxiv.org/abs/1505.05916

   @techreport{wood15_arxiv, title = {Rendering of Eyes for Eye-Shape Registration and Gaze Estimation}, author = {Wood, Erroll and Baltru{\v{s}}aitis, Tadas and Zhang, Xucong and Sugano, Yusuke and Robinson, Peter and Bulling, Andreas}, year = {2015}, code = {https://arxiv.org/abs/1505.05916} }
6. 

   Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments

   Marc Tonsen, Xucong Zhang, Yusuke Sugano, Andreas Bulling

   arXiv:1511.05768, 2015.

   Abstract Links BibTeX Project

   We present labelled pupils in the wild (LPW), a novel dataset of 66 high-quality, high-speed eye region videos for the development and evaluation of pupil detection algorithms. The videos in our dataset were recorded from 22 participants in everyday locations at about 95 FPS using a state-of-the-art dark-pupil head-mounted eye tracker. They cover people with different ethnicities, a diverse set of everyday indoor and outdoor illumination environments, as well as natural gaze direction distributions. The dataset also includes participants wearing glasses, contact lenses, as well as make-up. We benchmark five state-of-the-art pupil detection algorithms on our dataset with respect to robustness and accuracy. We further study the influence of image resolution, vision aids, as well as recording location (indoor, outdoor) on pupil detection performance. Our evaluations provide valuable insights into the general pupil detection problem and allow us to identify key challenges for robust pupil detection on head-mounted eye trackers.

   tonsen15_arxiv.pdf

   https://arxiv.org/abs/1511.05768

   @techreport{tonsen15_arxiv, title = {Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments}, author = {Tonsen, Marc and Zhang, Xucong and Sugano, Yusuke and Bulling, Andreas}, year = {2015}, code = {https://arxiv.org/abs/1511.05768} }

2014

Journal Articles

1. On the potential of human visual behaviour for memory augmentation and life logging

   Andreas Bulling

   Dagstuhl Reports/14362, , 2014.

   BibTeX Project

   @article{bulling14_dagstuhl, title = {On the potential of human visual behaviour for memory augmentation and life logging}, author = {Bulling, Andreas}, journal = {Dagstuhl Reports/14362}, year = {2014} }
2. 

   A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors

   Andreas Bulling, Ulf Blanke, Bernt Schiele

   ACM Computing Surveys, 46(3), pp. 33:1-33:33, 2014.

   Abstract Links BibTeX Project

   The last 20 years have seen an ever increasing research activity in the field of human activity recognition. With activity recognition having considerably matured so did the number of challenges in designing, implementing and evaluating activity recognition systems. This tutorial aims to provide a comprehensive hands-on introduction for newcomers to the field of human activity recognition. It specifically focuses on activity recognition using on-body inertial sensors. We first discuss the key research challenges that human activity recognition shares with general pattern recognition and identify those challenges that are specific to human activity recognition. We then describe the concept of an activity recognition chain (ARC) as a general-purpose framework for designing and evaluating activity recognition systems. We detail each component of the framework, provide references to related research and introduce the best practise methods developed by the activity recognition research community. We conclude with the educational example problem of recognising different hand gestures from inertial sensors attached to the upper and lower arm. We illustrate how each component of this framework can be implemented for this specific activity recognition problem and demonstrate how different implementations compare and how they impact overall recognition performance.

   doi:10.1145/2499621

   bulling14_csur.pdf

   @article{bulling14_csur, author = {Bulling, Andreas and Blanke, Ulf and Schiele, Bernt}, title = {A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors}, journal = {ACM Computing Surveys}, volume = {46}, number = {3}, year = {2014}, pages = {33:1-33:33}, doi = {10.1145/2499621} }
3. 

   Cognition-Aware Computing

   Andreas Bulling, Thorsten O. Zander

   IEEE Pervasive Computing, 13(3), pp. 80-83, 2014.

   Abstract Links BibTeX Project

   Despite significant advances in context sensing and inference since its inception in the late 1990s, context-aware computing still doesn’t implement a holistic view of all covert aspects of the user state. Here, the authors introduce the concept of cognitive context as an extension to the current notion of context with a cognitive dimension. They argue that visual behavior and brain activity are two promising sensing modalities for assessing the cognitive context and thus the development of cognition-aware computing systems.

   doi:10.1109/mprv.2014.42

   bulling14_pcm.pdf

   @article{bulling14_pcm, author = {Bulling, Andreas and Zander, Thorsten O.}, keywords = {bioinformatics, cognition, cognition-aware computing, Context modeling, Context-aware computing, electroencephalography, intelligent systems, Pervasive computing, Sensors, tracking, Visualization}, title = {Cognition-Aware Computing}, journal = {IEEE Pervasive Computing}, volume = {13}, number = {3}, year = {2014}, pages = {80-83}, doi = {10.1109/mprv.2014.42} }

Conference Papers

1. 

   Cross-Device Gaze-Supported Point-to-Point Content Transfer

   Jayson Turner, Andreas Bulling, Jason Alexander, Hans Gellersen

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 19-26, 2014.

   Abstract Links BibTeX Project

   Within a pervasive computing environment, we see content on shared displays that we wish to acquire and use in a specific way i.e., with an application on a personal device, transferring from point-to-point. The eyes as input can indicate intention to interact with a service, providing implicit pointing as a result. In this paper we investigate the use of gaze and manual input for the positioning of gaze-acquired content on personal devices. We evaluate two main techniques, (1) Gaze Positioning, transfer of content using gaze with manual input to confirm actions, (2) Manual Positioning, content is selected with gaze but final positioning is performed by manual input, involving a switch of modalities from gaze to manual input. A first user study compares these techniques applied to direct and indirect manual input configurations, a tablet with touch input and a laptop with mouse input. A second study evaluated our techniques in an application scenario involving distractor targets. Our overall results showed general acceptance and understanding of all conditions, although there were clear individual user preferences dependent on familiarity and preference toward gaze, touch, or mouse input.

   doi:10.1145/2578153.2578155

   turner14_etra.pdf

   @inproceedings{turner14_etra, author = {Turner, Jayson and Bulling, Andreas and Alexander, Jason and Gellersen, Hans}, title = {Cross-Device Gaze-Supported Point-to-Point Content Transfer}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2014}, pages = {19-26}, doi = {10.1145/2578153.2578155} }
2. 

   Ubic: Bridging the gap between digital cryptography and the physical world

   Mark Simkin, Dominique Schröder, Andreas Bulling, Mario Fritz

   Proc. European Symposium on Research in Computer Security (ESORICS), pp. 56-75, 2014.

   Abstract Links BibTeX Project

   Advances in computing technology increasingly blur the boundary between the digital domain and the physical world. Although the research community has developed a large number of cryptographic primitives and has demonstrated their usability in all-digital communication, many of them have not yet made their way into the real world due to usability aspects. We aim to make another step towards a tighter integration of digital cryptography into real world interactions. We describe Ubic, a framework that allows users to bridge the gap between digital cryptography and the physical world. Ubic relies on head-mounted displays, like Google Glass, resource-friendly computer vision techniques as well as mathematically sound cryptographic primitives to provide users with better security and privacy guarantees. The framework covers key cryptographic primitives, such as secure identification, document verification using a novel secure physical document format, as well as content hiding. To make a contribution of practical value, we focused on making Ubic as simple, easily deployable, and user friendly as possible.

   doi:10.1007/978-3-319-11203-9_4

   simkin14_esorics.pdf

   @inproceedings{simkin14_esorics, author = {Simkin, Mark and Schr{\"{o}}der, Dominique and Bulling, Andreas and Fritz, Mario}, keywords = {authentication, content hiding, content verification, cryptography, head-mounted displays, ubiquitous, usable security}, title = {Ubic: Bridging the gap between digital cryptography and the physical world}, booktitle = {Proc. European Symposium on Research in Computer Security (ESORICS)}, year = {2014}, pages = {56-75}, doi = {10.1007/978-3-319-11203-9_4} }
3. 

   In the Blink of an Eye: Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass

   Shoya Ishimaru, Jens Weppner, Kai Kunze, Koichi Kise, Andreas Dengel, Paul Lukowicz, Andreas Bulling

   Proc. Augmented Human International Conference (AH), pp. 15:1-15:4, 2014.

   Abstract Links BibTeX Project

   We demonstrate how information about eye blink frequency and head motion patterns derived from Google Glass sensors can be used to distinguish different types of high level activities. While it is well known that eye blink frequency is correlated with user activity, our aim is to show that (1) eye blink frequency data from an unobtrusive, commercial platform which is not a dedicated eye tracker is good enough to be useful and (2) that adding head motion patterns information significantly improves the recognition rates. The method is evaluated on a data set from an experiment containing five activity classes (reading, talking, watching TV, mathematical problem solving, and sawing) of eight participants showing 67% recognition accuracy for eye blinking only and 82% when extended with head motion patterns.

   doi:10.1145/2582051.2582066

   ishimaru14_ah.pdf

   @inproceedings{ishimaru14_ah, author = {Ishimaru, Shoya and Weppner, Jens and Kunze, Kai and Kise, Koichi and Dengel, Andreas and Lukowicz, Paul and Bulling, Andreas}, title = {In the Blink of an Eye: Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass}, booktitle = {Proc. Augmented Human International Conference (AH)}, year = {2014}, pages = {15:1-15:4}, doi = {10.1145/2582051.2582066} }
4. 

   SmudgeSafe: Geometric Image Transformations for Smudge-resistant User Authentication

   Stefan Schneegass, Frank Steimle, Andreas Bulling, Florian Alt, Albrecht Schmidt

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 775-786, 2014.

   Abstract Links BibTeX Project

   Touch-enabled user interfaces have become ubiquitous, such as on ATMs or portable devices. At the same time, authentication using touch input is problematic, since finger smudge traces may allow attackers to reconstruct passwords. We present SmudgeSafe, an authentication system that uses random geometric image transformations, such as translation, rotation, scaling, shearing, and flipping, to increase the security of cued-recall graphical passwords. We describe the design space of these transformations and report on two user studies: A lab-based security study involving 20 participants in attacking user-defined passwords, using high quality pictures of real smudge traces captured on a mobile phone display; and an in-the-field usability study with 374 participants who generated more than 130,000 logins on a mobile phone implementation of SmudgeSafe. Results show that SmudgeSafe significantly increases security compared to authentication schemes based on PINs and lock patterns, and exhibits very high learnability, efficiency, and memorability.

   doi:10.1145/2632048.2636090

   schneegass14_ubicomp.pdf

   @inproceedings{schneegass14_ubicomp, author = {Schneegass, Stefan and Steimle, Frank and Bulling, Andreas and Alt, Florian and Schmidt, Albrecht}, title = {SmudgeSafe: Geometric Image Transformations for Smudge-resistant User Authentication}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2014}, pages = {775-786}, doi = {10.1145/2632048.2636090} }
5. 

   Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction

   Moritz Kassner, William Patera, Andreas Bulling

   Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 1151-1160, 2014.

   Abstract Links BibTeX Project

   In this paper we present Pupil – an accessible, affordable, and extensible open source platform for pervasive eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight eye tracking headset, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a processing pipeline latency of only 0.045 seconds.

   doi:10.1145/2638728.2641695

   kassner14_ubicomp.pdf

   @inproceedings{kassner14_ubicomp, title = {Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction}, author = {Kassner, Moritz and Patera, William and Bulling, Andreas}, year = {2014}, doi = {10.1145/2638728.2641695}, booktitle = {Adj. Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, pages = {1151-1160} }
6. 

   EyeTab: Model-based gaze estimation on unmodified tablet computers

   Erroll Wood, Andreas Bulling

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 207-210, 2014.

   Abstract Links BibTeX Project

   Despite the widespread use of mobile phones and tablets, hand-held portable devices have only recently been identified as a promising platform for gaze-aware applications. Estimating gaze on portable devices is challenging given their limited computational resources, low quality integrated front-facing RGB cameras, and small screens to which gaze is mapped. In this paper we present EyeTab, a model-based approach for binocular gaze estimation that runs entirely on an unmodified tablet. EyeTab builds on set of established image processing and computer vision algorithms and adapts them for robust and near-realtime gaze estimation. A technical prototype evaluation with eight participants in a normal indoors office setting shows that EyeTab achieves an average gaze estimation accuracy of 6.88° of visual angle at 12 frames per second.

   doi:10.1145/2578153.2578185

   wood14_etra.pdf

   @inproceedings{wood14_etra, author = {Wood, Erroll and Bulling, Andreas}, title = {EyeTab: Model-based gaze estimation on unmodified tablet computers}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2014}, pages = {207-210}, doi = {10.1145/2578153.2578185} }
7. 

   Test-time Adaptation for 3D Human Pose Estimation

   Sikandar Amin, Mykhaylo Andriluka, Philipp Müller, Andreas Bulling

   Proc. German Conference on Pattern Recognition (GCPR), pp. 253-264, 2014.

   Abstract Links BibTeX Project

   In this paper we consider the task of articulated 3D human pose estimation in challenging scenes with dynamic background and multiple people. Initial progress on this task has been achieved building on discriminatively trained part-based models that deliver a set of 2D body pose candidates that are then subsequently refined by reasoning in 3D [1, 4, 5]. The performance of such methods is limited by the performance of the underlying 2D pose estimation approaches. In this paper we explore a way to boost the performance of 2D pose estimation based on the output of the 3D pose reconstruction process, thus closing the loop in the pose estimation pipeline. We build our approach around a component that is able to identify true positive pose estimation hypotheses with high confidence. We then either retrain 2D pose estimation models using such highly confident hypotheses as additional training examples, or we use similarity o these hypotheses as a cue for 2D pose estimation. We consider a number of features that can be used for assessing the confidence of the pose estimation results. The strongest feature in our comparison corresponds to the ensemble greement on the 3D pose output. We evaluate our approach on two publicly available datasets improving over state of the art in each case.

   doi:10.1007/978-3-319-11752-2_20

   amin14_gcpr.pdf

   @inproceedings{amin14_gcpr, author = {Amin, Sikandar and Andriluka, Mykhaylo and M{\"{u}}ller, Philipp and Bulling, Andreas}, title = {Test-time Adaptation for 3D Human Pose Estimation}, booktitle = {Proc. German Conference on Pattern Recognition (GCPR)}, year = {2014}, pages = {253-264}, doi = {10.1007/978-3-319-11752-2_20} }
8. 

   GazeHorizon: Enabling Passers-by to Interact with Public Displays by Gaze

   Yanxia Zhang, Hans Jörg Müller, Ming Ki Chong, Andreas Bulling, Hans Gellersen

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 559-563, 2014.

   Abstract Links BibTeX Project

   Public displays can be made interactive by adding gaze control. However, gaze interfaces do not offer any physical affordance, and require users to move into a tracking range. We present GazeHorizon, a system that provides interactive assistance to enable passers-by to walk up to a display and to navigate content using their eyes only. The system was developed through field studies culminating in a four-day deployment in a public environment. Our results show that novice users can be facilitated to successfully use gaze control by making them aware of the interface at first glance and guiding them interactively into the tracking range.

   doi:10.1145/2632048.2636071

   zhang14_ubicomp.pdf

   @inproceedings{zhang14_ubicomp, author = {Zhang, Yanxia and M{\"{u}}ller, Hans J{\"{o}}rg and Chong, Ming Ki and Bulling, Andreas and Gellersen, Hans}, title = {GazeHorizon: Enabling Passers-by to Interact with Public Displays by Gaze}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2014}, pages = {559-563}, doi = {10.1145/2632048.2636071} }
9. 

   Pupil-Canthi-Ratio: A Calibration-Free Method for Tracking Horizontal Gaze Direction

   Yanxia Zhang, Andreas Bulling, Hans Gellersen

   Proc. International Conference on Advanced Visual Interfaces (AVI), pp. 129-132, 2014.

   Abstract Links BibTeX Project

   Eye tracking is compelling for hands-free interaction with pervasive displays. However, most existing eye tracking systems require specialised hardware and explicit calibrations of equipment and individual users, which inhibit their widespread adoption. In this work, we present a light-weight and calibration-free gaze estimation method that leverages only an off-the-shelf camera to track users’ gaze horizontally. We introduce pupil-canthi-ratio (PCR), a novel measure for estimating gaze directions. By using the displacement vector between the inner eye corner and the pupil centre of an eye, PCR is calculated as the ratio of the displacement vectors from both eyes. We establish a mapping between PCR to gaze direction by Gaussian process regression, which inherently infers averted horizontal gaze directions of users. We present a study to identify the characteristics of PCR. The results show that PCR achieved an average accuracy of 3.9 degrees across different people. Finally, we show examples of real-time applications of PCR that allow users to interact with a display by moving only their eyes.

   doi:10.1145/2598153.2598186

   zhang14_avi.pdf

   @inproceedings{zhang14_avi, author = {Zhang, Yanxia and Bulling, Andreas and Gellersen, Hans}, title = {Pupil-Canthi-Ratio: A Calibration-Free Method for Tracking Horizontal Gaze Direction}, booktitle = {Proc. International Conference on Advanced Visual Interfaces (AVI)}, year = {2014}, pages = {129-132}, doi = {10.1145/2598153.2598186} }

Book Chapters

1. 

   Eye Tracking and Eye-Based Human-Computer Interaction

   Päivi Majaranta, Andreas Bulling

   Stephen H. Fairclough, Kiel Gilleade (Eds.): Advances in Physiological Computing, Springer Publishing London, pp. 39-65, 2014.

   Abstract Links BibTeX Project

   Eye tracking has a long history in medical and psychological research as a tool for recording and studying human visual behavior. Real-time gaze-based text entry can also be a powerful means of communication and control for people with physical disa-bilities. Following recent technological advances and the advent of affordable eye trackers, there is a growing interest in pervasive at-tention-aware systems and interfaces that have the potential to rev-olutionize mainstream human-technology interaction. In this chapter, we provide an introduction to the state-of-the art in eye tracking technology and gaze estimation. We discuss challenges involved in using a perceptual organ, the eye, as an input modality. Examples of real life applications are reviewed, together with design solutions derived from research results. We also discuss how to match the user requirements and key features of different eye tracking sys-tems to find the best system for each task and application.

   doi:10.1007/978-1-4471-6392-3_3

   majaranta14_apc.pdf

   @inbook{majaranta14_apc, author = {Majaranta, P{\"{a}}ivi and Bulling, Andreas}, title = {Eye Tracking and Eye-Based Human-Computer Interaction}, booktitle = {Advances in Physiological Computing}, editor = {Fairclough, Stephen H. and Gilleade, Kiel}, year = {2014}, pages = {39-65}, publisher = {Springer Publishing London}, doi = {10.1007/978-1-4471-6392-3_3} }

Technical Reports

1. 

   Ubic: Bridging the gap between digital cryptography and the physical world

   Mark Simkin, Andreas Bulling, Mario Fritz, Dominique Schröder

   arXiv:1403.1343, 2014.

   Abstract Links BibTeX Project

   Advances in computing technology increasingly blur the boundary between the digital domain and the physical world. Although the research community has developed a large number of cryptographic primitives and has demonstrated their usability in all-digital communication, many of them have not yet made their way into the real world due to usability aspects. We aim to make another step towards a tighter integration of digital cryptography into real world interactions. We describe Ubic, a framework that allows users to bridge the gap between digital cryptography and the physical world. Ubic relies on head-mounted displays, like Google Glass, resource-friendly computer vision techniques as well as mathematically sound cryptographic primitives to provide users with better security and privacy guarantees. The framework covers key cryptographic primitives, such as secure identification, document verification using a novel secure physical document format, as well as content hiding. To make a contribution of practical value, we focused on making Ubic as simple, easily deployable, and user friendly as possible.

   simkin14_arxiv.pdf

   https://arxiv.org/abs/1403.1343

   @techreport{simkin14_arxiv, author = {Simkin, Mark and Bulling, Andreas and Fritz, Mario and Schr{\"{o}}der, Dominique}, title = {Ubic: Bridging the gap between digital cryptography and the physical world}, year = {2014}, code = {https://arxiv.org/abs/1403.1343} }
2. 

   Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction

   Moritz Kassner, William Patera, Andreas Bulling

   arXiv:1405.0006, 2014.

   Abstract Links BibTeX Project

   Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software. This limits the application areas and use of mobile eye tracking to expert users and inhibits user-driven development, customisation, and extension. In this paper we present Pupil – an accessible, affordable, and extensible open source platform for mobile eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight headset with high-resolution cameras, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface (GUI) to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a latency of the processing pipeline of only 0.045 seconds.

   kassner14_arxiv.pdf

   https://arxiv.org/abs/1405.0006

   @techreport{kassner14_arxiv, author = {Kassner, Moritz and Patera, William and Bulling, Andreas}, keywords = {eye movement, Gaze-based Interaction, Mobile Eye Tracking, Wearable Computing}, title = {Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction}, year = {2014}, institution = {Pupil Labs UG, Berlin, Germany and Max Planck Institute for Informatics, Saarbr{\"{u}}cken, Germany}, code = {https://arxiv.org/abs/1405.0006} }

2013

Journal Articles

1. 

   Using eye-tracking glasses to evaluate the effect of visual scanning training on everyday activities

   Tobias Loetscher, Michael Nicholls, Nicole Thomas, Andreas Bulling, Gayle Clarke, Allison Hayes, Celia Chen

   Brain Impairment, 14(2), pp. 354-355, 2013.

   Abstract Links BibTeX Project

   Screening for cognitive impairment may help predict neurorehabilitation outcomes. We investigated (1) the use of the ACE-R in predicting functional gain during in-patient rehabilitation, and (2) whether ACE-R scores identified patients requiring additional therapy support during their admission.

   loetscher13_bi.pdf

   @article{loetscher13_bi, title = {Using eye-tracking glasses to evaluate the effect of visual scanning training on everyday activities}, author = {Loetscher, Tobias and Nicholls, Michael and Thomas, Nicole and Bulling, Andreas and Clarke, Gayle and Hayes, Allison and Chen, Celia}, year = {2013}, journal = {Brain Impairment}, volume = {14}, number = {2}, pages = {354-355} }

Conference Papers

1. 

   MotionMA: Motion Modelling and Analysis by Demonstration

   Eduardo Velloso, Andreas Bulling, Hans Gellersen

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1309-1318, 2013.

   Abstract Links BibTeX Project

   Particularly in sports or physical rehabilitation, users have to perform body movements in a specific manner for the exercises to be most effective. It remains a challenge for experts to specify how to perform such movements so that an automated system can analyse further performances of it. In a user study with 10 participants we show that experts’ explicit estimates do not correspond to their performances. To address this issue we present MotionMA, a system that: (1) automatically extracts a model of movements demonstrated by one user, e.g. a trainer, (2) assesses the performance of other users repeating this movement in real time, and (3) provides real-time feedback on how to improve their performance. We evaluated the system in a second study in which 10 other participants used the system to demonstrate arbitrary movements. Our results demonstrate that MotionMA is able to extract an accurate movement model to spot mistakes and variations in movement execution.

   doi:10.1145/2470654.2466171

   velloso13_chi.pdf

   @inproceedings{velloso13_chi, author = {Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans}, title = {MotionMA: Motion Modelling and Analysis by Demonstration}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2013}, pages = {1309-1318}, doi = {10.1145/2470654.2466171} }
2. 

   Pursuits: eye-based interaction with moving targets

   Mélodie Vidal, Ken Pfeuffer, Andreas Bulling, Hans Gellersen

   Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 3147–3150, 2013.

   Abstract Links BibTeX Project

   Eye-based interaction has commonly been based on estimation of eye gaze direction, to locate objects for interaction. We introduce Pursuits, a novel and very different eye tracking method that instead is based on following the trajectory of eye movement and comparing this with trajectories of objects in the field of view. Because the eyes naturally follow the trajectory of moving objects of interest, our method is able to detect what the user is looking at, by matching eye movement and object movement. We illustrate Pursuits with three applications that demonstrate how the method facilitates natural interaction with moving targets.

   doi:10.1145/2468356.2479632

   vidal13_chi.pdf

   @inproceedings{vidal13_chi, author = {Vidal, M{\'{e}}lodie and Pfeuffer, Ken and Bulling, Andreas and Gellersen, Hans}, keywords = {eye gaze, natural user interface, smooth pursuit eye movement}, title = {Pursuits: eye-based interaction with moving targets}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2013}, pages = {3147--3150}, doi = {10.1145/2468356.2479632} }
3. 

   Pursuits: Spontaneous Interaction with Displays based on Smooth Pursuit Eye Movement and Moving Targets

   Mélodie Vidal, Andreas Bulling, Hans Gellersen

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 439-448, 2013.

   Abstract Links BibTeX Project

   Although gaze is an attractive modality for pervasive interactions, the real-world implementation of eye-based interfaces poses significant challenges, such as calibration. We present Pursuits, an innovative interaction technique that enables truly spontaneous interaction with eye-based interfaces. A user can simply walk up to the screen and readily interact with moving targets. Instead of being based on gaze location, Pursuits correlates eye pursuit movements with objects dynamically moving on the interface. We evaluate the influence of target speed, number and trajectory and develop guidelines for designing Pursuits-based interfaces. We then describe six realistic usage scenarios and implement three of them to evaluate the method in a usability study and a field study. Our results show that Pursuits is a versatile and robust technique and that users can interact with Pursuits-based interfaces without prior knowledge or preparation phase.

   doi:10.1145/2468356.2479632

   vidal13_ubicomp.pdf

   @inproceedings{vidal13_ubicomp, author = {Vidal, M{\'{e}}lodie and Bulling, Andreas and Gellersen, Hans}, title = {Pursuits: Spontaneous Interaction with Displays based on Smooth Pursuit Eye Movement and Moving Targets}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2013}, pages = {439-448}, doi = {10.1145/2468356.2479632} }
4. 

   I know what you are reading – Recognition of document types using mobile eye tracking

   Kai Kunze, Andreas Bulling, Yuzuko Utsumi, Shiga Yuki, Koichi Kise

   Proc. IEEE International Symposium on Wearable Computers (ISWC), pp. 113-116, 2013.

   Abstract Links BibTeX Project

   Reading is a ubiquitous activity that many people even perform in transit, such as while on the bus or while walking. Tracking reading enables us to gain more insights about expertise level and potential knowledge of users – towards a reading log tracking and improve knowledge acquisition. As a first step towards this vision, in this work we investigate whether different document types can be automatically detected from visual behaviour recorded using a mobile eye tracker. We present an initial recognition approach that com- bines special purpose eye movement features as well as machine learning for document type detection. We evaluate our approach in a user study with eight participants and five Japanese document types and achieve a recognition performance of 74% using user-independent training.

   doi:10.1145/2493988.2494354

   kunze13_iswc.pdf

   @inproceedings{kunze13_iswc, author = {Kunze, Kai and Bulling, Andreas and Utsumi, Yuzuko and Yuki, Shiga and Kise, Koichi}, title = {I know what you are reading -- Recognition of document types using mobile eye tracking}, booktitle = {Proc. IEEE International Symposium on Wearable Computers (ISWC)}, year = {2013}, pages = {113-116}, doi = {10.1145/2493988.2494354} }
5. 

   AutoBAP: Automatic Coding of Body Action and Posture Units from Wearable Sensors

   Eduardo Velloso, Andreas Bulling, Hans Gellersen

   Proc. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 135-140, 2013.

   Abstract Links BibTeX Project

   Manual annotation of human body movement is an integral part of research on non-verbal communication and computational behaviour analysis but also a very time-consuming and tedious task. In this paper we present AutoBAP, a system that automates the coding of bodily expressions according to the body action and posture (BAP) coding scheme. Our system takes continuous body motion and gaze behaviour data as its input. The data is recorded using a full body motion tracking suit and a wearable eye tracker. From the data our system automatically generates a labelled XML file that can be visualised and edited with off-the-shelf video annotation tools. We evaluate our system in a laboratory-based user study with six participants performing scripted sequences of 184 actions. Results from the user study show that our prototype system is able to annotate 172 out of the 274 labels of the full BAP coding scheme with good agreement with a manual annotator (Cohen’s kappa > 0.6).

   doi:10.1109/ACII.2013.29

   velloso13_acii.pdf

   @inproceedings{velloso13_acii, author = {Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans}, title = {AutoBAP: Automatic Coding of Body Action and Posture Units from Wearable Sensors}, booktitle = {Proc. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII)}, year = {2013}, pages = {135-140}, doi = {10.1109/ACII.2013.29} }
6. 

   Eye Drop: An Interaction Concept for Gaze-Supported Point-to-Point Content Transfer

   Jayson Turner, Andreas Bulling, Jason Alexander, Hans Gellersen

   Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM), pp. article 37, 2013.

   Abstract Links BibTeX Project

   The shared displays in our environment contain content that we desire. Furthermore, we often acquire content for a specific purpose, i.e., the acquisition of a phone number to place a call. We have developed a content transfer concept, Eye Drop. Eye Drop provides techniques that allow fluid content acquisition, transfer from shared displays, and local positioning on personal devices using gaze combined with manual input. The eyes naturally focus on content we desire. Our techniques use gaze to point remotely, removing the need for explicit pointing on the user’s part. A manual trigger from a personal device confirms selection. Transfer is performed using gaze or manual input to smoothly transition content to a specific location on a personal device. This work demonstrates how techniques can be applied to acquire and apply actions to content through a natural sequence of interaction. We demonstrate a proof of concept prototype through five implemented application scenarios.

   doi:10.1145/2541831.2541868

   turner13_mum.pdf

   @inproceedings{turner13_mum, author = {Turner, Jayson and Bulling, Andreas and Alexander, Jason and Gellersen, Hans}, title = {Eye Drop: An Interaction Concept for Gaze-Supported Point-to-Point Content Transfer}, booktitle = {Proc. International Conference on Mobile and Ubiquitous Multimedia (MUM)}, year = {2013}, pages = {article 37}, doi = {10.1145/2541831.2541868} }
7. 

   SideWays: A Gaze Interface for Spontaneous Interaction with Situated Displays

   Yanxia Zhang, Andreas Bulling, Hans Gellersen

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 851-860, 2013.

   Abstract Links BibTeX Project

   Eye gaze is compelling for interaction with situated displays as we naturally use our eyes to engage with them. In this work we present SideWays, a novel person-independent eye gaze interface that supports spontaneous interaction with displays: users can just walk up to a display and immediately interact using their eyes, without any prior user calibration or training. Requiring only a single off-the-shelf camera and lightweight image processing, SideWays robustly detects whether users attend to the centre of the display or cast glances to the left or right. The system supports an interaction model in which attention to the central display is the default state, while "sidelong glances" trigger input or actions. The robustness of the system and usability of the interaction model are validated in a study with 14 participants. Analysis of the participants’ strategies in performing different tasks provides insights on gaze control strategies for design of SideWays applications.

   doi:10.1145/2470654.2470775

   zhang13_chi.pdf

   @inproceedings{zhang13_chi, author = {Zhang, Yanxia and Bulling, Andreas and Gellersen, Hans}, title = {SideWays: A Gaze Interface for Spontaneous Interaction with Situated Displays}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2013}, pages = {851-860}, doi = {10.1145/2470654.2470775} }
8. 

   EyeContext: Recognition of High-level Contextual Cues from Human Visual Behaviour

   Andreas Bulling, Christian Weichel, Hans Gellersen

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 305-308, 2013.

   Abstract Links BibTeX Project

   In this work we present EyeContext, a system to infer high-level contextual cues from human visual behaviour. We conducted a user study to record eye movements of four participants over a full day of their daily life, totalling 42.5 hours of eye movement data. Participants were asked to self-annotate four non-mutually exclusive cues: social (interacting with somebody vs. no interaction), cognitive (concentrated work vs. leisure), physical (physically active vs. not active), and spatial (inside vs. outside a building). We evaluate a proof-of-concept EyeContext system that combines encoding of eye movements into strings and a spectrum string kernel support vector machine (SVM) classifier. Our results demonstrate the large information content available in long-term human visual behaviour and opens up new venues for research on eye-based behavioural monitoring and life logging.

   doi:10.1145/2470654.2470697

   bulling13_chi.pdf

   @inproceedings{bulling13_chi, author = {Bulling, Andreas and Weichel, Christian and Gellersen, Hans}, title = {EyeContext: Recognition of High-level Contextual Cues from Human Visual Behaviour}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2013}, pages = {305-308}, doi = {10.1145/2470654.2470697} }
9. 

   Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze & Touch

   Jayson Turner, Jason Alexander, Andreas Bulling, Dominik Schmidt, Hans Gellersen

   Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 170-186, 2013.

   Abstract Links BibTeX Project

   Previous work has validated the eyes and mobile input as a viable approach for pointing at, and selecting out of reach objects. This work presents Eye Pull, Eye Push, a novel interaction concept for content transfer between public and personal devices using gaze and touch. We present three techniques that enable this interaction: Eye Cut & Paste, Eye Drag & Drop, and Eye Summon & Cast. We outline and discuss several scenarios in which these techniques can be used. In a user study we found that participants responded well to the visual feedback provided by Eye Drag & Drop during object movement. In contrast, we found that although Eye Summon & Cast significantly improved performance, participants had difficulty coordinating their hands and eyes during interaction.

   doi:10.1007/978-3-642-40480-1_11

   turner13_interact.pdf

   @inproceedings{turner13_interact, author = {Turner, Jayson and Alexander, Jason and Bulling, Andreas and Schmidt, Dominik and Gellersen, Hans}, title = {Eye Pull, Eye Push: Moving Objects between Large Screens and Personal Devices with Gaze \& Touch}, booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)}, year = {2013}, pages = {170-186}, doi = {10.1007/978-3-642-40480-1_11} }
10. 

    Qualitative Activity Recognition of Weight Lifting Exercises

    Eduardo Velloso, Andreas Bulling, Hans Gellersen, Wallace Ugulino, Hugo Fuks

    Proc. Augmented Human International Conference (AH), pp. 116-123, 2013.

    Abstract Links BibTeX Project

    Research on human activity recognition has traditionally focused on discriminating between different activities, i.e. to predict “which” activity was performed at a specific point in time. The quality of executing an activity, the “how (well)”, has only received little attention so far, even though it potentially provides useful information for a large variety of applications, such as sports training. In this work we first define quality of execution and investigate three aspects that pertain to qualitative activity recognition: the problem of specifying correct execution, the automatic and robust detection of execution mistakes, and how to provide feedback on the quality of execution to the user. We illustrate our approach on the example problem of qualitatively assessing and providing feedback on weight lifting exercises. In two user studies we try out a sensor- and a model-based approach to qualitative activity recognition. Our results underline the potential of model-based assessment and the positive impact of real-time user feedback on the quality of execution.

    doi:10.1145/2459236.2459256

    velloso13_ah.pdf

    @inproceedings{velloso13_ah, author = {Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans and Ugulino, Wallace and Fuks, Hugo}, title = {Qualitative Activity Recognition of Weight Lifting Exercises}, booktitle = {Proc. Augmented Human International Conference (AH)}, year = {2013}, pages = {116-123}, doi = {10.1145/2459236.2459256} }
11. 

    Pursuit Calibration: Making Gaze Calibration Less Tedious and More Flexible

    Ken Pfeuffer, Mélodie Vidal, Jayson Turner, Andreas Bulling, Hans Gellersen

    Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 261-270, 2013.

    Abstract Links BibTeX Project

    Eye gaze is a compelling interaction modality but requires a user calibration before interaction can commence. State of the art procedures require the user to fixate on a succession of calibration markers, a task that is often experienced as difficult and tedious. We present a novel approach, pursuit calibration, that instead uses moving targets for calibration. Users naturally perform smooth pursuit eye movements when they follow a moving target, and we use correlation of eye and target movement to detect the users attention and to sample data for calibration. Because the method knows when the users is attending to a target, the calibration can be performed implicitly, which enables more flexible design of the calibration task. We demonstrate this in application examples and user studies, and show that pursuit calibration is tolerant to interruption, can blend naturally with applications, and is able to calibrate users without their awareness.

    doi:10.1145/2501988.2501998

    pfeuffer13_uist.pdf

    @inproceedings{pfeuffer13_uist, author = {Pfeuffer, Ken and Vidal, M{\'{e}}lodie and Turner, Jayson and Bulling, Andreas and Gellersen, Hans}, title = {Pursuit Calibration: Making Gaze Calibration Less Tedious and More Flexible}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, year = {2013}, pages = {261-270}, doi = {10.1145/2501988.2501998} }

Book Chapters

1. Proc. 4th Augmented Human International Conference (AH)

   Albrecht Schmidt, Andreas Bulling, Christian Holz

   Stuttgart, Germany, ACM, 2013.

   Abstract BibTeX Project

   We are very happy to present the proceedings of the 4th Augmented Human International Conference (Augmented Human 2013). Augmented Human 2013 focuses on augmenting human capabilities through technology for increased well-being and enjoyable human experience. The conference is in cooperation with ACM SIGCHI, with its proceedings to be archived in ACM’s Digital Library. With technological advances, computing has progressively moved beyond the desktop into new physical and social contexts. As physical artifacts gain new computational behaviors, they become reprogrammable, customizable, repurposable, and interoperable in rich ecologies and diverse contexts. They also become more complex, and require intense design effort in order to be functional, usable, and enjoyable. Designing such systems requires interdisciplinary thinking. Their creation must not only encompass software, electronics, and mechanics, but also the system’s physical form and behavior, its social and physical milieu, and beyond.

   @inbook{schmidt13_ah, author = {Schmidt, Albrecht and Bulling, Andreas and Holz, Christian}, title = {Proc. 4th Augmented Human International Conference (AH)}, year = {2013}, publisher = {ACM}, location = {Stuttgart, Germany}, address = {New York, NY, USA}, url = {http://dl.acm.org/citation.cfm?id=2459236} }

2. Signal processing technologies for activity-aware smart textiles

   Daniel Roggen, Andreas Bulling, Gerhard Tröster

   Tünde Kirstein (Eds.): Woodhead Publishing Series in Textiles, Woodhead Publishing Limited, pp. 329-366, 2013.

   Abstract Links BibTeX Project

   Garments made of smart textiles have an enormous potential for embedding sensors in close proximity to the body in an unobtrusive and comfortable manner. Combined with signal processing and pattern recognition technologies, complex high-level information about human behaviors or situations can be inferred from the sensor data. The goal of this chapter is to introduce the reader to the design of activity-aware systems that use body-worn sensors, such as those that can be made available through smart textiles. We start this chapter by emphasizing recent trends towards ‘wearable’ sensing and computing and we present several examples of activity-aware applications. Then we outline the role that smart textiles can play in activity-aware applications, but also the challenges that they pose. We conclude by discussing the design process followed to devise activity-aware systems: the choice of sensors, the available data processing methods, and the evaluation techniques. We discuss recent data processing methods that address the challenges resulting from the use of smart textiles.

   doi:10.1533/9780857093530.2.329

   @inbook{roggen13_wpt, author = {Roggen, Daniel and Bulling, Andreas and Tr{\"{o}}ster, Gerhard}, keywords = {Activity Recognition, context recognition, signal processing, Wearable Sensing}, title = {Signal processing technologies for activity-aware smart textiles}, booktitle = {Woodhead Publishing Series in Textiles}, editor = {Kirstein, Tünde}, number = {139}, chapter = {12}, year = {2013}, pages = {329-366}, publisher = {Woodhead Publishing Limited}, doi = {10.1533/9780857093530.2.329} }

3. Proc. 3rd International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)

   Andreas Bulling, Roman Bednarik

   Proc. European Conference on Eye Movements (ECEM), 2013.

   BibTeX Project

   @inbook{bulling13_petmei, author = {Bulling, Andreas and Bednarik, Roman}, title = {Proc. 3rd International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, year = {2013}, booktitle = {Proc. European Conference on Eye Movements (ECEM)} }

2012

Journal Articles

1. 

   Multimodal Recognition of Reading Activity in Transit Using Body-Worn Sensors

   Andreas Bulling, Jamie A. Ward, Hans Gellersen

   ACM Transactions on Applied Perception, 9(1), pp. 2:1–2:21, 2012.

   Abstract Links BibTeX Project

   Reading is one of the most well studied visual activities. Vision research traditionally focuses on understanding the perceptual and cognitive processes involved in reading. In this work we recognise reading activity by jointly analysing eye and head movements of people in an everyday environment. Eye movements are recorded using an electrooculography (EOG) system; body movements using body-worn inertial measurement units. We compare two approaches for continuous recognition of reading: String matching (STR) that explicitly models the characteristic horizontal saccades during reading, and a support vector machine (SVM) that relies on 90 eye movement features extracted from the eye movement data. We evaluate both methods in a study performed with eight participants reading while sitting at a desk, standing, walking indoors and outdoors, and riding a tram. We introduce a method to segment reading activity by exploiting the sensorimotor coordination of eye and head movements during reading. Using person-independent training, we obtain an average precision for recognising reading of 88.9% (recall 72.3%) using STR and of 87.7% (recall 87.9%) using SVM over all participants. We show that the proposed segmentation scheme improves the performance of recognising reading events by more than 24%. Our work demonstrates that the joint analysis of multiple modalities is beneficial for reading recognition and opens up discussion on the wider applicability of this recognition approach to other visual and physical activities.

   doi:10.1145/2134203.2134205

   bulling12_tap.pdf

   @article{bulling12_tap, author = {Bulling, Andreas and Ward, Jamie A. and Gellersen, Hans}, title = {Multimodal {R}ecognition of {R}eading {A}ctivity in {T}ransit {U}sing {B}ody-{W}orn {S}ensors}, journal = {ACM Transactions on Applied Perception}, volume = {9}, number = {1}, year = {2012}, pages = {2:1--2:21}, doi = {10.1145/2134203.2134205} }
2. 

   Wearable Eye Tracking for Mental Health Monitoring

   Mélodie Vidal, Jayson Turner, Andreas Bulling, Hans Gellersen

   Computer Communications, 35(11), pp. 1306-1311, 2012.

   Abstract Links BibTeX Project

   Pervasive healthcare is a promising field of research as small and unobtrusive on-body sensors become available. However, despite considerable advances in the field, current systems are limited in terms of the pathologies they can detect, particularly regarding mental disorders. In this work we propose wearable eye tracking as a new method for mental health monitoring. We provide two reviews: one of the state-of-the-art in wearable eye tracking equipment and a second one of the work in experimental psychology and clinical research on the link between eye movements and cognition. Both reviews show a significant potential of wearable eye tracking for mental health monitoring in daily life settings. This finding calls for further research on unobtrusive sensing equipment and novel algorithms for automated analysis of long-term eye movement data.

   doi:10.1016/j.comcom.2011.11.002

   vidal12_comcom.pdf

   @article{vidal12_comcom, author = {Vidal, M{\'{e}}lodie and Turner, Jayson and Bulling, Andreas and Gellersen, Hans}, title = {Wearable Eye Tracking for Mental Health Monitoring}, journal = {Computer Communications}, volume = {35}, number = {11}, year = {2012}, pages = {1306-1311}, doi = {10.1016/j.comcom.2011.11.002} }

Conference Papers

1. 

   Increasing the Security of Gaze-Based Cued-Recall Graphical Passwords Using Saliency Masks

   Andreas Bulling, Florian Alt, Albrecht Schmidt

   Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 3011-3020, 2012.

   Abstract Links BibTeX Project

   With computers being used ever more ubiquitously in situations where privacy is important, secure user authentication is a central requirement. Gaze-based graphical passwords are a particularly promising means for shoulder-surfing-resistant authentication, but selecting secure passwords remains challenging. In this paper, we present a novel gaze-based authentication scheme that makes use of cued-recall graphical pass- words on a single image. In order to increase password security, our approach uses a computational model of visual attention to mask those areas of the image that are most likely to attract visual attention. We create a realistic threat model for attacks that may occur in public settings, such as filming the user’s interaction while drawing money from an ATM. Based on a 12-participant user study, we show that our approach is significantly more secure than a standard image-based authentication and gaze-based 4-digit PIN entry.

   doi:10.1145/2207676.2208712

   bulling12_chi.pdf

   @inproceedings{bulling12_chi, author = {Bulling, Andreas and Alt, Florian and Schmidt, Albrecht}, keywords = {Cued-recall graphical passwords, Eye Tracking, Gaze-based, Saliency masks, user authentication}, title = {Increasing the Security of Gaze-Based Cued-Recall Graphical Passwords Using Saliency Masks}, booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2012}, pages = {3011-3020}, doi = {10.1145/2207676.2208712} }
2. 

   Gaze interaction in the post-WIMP world

   Andreas Bulling, Raimund Dachselt, Andrew Duchowski, Robert Jacob, Sophie Stellmach, Veronica Sundstedt

   Ext. Abstracts of the 2012 International Conference on Human Factors in Computing Systems (CHI), pp. 1221-1224, 2012.

   Abstract Links BibTeX Project

   With continuous progression away from desktop to post-WIMP applications, including multi-touch, gestural, or tangible interaction, there is high potential for eye gaze as a more natural human-computer interface in numerous contexts. Examples include attention-aware adaptations or the combination of gaze and hand gestures for interaction with distant displays. This SIG meeting provides a discussion venue for researchers and practitioners interested in gaze interaction in the post-WIMP era. We wish to draw attention to this emerging field and eventually formulate fundamental research questions. We will discuss the potential of gaze interaction for diverse application areas, interaction tasks, and multimodal user interface combinations. Our aims are to promote this research field, foster a larger research community, and establish the basis for a workshop at CHI 2013.

   doi:10.1145/2212776.2212428

   bulling12_chi_sig.pdf

   @inproceedings{bulling12_chi_sig, title = {Gaze interaction in the post-WIMP world}, author = {Bulling, Andreas and Dachselt, Raimund and Duchowski, Andrew and Jacob, Robert and Stellmach, Sophie and Sundstedt, Veronica}, isbn = {978-1-4503-1016-1}, doi = {10.1145/2212776.2212428}, year = {2012}, booktitle = {Ext. Abstracts of the 2012 International Conference on Human Factors in Computing Systems (CHI)}, pages = {1221-1224} }
3. 

   Extending the Visual Field of a Head-Mounted Eye Tracker for Pervasive Eye-Based Interaction

   Jayson Turner, Andreas Bulling, Hans Gellersen

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 269-272, 2012.

   Abstract Links BibTeX Project

   Pervasive eye-based interaction refers to the vision of eye-based interaction becoming ubiquitously usable in everyday life, e. g. across multiple displays in the environment. While current head-mounted eye trackers work well for interaction with displays at similar distances, the scene camera often fails to cover both remote and close proximity displays, e. g. a public display on a wall and a handheld portable device. In this paper we describe an approach that allows for robust detection and gaze mapping across multiple such displays. Our approach uses an additional scene camera to extend the viewing and gaze mapping area of the eye tracker and automatically switches between both cameras depending on the display in view. Results from a pilot study show that our system achieves a similar gaze estimation accuracy to a single-camera system while at the same time increasing usability.

   doi:10.1145/2168556.2168613

   turner12_etra.pdf

   @inproceedings{turner12_etra, author = {Turner, Jayson and Bulling, Andreas and Gellersen, Hans}, title = {Extending the Visual Field of a Head-Mounted Eye Tracker for Pervasive Eye-Based Interaction}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2012}, pages = {269-272}, doi = {10.1145/2168556.2168613} }
4. 

   Eye Gesture Recognition on Portable Devices

   Vytautas Vaitukaitis, Andreas Bulling

   Proc. International Workshop on Pervasive Eye Tracking and Mobile Gaze-Based Interaction (PETMEI), pp. 711-714, 2012.

   Abstract Links BibTeX Project

   Eye tracking using special-purpose devices has a large number of applications, such as usability testing or marketing research. In contrast, portable devices, such as mobile phones and tablets, have received only little attention so far. This is mainly due to their - until recently - limited sensing capabilities and processing power. In this work-in-progress paper we present the first prototype eye gesture recognition system for portable devices that does not require any additional equipment. The system combines techniques from image processing, computer vision and pattern recognition to detect eye gestures in the video recorded using the built-in front-facing camera. In a five-participant pilot study we show that our prototype can recognise four different continuous eye gestures in near real-time with an average accuracy of 60% on an Android-based smartphone (17.6% false positives) and 67.3% on a laptop (5.9% false positives). This initial result is promising and underlines the potential of eye tracking and eye-based interaction on portable devices.

   doi:10.1145/2370216.2370370

   vaitukaitis12_petmei.pdf

   @inproceedings{vaitukaitis12_petmei, author = {Vaitukaitis, Vytautas and Bulling, Andreas}, keywords = {Eye Gesture, Eye Tracking, gaze estimation, Laptop, Mobile Phone}, title = {Eye Gesture Recognition on Portable Devices}, booktitle = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Gaze-Based Interaction (PETMEI)}, year = {2012}, pages = {711-714}, doi = {10.1145/2370216.2370370} }
5. 

   Robust, real-time pupil tracking in highly off-axis images

   Lech Świrski, Andreas Bulling, Neil Dodgson

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 173-176, 2012.

   Abstract Links BibTeX Project

   Robust, accurate, real-time pupil tracking is a key component for online gaze estimation. On head-mounted eye trackers, existing algorithms that rely on circular pupils or contiguous pupil regions fail to detect or accurately track the pupil. This is because the pupil ellipse is often highly eccentric and partially occluded by eyelashes. We present a novel, real-time dark-pupil tracking algorithm that is robust under such conditions. Our approach uses a Haar-like feature detector to roughly estimate the pupil location, performs a k-means segmentation on the surrounding region to refine the pupil centre, and fits an ellipse to the pupil using a novel image-aware Random Sample Concensus (RANSAC) ellipse fitting. We compare our approach against existing real-time pupil tracking implementations, using a set of manually labelled infra-red dark-pupil eye images. We show that our technique has a higher pupil detection rate and greater pupil tracking accuracy.

   doi:10.1145/2168556.2168585

   swirski12_etra.pdf

   @inproceedings{swirski12_etra, author = wirski, Lech and Bulling, Andreas and Dodgson, Neil}, title = {Robust, real-time pupil tracking in highly off-axis images}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2012}, pages = {173-176}, doi = {10.1145/2168556.2168585} }
6. 

   Analysing the Potential of Adapting Head-Mounted Eye Tracker Calibration to a New User

   Benedict Fehringer, Andreas Bulling, Antonio Krüger

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 245-248, 2012.

   Abstract Links BibTeX Project

   A key issue with state-of-the-art mobile eye trackers, particularly during long-term recordings in daily life, is the need for cumbersome and time consuming (re)calibration. To reduce this burden, in this paper we investigate the feasibility of adapting the calibration obtained for one user to another. Calibration adaptation is automatically performed using a light-weight linear translation. We compare three different methods to compute the translation: "multi-point", where all calibration-points are used, "1-point", and "0-point" that uses only an external parameter. We evaluate these methods in a 6-participant user study in a controlled laboratory setting by measuring the error in visual angle between the predicted gaze point and the true gaze point. Our results show that, averaged across all participants, the best adapted calibration is only 0.8° (mean) off the calibration obtained for that specific user. We also show the potential of the 1-point and 0-point methods compared to the time-consuming multi-point computation.

   doi:10.1145/2168556.2168607

   fehringer12_etra.pdf

   @inproceedings{fehringer12_etra, author = {Fehringer, Benedict and Bulling, Andreas and Kr{\"{u}}ger, Antonio}, title = {Analysing the Potential of Adapting Head-Mounted Eye Tracker Calibration to a New User}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2012}, pages = {245-248}, doi = {10.1145/2168556.2168607} }
7. 

   Detection of smooth pursuits using eye movement shape features

   Mélodie Vidal, Andreas Bulling, Hans Gellersen

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 177-180, 2012.

   Abstract Links BibTeX Project

   Smooth pursuit eye movements hold information about the health, activity and situation of people, but to date there has been no efficient method for their automated detection. In this work we present a method to tackle the problem, based on machine learning. At the core of our method is a novel set of shape features that capture the characteristic shape of smooth pursuit movements over time. The features individually represent incomplete information about smooth pursuits but are combined in a machine learning approach. In an evaluation with eye movements collected from 18 participants, we show that our method can detect smooth pursuit movements with an accuracy of up to 92%, depending on the size of the feature set used for their prediction. Our results have twofold significance. First, they demonstrate a method for smooth pursuit detection in mainstream eye tracking, and secondly they highlight the utility of machine learning for eye movement analysis.

   doi:10.1145/2168556.2168586

   vidal12_etra.pdf

   @inproceedings{vidal12_etra, author = {Vidal, M{\'{e}}lodie and Bulling, Andreas and Gellersen, Hans}, title = {Detection of smooth pursuits using eye movement shape features}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2012}, pages = {177-180}, doi = {10.1145/2168556.2168586} }
8. 

   Towards pervasive gaze tracking with low-level image features

   Yanxia Zhang, Andreas Bulling, Hans Gellersen

   Proc. International Symposium on Eye Tracking Research and Applications (ETRA), pp. 261-264, 2012.

   Abstract Links BibTeX Project

   We contribute a novel gaze estimation technique, which is adaptable for person-independent applications. In a study with 17 participants, using a standard webcam, we recorded the subjects’ left eye images for different gaze locations. From these images, we extracted five types of basic visual features. We then sub-selected a set of features with minimum Redundancy Maximum Relevance (mRMR) for the input of a 2-layer regression neural network for estimating the subjects’ gaze. We investigated the effect of different visual features on the accuracy of gaze estimation. Using machine learning techniques, by combing different features, we achieved average gaze estimation error of 3.44° horizontally and 1.37° vertically for person-dependent.

   doi:10.1145/2168556.2168611

   zhang12_etra.pdf

   @inproceedings{zhang12_etra, author = {Zhang, Yanxia and Bulling, Andreas and Gellersen, Hans}, title = {Towards pervasive gaze tracking with low-level image features}, booktitle = {Proc. International Symposium on Eye Tracking Research and Applications (ETRA)}, year = {2012}, pages = {261-264}, doi = {10.1145/2168556.2168611} }

Book Chapters

1. Proc. 2nd International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)

   Andreas Bulling, Geert Brône, Shiwei Cheng, Päivi Majaranta

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 673-676, 2012.

   Abstract Links BibTeX Project

   Early work on applied eye tracking investigated gaze as an input modality to interact with a desktop computer and discussed some of the human factors and technical aspects involved in performing common computer tasks with the eyes such as pointing and menu selection. Since then, eye tracking technology has considerably matured. Research on eye-based interaction is starting to gain interest in various specialized areas that are no longer restricted to desktop environment, such as virtual reality, human-human and humanrobot interaction. There is also a growing interest to take eye tracking out into the wild, to mobile and pervasive settings.

   doi:10.1145/2370216.2370362

   @inbook{bulling12_petmei, author = {Bulling, Andreas and Br{\^{o}}ne, Geert and Cheng, Shiwei and Majaranta, P{\"{a}}ivi}, title = {Proc. 2nd International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2012}, pages = {673-676}, doi = {10.1145/2370216.2370362} }

2011

Journal Articles

1. 

   Eye Movement Analysis for Activity Recognition Using Electrooculography

   Andreas Bulling, Jamie A. Ward, Hans Gellersen, Gerhard Tröster

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 33(4), pp. 741-753, 2011.

   Abstract Links BibTeX Project

   In this work we investigate eye movement analysis as a new sensing modality for activity recognition. Eye movement data was recorded using an electrooculography (EOG) system. We first describe and evaluate algorithms for detecting three eye movement characteristics from EOG signals - saccades, fixations, and blinks - and propose a method for assessing repetitive patterns of eye movements. We then devise 90 different features based on these characteristics and select a subset of them using minimum redundancy maximum relevance feature selection (mRMR). We validate the method using an eight participant study in an office environment using an example set of five activity classes: copying a text, reading a printed paper, taking hand-written notes, watching a video, and browsing the web. We also include periods with no specific activity (the NULL class). Using a support vector machine (SVM) classifier and a person-independent (leave-one-out) training scheme, we obtain an average precision of 76.1% and recall of 70.5% over all classes and participants. The work demonstrates the promise of eye-based activity recognition (EAR) and opens up discussion on the wider applicability of EAR to other activities that are difficult, or even impossible, to detect using common sensing modalities.

   doi:10.1109/TPAMI.2010.86

   bulling11_pami.pdf

   @article{bulling11_pami, author = {Bulling, Andreas and Ward, Jamie A. and Gellersen, Hans and Tr{\"{o}}ster, Gerhard}, keywords = {Feature evaluation and selection, signal processing, Ubiquitous computing}, title = {Eye {M}ovement {A}nalysis for {A}ctivity {R}ecognition {U}sing {E}lectrooculography}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)}, volume = {33}, number = {4}, year = {2011}, pages = {741-753}, doi = {10.1109/TPAMI.2010.86} }
2. 

   What’s in the Eyes for Context-Awareness?

   Andreas Bulling, Daniel Roggen, Gerhard Tröster

   IEEE Pervasive Computing, 10(2), pp. 48-57, 2011.

   Abstract Links BibTeX Project

   Eye movements are a rich source of information about a person’s context. Analyzing the link between eye movements and cognition might even allow us to develop cognition-aware pervasive computing systems that assess a person’s cognitive context.

   doi:10.1109/MPRV.2010.49

   bulling11_pcm.pdf

   @article{bulling11_pcm, author = {Bulling, Andreas and Roggen, Daniel and Tr{\"{o}}ster, Gerhard}, keywords = {Machine learning, Pervasive computing, signal processing, Wearable Computing}, title = {What's in the Eyes for Context-Awareness?}, journal = {IEEE Pervasive Computing}, volume = {10}, number = {2}, year = {2011}, pages = {48-57}, doi = {10.1109/MPRV.2010.49} }

Conference Papers

1. 

   Recognition of Visual Memory Recall Processes Using Eye Movement Analysis

   Andreas Bulling, Daniel Roggen

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 455-464, 2011.

   Abstract Links BibTeX Project

   Physical activity, location, as well as a person’s psychophysiological and affective state are common dimensions for developing context-aware systems in ubiquitous computing. An important yet missing contextual dimension is the cognitive context that comprises all aspects related to mental information processing, such as perception, memory, knowledge, or learning. In this work we investigate the feasibility of recognising visual memory recall. We use a recognition methodology that combines minimum redundancy maximum relevance feature selection (mRMR) with a support vector machine (SVM) classifier. We validate the methodology in a dual user study with a total of fourteen participants looking at familiar and unfamiliar pictures from four picture categories: abstract, landscapes, faces, and buildings. Using person-independent training, we are able to discriminate between familiar and unfamiliar abstract pictures with a top recognition rate of 84.3% (89.3% recall, 21.0% false positive rate) over all participants. We show that eye movement analysis is a promising approach to infer the cognitive context of a person and discuss the key challenges for the real-world implementation of eye-based cognition-aware systems.

   doi:10.1145/2030112.2030172

   bulling11_ubicomp.pdf

   @inproceedings{bulling11_ubicomp, author = {Bulling, Andreas and Roggen, Daniel}, keywords = {Cognition- Awareness, Cognitive Context, Electrooculography (EOG), Eye Movement Analysis, Visual Memory Recall}, title = {Recognition of Visual Memory Recall Processes Using Eye Movement Analysis}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2011}, pages = {455-464}, doi = {10.1145/2030112.2030172} }
2. 

   The Web of Things as an Infrastructure for Improving Users’ Health and Wellbeing

   Eduardo Velloso, Débora Cardador, Katia Vega, Wallace Ugulino, Andreas Bulling, Hans Gellersen, Hugo Fuks

   Proc. Workshop of the Brazilian Institute for Web Science Research, 2011.

   Abstract Links BibTeX Project

   This position paper outlines the authors’ vision on how the Web of Things, using interconnected devices, including sensor nodes, mobile phones and conventional computers can help improve the overall health and wellbeing of its users. We describe ongoing work being carried by our research group both at PUC-Rio and at Lancaster University as well as the motivating background.

   velloso11_wsr.pdf

   @inproceedings{velloso11_wsr, author = {Velloso, Eduardo and Cardador, D{\'{e}}bora and Vega, Katia and Ugulino, Wallace and Bulling, Andreas and Gellersen, Hans and Fuks, Hugo}, title = {The Web of Things as an Infrastructure for Improving Users' Health and Wellbeing}, booktitle = {Proc. Workshop of the Brazilian Institute for Web Science Research}, year = {2011} }
3. 

   Analysing EOG Signal Features for the Discrimination of Eye Movements with Wearable Devices

   Mélodie Vidal, Andreas Bulling, Hans Gellersen

   Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI), pp. 15-20, 2011.

   Abstract Links BibTeX Project

   Eye tracking research in human-computer interaction and experimental psychology traditionally focuses on stationary devices and a small number of common eye movements. The advent of pervasive eye tracking promises new applications, such as eye-based mental health monitoring or eye-based activity and context recognition. These applications might require further research on additional eye movement types such as smooth pursuits and the vestibulo-ocular reflex as these movements have not been studied as extensively as saccades, fixations and blinks. In this paper we report our first step towards an effective discrimination of these movements. In a user study we collect naturalistic eye movements from 19 people using the two most common measurement techniques (EOG and IR-based). We develop a set of basic signal features that we extract from the collected eye movement data and show that a feature-based approach has the potential to discriminate between saccades, smooth pursuits, and vestibulo-ocular reflex movements.

   doi:10.1145/2029956.2029962

   vidal11_petmei.pdf

   @inproceedings{vidal11_petmei, author = {Vidal, M{\'{e}}lodie and Bulling, Andreas and Gellersen, Hans}, title = {Analysing EOG Signal Features for the Discrimination of Eye Movements with Wearable Devices}, booktitle = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, year = {2011}, pages = {15-20}, doi = {10.1145/2029956.2029962} }
4. 

   Towards Qualitative Assessment of Weight Lifting Exercises Using Body-Worn Sensors

   Eduardo Velloso, Andreas Bulling, Hans Gellersen

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 587-588, 2011.

   Abstract Links BibTeX Project

   Sports exercises are beneficial for general health and fitness. Some exercises such as weight lifting are particularly error-prone and using incorrect techniques can result in serious injuries. The current work aims to develop a weight lifting assistant that relies on motion sensors mounted on the body and integrated into gym equipment that provides qualitative feedback on the user’s performance. We believe that by comparing motion data recorded from different parts of the body with a mathematical model of the correct technique, we will be able to qualitatively assess the user’s performance, and provide a score and suggestions for improvement.

   doi:10.1145/2030112.2030226

   velloso11_ubicomp.pdf

   @inproceedings{velloso11_ubicomp, author = {Velloso, Eduardo and Bulling, Andreas and Gellersen, Hans}, title = {Towards Qualitative Assessment of Weight Lifting Exercises Using Body-Worn Sensors}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2011}, pages = {587-588}, doi = {10.1145/2030112.2030226} }
5. 

   Discrimination of Gaze Directions Using Low-Level Eye Image Features

   Yanxia Zhang, Andreas Bulling, Hans Gellersen

   Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI), pp. 9-13, 2011.

   Abstract Links BibTeX Project

   In mobile daily life settings, video-based gaze tracking faces challenges associated with changes in lighting conditions and artefacts in the video images caused by head and body movements. These challenges call for the development of new methods that are robust to such influences. In this paper we investigate the problem of gaze estimation, more specifically how to discriminate different gaze directions from eye images. In a 17 participant user study we record eye images for 13 different gaze directions from a standard web- cam. We extract a total of 50 features from these images that encode information on color, intensity and orientations. Using mRMR feature selection and a k-nearest neighbor (kNN) classifier we show that we can estimate these gaze directions with a mean recognition performance of 86%.

   doi:10.1145/2029956.2029961

   zhang11_petmei.pdf

   @inproceedings{zhang11_petmei, author = {Zhang, Yanxia and Bulling, Andreas and Gellersen, Hans}, title = {Discrimination of Gaze Directions Using Low-Level Eye Image Features}, booktitle = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, year = {2011}, pages = {9-13}, doi = {10.1145/2029956.2029961} }

6. Kontexterkennung für Hörgeräte mittels zusätzlicher Sensormodalitäten

   Bernd Tessendorf, Andreas Bulling, Daniel Roggen, Thomas Stiefmeier, Manuela Feilner, Peter Derleth, Gerhard Tröster

   Proc. Annual Convention for Acoustics (DAGA), 2011.

   BibTeX Project

   @inproceedings{tessendorf11_daga, author = {Tessendorf, Bernd and Bulling, Andreas and Roggen, Daniel and Stiefmeier, Thomas and Feilner, Manuela and Derleth, Peter and Tr{\"{o}}ster, Gerhard}, title = {Kontexterkennung f{\"{u}}r H{\"{o}}rger{\"{a}}te mittels zus{\"{a}}tzlicher Sensormodalit{\"{a}}ten}, booktitle = {Proc. Annual Convention for Acoustics (DAGA)}, year = {2011} }
7. 

   Recognition of Hearing Needs From Body and Eye Movements to Improve Hearing Instruments

   Bernd Tessendorf, Andreas Bulling, Daniel Roggen, Thomas Stiefmeier, Manuela Feilner, Peter Derleth, Gerhard Tröster

   Proc. International Conference on Pervasive Computing (Pervasive), pp. 314-331, 2011.

   Abstract Links BibTeX Project

   Hearing instruments (HIs) have emerged as true pervasive computers as they continuously adapt the hearing program to the user’s context. However, current HIs are not able to distinguish different hearing needs in the same acoustic environment. In this work, we explore how information derived from body and eye movements can be used to improve the recognition of such hearing needs. We conduct an experiment to provoke an acoustic environment in which different hearing needs arise: active conversation and working while colleagues are having a conversation in a noisy office environment. We record body movements on nine body locations, eye movements using electrooculography (EOG), and sound using commercial HIs for eleven participants. Using a support vector machine (SVM) classifier and person-independent training we improve the accuracy of 77% based on sound to an accuracy of 92% using body movements. With a view to a future implementation into a HI we then perform a detailed analysis of the sensors attached to the head. We achieve the best accuracy of 86% using eye movements compared to 84% for head movements. Our work demonstrates the potential of additional sensor modalities for future HIs and motivates to investigate the wider applicability of this approach on further hearing situations and needs.

   doi:10.1007/978-3-642-21726-5_20

   tessendorf11_pervasive.pdf

   @inproceedings{tessendorf11_pervasive, author = {Tessendorf, Bernd and Bulling, Andreas and Roggen, Daniel and Stiefmeier, Thomas and Feilner, Manuela and Derleth, Peter and Tr{\"{o}}ster, Gerhard}, title = {Recognition of Hearing Needs From Body and Eye Movements to Improve Hearing Instruments}, booktitle = {Proc. International Conference on Pervasive Computing (Pervasive)}, year = {2011}, pages = {314-331}, doi = {10.1007/978-3-642-21726-5_20} }
8. 

   Combining Gaze with Manual Interaction to Extend Physical Reach

   Jayson Turner, Andreas Bulling, Hans Gellersen

   Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI), pp. 33-36, 2011.

   Abstract Links BibTeX Project

   Situated public displays and interactive surfaces are becoming ubiquitous in our daily lives. Issues arise with these devices when attempting to interact over a distance or with content that is physically out of reach. In this paper we outline three techniques that combine gaze with manual hand-controlled input to move objects. We demonstrate and discuss how these techniques could be applied to two scenarios involving, (1) a multi-touch surface and (2) a public display and a mobile device.

   doi:10.1145/2029956.2029966

   turner11_petmei.pdf

   @inproceedings{turner11_petmei, author = {Turner, Jayson and Bulling, Andreas and Gellersen, Hans}, title = {Combining Gaze with Manual Interaction to Extend Physical Reach}, booktitle = {Proc. International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, year = {2011}, pages = {33-36}, doi = {10.1145/2029956.2029966} }

Book Chapters

1. Proc. 1st International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)

   Andreas Bulling, Andrew T. Duchowski, Päivi Majaranta

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 627-628, 2011.

   Abstract Links BibTeX Project

   Recent developments in mobile eye tracking equipment and automated eye movement analysis point the way toward unobtrusive eye-based human-computer interfaces that are pervasively usable in everyday life. We call this new paradigm pervasive eye tracking - continuous eye monitoring and analysis 24/7. PETMEI 2011 provides a forum for researcher from human-computer interaction, context-aware computing, and eye tracking to discuss techniques and applications that go beyond classical eye tracking and stationary eye-based interaction. We aim to discuss the implications of pervasive eye tracking for context-aware computing and to identify the key research challenges of mobile eye-based interaction. The long-term goal is to create a strong interdisciplinary research community linking these research fields together and to establish the workshop as the premier forum for research on pervasive eye tracking and mobile eye-based interaction.

   doi:10.1145/2030112.2030248

   @inbook{bulling11_petmei, author = {Bulling, Andreas and Duchowski, Andrew T. and Majaranta, P{\"{a}}ivi}, keywords = {activity and context recognition, Cognition-Awareness, Eye Movement Analysis, Eye Tracking, eye-based interaction}, title = {Proc. 1st International Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (PETMEI)}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2011}, pages = {627-628}, doi = {10.1145/2030112.2030248} }

2010

Journal Articles

1. 

   Toward Mobile Eye-Based Human-Computer Interaction

   Andreas Bulling, Hans Gellersen

   IEEE Pervasive Computing, 9(4), pp. 8-12, 2010.

   Abstract Links BibTeX Project

   Research in eye-based human-computer interaction (HCI) has matured over the past 20 years with current HCI research mostly focusing on stationary eye trackers in laboratory settings. This survey of latest advances in eye-tracking equipment and automated eye movement analysis suggests a new generation of mobile eye-based interfaces that will become pervasive and seamlessly integrated into people’s everyday lives.

   doi:10.1109/MPRV.2010.86

   bulling10_pcm.pdf

   @article{bulling10_pcm, author = {Bulling, Andreas and Gellersen, Hans}, title = {Toward {M}obile {E}ye-{B}ased {H}uman-{C}omputer {I}nteraction}, journal = {IEEE Pervasive Computing}, volume = {9}, number = {4}, year = {2010}, pages = {8-12}, doi = {10.1109/MPRV.2010.86} }

Conference Papers

1. 

   Towards Multi-Modal Context Recognition for Hearing Instruments by Analysing Eye and Head Movements

   Bernd Tessendorf, Andreas Bulling, Daniel Roggen, Thomas Stiefmeier, Gerhard Tröster, Manuela Feilner, Peter Derleth

   Proc. IEEE International Symposium on Wearable Computers (ISWC), pp. 1-2, 2010.

   Abstract Links BibTeX Project

   Current hearing instruments (HI) only rely on auditory scene analysis to adapt to the situation of the user. It is for this reason that these systems are limited in the number and type of situations they can detect. We investigate how context information derived from eye and head movements can be used to resolve such situations. We focus on two example problems that are challenging for current HIs: To distinguish concentrated from interaction, and to detect whether a person is walking alone or walking while having a conversation. We collect an eleven participant (6 male, 5 female, age 24-59) dataset that covers different typical office activities. Using person-independent training and isolated recognition we achieve an average precision of 71.7% (recall: 70.1%) for recognising concentrated work and 57.2% precision (recall: 81.3%) for detecting walking while conversing.

   doi:10.1109/ISWC.2010.5665855

   tessendorf10_iswc.pdf

   @inproceedings{tessendorf10_iswc, author = {Tessendorf, Bernd and Bulling, Andreas and Roggen, Daniel and Stiefmeier, Thomas and Tr{\"{o}}ster, Gerhard and Feilner, Manuela and Derleth, Peter}, title = {Towards Multi-Modal Context Recognition for Hearing Instruments by Analysing Eye and Head Movements}, booktitle = {Proc. IEEE International Symposium on Wearable Computers (ISWC)}, year = {2010}, pages = {1-2}, doi = {10.1109/ISWC.2010.5665855} }
2. 

   On the issue of variability in labels and sensor configurations in activity recognition systems

   Daniel Roggen, Kilian Förster, Alberto Calatroni, Andreas Bulling, Gerhard Tröster

   Proc. "How to do good activity recognition research? Experimental methodologies, evaluation metrics, and reproducibility issues" (Pervasive), 2010.

   Abstract Links BibTeX Project

   Two aspects of the design and characterization of activity recognition systems are rarely elaborated in the literature. First, the influence of system performance with variability in sensor placement and orientation is often overlooked. This is important for the deployment of robust activity recognition systems. Second, the influence of labeling variability is also overlooked, especially w.r.t. label boundary jitter and labeling errors. This is important during the development of an activity recognition system as acquiring labels is costly. We argue that there is a need to explicitly address the consequences of such variability in publications, together with the mitigation strategies that are used. Elaborating on this is required to move the state of the art towards real-world applications, such as in industrial wearable assistance applications or pervasive healthcare.

   roggen10_pervasive.pdf

   @inproceedings{roggen10_pervasive, author = {Roggen, Daniel and F{\"{o}}rster, Kilian and Calatroni, Alberto and Bulling, Andreas and Tr{\"{o}}ster, Gerhard}, title = {On the issue of variability in labels and sensor configurations in activity recognition systems}, booktitle = {Proc. "How to do good activity recognition research? Experimental methodologies, evaluation metrics, and reproducibility issues" (Pervasive)}, year = {2010} }

2009

Journal Articles

1. 

   Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments

   Andreas Bulling, Daniel Roggen, Gerhard Tröster

   Journal of Ambient Intelligence and Smart Environments, 1(2), pp. 157-171, 2009.

   Abstract Links BibTeX Project

   In this article we introduce the analysis of eye motion as a new input modality for activity recognition, context-awareness and mobile HCI applications. We describe a novel embedded eye tracker that, in contrast to common systems using video cameras, relies on Electrooculography (EOG). This self-contained wearable device consists of goggles with dry electrodes integrated into the frame and a small pocket-worn component with a DSP for real-time EOG signal processing. It can store data locally for long-term recordings or stream processed EOG signals to a remote device over Bluetooth. We show how challenges associated with wearability, eye motion analysis and signal artefacts caused by physical activity can be addressed with a combination of a special mechanical design, optimised algorithms for eye movement detection and adaptive signal processing. In two case studies, we demonstrate that EOG is a suitable measurement technique for the recognition of reading activity and eye-based human-computer interaction. Eventually, wearable EOG goggles may pave the way for seamless eye movement analysis and new forms of context-awareness not possible today.

   doi:10.3233/AIS-2009-0020

   bulling09_jaise.pdf

   @article{bulling09_jaise, author = {Bulling, Andreas and Roggen, Daniel and Tr{\"{o}}ster, Gerhard}, keywords = {Activity Recognition, Context-awareness, Electrooculography (EOG), Human-Computer Interaction (HCI), Wearable Eye Tracking}, title = {Wearable {EOG} goggles: {S}eamless sensing and context-awareness in everyday environments}, journal = {Journal of Ambient Intelligence and Smart Environments}, volume = {1}, number = {2}, year = {2009}, pages = {157-171}, doi = {10.3233/AIS-2009-0020} }

Conference Papers

1. 

   Wearable EOG Goggles: Eye-Based Interaction in Everyday Environments

   Andreas Bulling, Daniel Roggen, Gerhard Tröster

   Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 3259-3264, 2009.

   Abstract Links BibTeX Project

   In this paper, we present an embedded eye tracker for context-awareness and eye-based human-computer interaction – the wearable EOG goggles. In contrast to common systems using video, this unobtrusive device relies on Electrooculography (EOG). It consists of goggles with dry electrodes integrated into the frame and a small pocket-worn component with a powerful microcontroller for EOG signal processing. Using this lightweight system, sequences of eye movements, so-called eye gestures, can be efficiently recognised from EOG signals in real-time for HCI purposes. The device is self-contained solution and allows for seamless eye motion sensing, context-recognition and eye-based interaction in everyday environments.

   doi:10.1145/1520340.1520468

   bulling09_chi.pdf

   @inproceedings{bulling09_chi, author = {Bulling, Andreas and Roggen, Daniel and Tr{\"{o}}ster, Gerhard}, keywords = {Context-awareness, Electrooculography (EOG), Eye Gestures, Eye Tracking, Human-Computer Interaction (HCI), Wearable Computing}, title = {Wearable EOG Goggles: Eye-Based Interaction in Everyday Environments}, booktitle = {Ext. Abstr. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)}, year = {2009}, pages = {3259-3264}, doi = {10.1145/1520340.1520468} }
2. 

   Eye Movement Analysis for Activity Recognition

   Andreas Bulling, Jamie A. Ward, Hans Gellersen, Gerhard Tröster

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 41-50, 2009.

   Abstract Links BibTeX Project

   In this work we investigate eye movement analysis as a new modality for recognising human activity. We devise 90 different features based on the main eye movement characteristics: saccades, fixations and blinks. The features are derived from eye movement data recorded using a wearable electrooculographic (EOG) system. We describe a recognition methodology that combines minimum redundancy maximum relevance feature selection (mRMR) with a support vector machine (SVM) classifier. We validate the method in an eight participant study in an office environment using five activity classes: copying a text, reading a printed paper, taking hand-written notes, watching a video and browsing the web. In addition, we include periods with no specific activity. Using a person-independent (leave-one-out) training scheme, we obtain an average precision of 76.1% and recall of 70.5% over all classes and participants. We discuss the most relevant features and show that eye movement analysis is a rich and thus promising modality for activity recognition.

   doi:10.1145/1620545.1620552

   bulling09_ubicomp.pdf

   @inproceedings{bulling09_ubicomp, author = {Bulling, Andreas and Ward, Jamie A. and Gellersen, Hans and Tr{\"{o}}ster, Gerhard}, keywords = {Activity Recognition, Electrooculography (EOG), Eye Movement Analysis, Recognition of Office Activities}, title = {Eye Movement Analysis for Activity Recognition}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2009}, pages = {41-50}, doi = {10.1145/1620545.1620552} }
3. 

   Speech as a Feedback Modality for Smart Objects

   Clemens Lombriser, Andreas Bulling, Andreas Breitenmoser, Gerhard Tröster

   Proc. International Workshop on Intelligent Pervasive Devices (PerDev), pp. 1-5, 2009.

   Abstract Links BibTeX Project

   One part of the vision of ubiquitous computing is the integration of sensing and actuation nodes into everyday objects, clothes worn on the body, and in large numbers in the environment. These augmented environments require novel types of interfaces that provide for naturalistic and adaptive interaction depending on the user context. In this paper, we evaluate the use of speech synthesis on small, low-power sensor nodes that may be integrated into smart objects. We evaluate the so-called Wireless Voice Node, a wireless sensor node with the ability to produce speech as a novel feedback modality for ambient intelligence applications. As an example, we present a doll that aims at using speech synthesis to improve the playing experience of children.

   doi:10.1109/PERCOM.2009.4912831

   lombriser09_perdev.pdf

   @inproceedings{lombriser09_perdev, author = {Lombriser, Clemens and Bulling, Andreas and Breitenmoser, Andreas and Tr{\"{o}}ster, Gerhard}, title = {Speech as a Feedback Modality for Smart Objects}, booktitle = {Proc. International Workshop on Intelligent Pervasive Devices (PerDev)}, year = {2009}, pages = {1-5}, doi = {10.1109/PERCOM.2009.4912831} }

4. OPPORTUNITY: Activity and context awareness in opportunistic open-ended sensor environments

   Daniel Roggen, Kilian Förster, Alberto Calatroni, Andreas Bulling, Thomas Holleczek, Gerhard Tröster, Paul Lukowicz, Gerald Pirkl, David Bannach, Alois Ferscha, Andreas Riener, Clemens Holzmann, Ricardo Chavarriaga, José R. Millán

   Proc. European Future and Emerging Technologies Conference (FET), 2009.

   BibTeX Project

   @inproceedings{roggen09_fet, author = {Roggen, Daniel and F{\"{o}}rster, Kilian and Calatroni, Alberto and Bulling, Andreas and Holleczek, Thomas and Tr{\"{o}}ster, Gerhard and Lukowicz, Paul and Pirkl, Gerald and Bannach, David and Ferscha, Alois and Riener, Andreas and Holzmann, Clemens and Chavarriaga, Ricardo and del R. Mill{\'{a}}n, Jos{\'{e}}}, keywords = {OPPORTUNITY}, title = {OPPORTUNITY: Activity and context awareness in opportunistic open-ended sensor environments}, booktitle = {Proc. European Future and Emerging Technologies Conference (FET)}, year = {2009}, publisher = {European Commission}, location = {Prague, Czech Republic} }

2008

Conference Papers

1. 

   EyeMote - Towards Context-Aware Gaming Using Eye Movements Recorded From Wearable Electrooculography

   Andreas Bulling, Daniel Roggen, Gerhard Tröster

   Proc. International Conference on Fun and Games (FnG), pp. 33-45, 2008.

   Abstract Links BibTeX Project

   Physical activity has emerged as a novel input modality for so-called active video games. Input devices such as music instruments, dance mats or the Wii accessories allow for novel ways of interaction and a more immersive gaming experience. In this work we describe how eye movements recognised from electrooculographic (EOG) signals can be used for gaming purposes in three different scenarios. In contrast to common video-based systems, EOG can be implemented as a wearable and light-weight system which allows for long-term use with unconstrained simultaneous physical activity. In a stationary computer game we show that eye gestures of varying complexity can be recognised online with equal performance to a state-of-the-art video-based system. For pervasive gaming scenarios, we show how eye movements can be recognised in the presence of signal artefacts caused by physical activity such as walking. Finally, we describe possible future context-aware games which exploit unconscious eye movements and show which possibilities this new input modality may open up.

   doi:10.1007/978-3-540-88322-7_4

   bulling08_fng.pdf

   @inproceedings{bulling08_fng, author = {Bulling, Andreas and Roggen, Daniel and Tr{\"{o}}ster, Gerhard}, keywords = {Active Video Games, Context-awareness, Electrooculography (EOG), Eye Tracking, Human-Computer Interaction (HCI), Location-Based Gaming, Pervasive Gaming}, title = {EyeMote - Towards Context-Aware Gaming Using Eye Movements Recorded From Wearable Electrooculography}, booktitle = {Proc. International Conference on Fun and Games (FnG)}, year = {2008}, pages = {33-45}, doi = {10.1007/978-3-540-88322-7_4} }
2. 

   It’s in Your Eyes - Towards Context-Awareness and Mobile HCI Using Wearable EOG Goggles

   Andreas Bulling, Daniel Roggen, Gerhard Tröster

   Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), pp. 84-93, 2008.

   Abstract Links BibTeX Project

   In this work we describe the design, implementation and evaluation of a novel eye tracker for context-awareness and mobile HCI applications. In contrast to common systems using video cameras, this compact device relies on Electrooculography (EOG). It consists of goggles with dry electrodes integrated into the frame and a small pocket-worn component with a DSP for real-time EOG signal processing. The device is intended for wearable and standalone use: It can store data locally for long-term recordings or stream processed EOG signals to a remote device over Bluetooth. We describe how eye gestures can be efficiently recognised from EOG signals for HCI purposes. In an experiment conducted with 11 subjects playing a computer game we show that 8 eye gestures of varying complexity can be continuously recognised with equal performance to a state-of-the-art video-based system. Physical activity leads to artefacts in the EOG signal. We describe how these artefacts can be removed using an adaptive filtering scheme and characterise this approach on a 5-subject dataset. In addition to explicit eye movements for HCI, we discuss how the analysis of unconscious eye movements may eventually allow to deduce information on user activity and context not available with current sensing modalities.

   doi:10.1145/1409635.1409647

   bulling08_ubicomp.pdf

   @inproceedings{bulling08_ubicomp, author = {Bulling, Andreas and Roggen, Daniel and Tr{\"{o}}ster, Gerhard}, keywords = {Context-awareness, Electrooculography (EOG), Eye Gestures, Eye Tracking, Human-Computer Interaction (HCI), Wearable Computing}, title = {It's in {Y}our {E}yes - {T}owards {C}ontext-{A}wareness and {M}obile {HCI} {U}sing {W}earable {EOG} {G}oggles}, booktitle = {Proc. ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp)}, year = {2008}, pages = {84-93}, doi = {10.1145/1409635.1409647} }
3. 

   Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography

   Andreas Bulling, Jamie A. Ward, Hans Gellersen, Gerhard Tröster

   Proc. International Conference on Pervasive Computing (Pervasive), pp. 19-37, 2008.

   Abstract Links BibTeX Project

   In this work we analyse the eye movements of people in transit in an everyday environment using a wearable electrooculographic (EOG) system. We compare three approaches for continuous recognition of reading activities: a string matching algorithm which exploits typical characteristics of reading signals, such as saccades and fixations; and two variants of Hidden Markov Models (HMMs) - mixed Gaussian and discrete. The recognition algorithms are evaluated in an experiment performed with eight subjects reading freely chosen text without pictures while sitting at a desk, standing, walking indoors and outdoors, and riding a tram. A total dataset of roughly 6 hours was collected with reading activity accounting for about half of the time. We were able to detect reading activities over all subjects with a top recognition rate of 80.2% (71.0% recall, 11.6% false positives) using string matching. We show that EOG is a potentially robust technique for reading recognition across a number of typical daily situations.

   doi:10.1007/978-3-540-79576-6_2

   bulling08_pervasive.pdf

   @inproceedings{bulling08_pervasive, author = {Bulling, Andreas and Ward, Jamie A. and Gellersen, Hans and Tr{\"{o}}ster, Gerhard}, keywords = {Activity Recognition, Electrooculography (EOG), Reading Activity, Recognition of Reading, Transit, wearable}, title = {Robust {R}ecognition of {R}eading {A}ctivity in {T}ransit {U}sing {W}earable {E}lectrooculography}, booktitle = {Proc. International Conference on Pervasive Computing (Pervasive)}, year = {2008}, pages = {19-37}, doi = {10.1007/978-3-540-79576-6_2} }