Publications
2019

MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation
Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41 (1), pp. 162-175, 2019.
AbstractLinksBibTeXProject
Learning-based methods are believed to work well for unconstrained gaze estimation, i.e. gaze estimation from a monocular RGB camera without assumptions regarding user, environment, or camera. However, current gaze datasets were collected under laboratory conditions and methods were not evaluated across multiple datasets. Our work makes three contributions towards addressing these limitations. First, we present the MPIIGaze dataset, which contains 213,659 full face images and corresponding ground-truth gaze positions collected from 15 users during everyday laptop use over several months. An experience sampling approach ensured continuous gaze and head poses and realistic variation in eye appearance and illumination. To facilitate cross-dataset evaluations, 37,667 images were manually annotated with eye corners, mouth corners, and pupil centres. Second, we present an extensive evaluation of state-of-the-art gaze estimation methods on three current datasets, including MPIIGaze. We study key challenges including target gaze range, illumination conditions, and facial appearance variation. We show that image resolution and the use of both eyes affect gaze estimation performance, while head pose and pupil centre information are less informative. Finally, we propose GazeNet, the first deep appearance-based gaze estimation method. GazeNet improves on the state of the art by 22% (from a mean error of 13.9 degrees to 10.8 degrees) for the most challenging cross-dataset evaluation.
@article{zhang19_pami,
title = {MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation},
author = {Xucong Zhang and Yusuke Sugano and Mario Fritz and Andreas Bulling},
year = {2019},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
volume = {41},
number = {1},
pages = {162-175},
doi = {10.1109/TPAMI.2017.2778103},
}

Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications
Xucong Zhang, Yusuke Sugano, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--13, 2019.
AbstractLinksBibTeXProject
Appearance-based gaze estimation methods that only require an off-the-shelf camera have significantly improved but they are still not yet widely used in the human-computer interaction (HCI) community. This is partly because it remains unclear how they perform compared to model-based approaches as well as dominant, special-purpose eye tracking equipment. To address this limitation, we evaluate the performance of state-of-the-art appearance-based gaze estimation for interaction scenarios with and without personal calibration, indoors and outdoors, for different sensing distances, as well as for users with and without glasses. We discuss the obtained findings and their implications for the most important gaze-based applications, namely explicit eye input, attentive user interfaces, gaze-based user modelling, and passive eye monitoring. To democratise the use of appearance-based gaze estimation and interaction in HCI, we finally present OpenGaze (www.opengaze.org), the first software toolkit for appearance-based gaze estimation and interaction.
@inproceedings{zhang19_chi,
title = {Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications},
author = {Xucong Zhang and Yusuke Sugano and Andreas Bulling},
year = {2019},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--13},
doi = {10.1145/3290605.3300646},
}
2018

Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour
Philipp Müller, Michael Xuelin Huang, Xucong Zhang, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--10, 2018.
AbstractLinksBibTeXProject
Eye contact is one of the most important non-verbal social cues and fundamental to human interactions. However, detecting eye contact without specialized eye tracking equipment poses significant challenges, particularly for multiple people in real-world settings. We present a novel method to robustly detect eye contact in natural three- and four-person interactions using off-the-shelf ambient cameras. Our method exploits that, during conversations, people tend to look at the person who is currently speaking. Harnessing the correlation between people's gaze and speaking behaviour therefore allows our method to automatically acquire training data during deployment and adaptively train eye contact detectors for each target user. We empirically evaluate the performance of our method on a recent dataset of natural group interactions and demonstrate that it achieves a relative improvement over the state-of-the-art method of more than 60%, and also improves over a head pose based baseline.
@inproceedings{mueller18_etra,
title = {Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour},
author = {Philipp M{\"{u}}ller and Michael Xuelin Huang and Xucong Zhang and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--10},
doi = {10.1145/3204493.3204549},
}

Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Seonwook Park, Xucong Zhang, Andreas Bulling, Otmar Hilliges
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--10, 2018.
AbstractLinksBibTeXProject Best Presentation Award
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation.
@inproceedings{park18_etra,
title = {Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings},
author = {Seonwook Park and Xucong Zhang and Andreas Bulling and Otmar Hilliges},
year = {2018},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--10},
doi = {10.1145/3204493.3204545},
video = {https://www.youtube.com/watch?v=I8WlEHgDBV4},
}

Training Person-Specific Gaze Estimators from Interactions with Multiple Devices
Xucong Zhang, Michael Xuelin Huang, Yusuke Sugano, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--12, 2018.
AbstractLinksBibTeXProject
Learning-based gaze estimation has significant potential to enable attentive user interfaces and gaze-based interaction on the billions of camera-equipped handheld devices and ambient displays. While training accurate person- and device-independent gaze estimators remains challenging, person-specific training is feasible but requires tedious data collection for each target device. To address these limitations, we present the first method to train person-specific gaze estimators across multiple devices. At the core of our method is a single convolutional neural network with shared feature extraction layers and device-specific branches that we train from face images and corresponding on-screen gaze locations. Detailed evaluations on a new dataset of interactions with five common devices (mobile phone, tablet, laptop, desktop computer, smart TV) and three common applications (mobile game, text editing, media center) demonstrate the significant potential of cross-device training. We further explore training with gaze locations derived from natural interactions, such as mouse or touch input.
@inproceedings{zhang18_chi,
title = {Training Person-Specific Gaze Estimators from Interactions with Multiple Devices},
author = {Xucong Zhang and Michael Xuelin Huang and Yusuke Sugano and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--12},
doi = {10.1145/3173574.3174198},
}

Revisiting Data Normalization for Appearance-Based Gaze Estimation
Xucong Zhang, Yusuke Sugano, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--9, 2018.
AbstractLinksBibTeXProject
Appearance-based gaze estimation is promising for unconstrained real-world settings, but the significant variability in head pose and user-camera distance poses significant challenges for training generic gaze estimators. Data normalization was proposed to cancel out this geometric variability by mapping input images and gaze labels to a normalized space. Although used successfully in prior works, the role and importance of data normalization remains unclear. To fill this gap, we study data normalization for the first time using principled evaluations on both simulated and real data. We propose a modification to the current data normalization formulation by removing the scaling factor and show that our new formulation performs significantly better (between 9.5% and 32.7%) in the different evaluation settings. Using images synthesized from a 3D face model, we demonstrate the benefit of data normalization for the efficiency of the model training. Experiments on real-world images confirm the advantages of data normalization in terms of gaze estimation performance.
@inproceedings{zhang18_etra,
title = {Revisiting Data Normalization for Appearance-Based Gaze Estimation},
author = {Xucong Zhang and Yusuke Sugano and Andreas Bulling},
year = {2018},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--9},
doi = {10.1145/3204493.3204548},
}
2017

It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation
Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling
Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2299-2308, 2017.
AbstractLinksBibTeXProject
Eye gaze is an important non-verbal cue for human affect analysis. Recent gaze estimation work indicated that information from the full face region can benefit performance. Pushing this idea further, we propose an appearance-based method that, in contrast to a long-standing line of work in computer vision, only takes the full face image as input. Our method encodes the face image using a convolutional neural network with spatial weights applied on the feature maps to flexibly suppress or enhance information in different facial regions. Through extensive evaluation, we show that our full-face method significantly outperforms the state of the art for both 2D and 3D gaze estimation, achieving improvements of up to 14.3% on MPIIGaze and 27.7% on EYEDIAP for person-independent 3D gaze estimation. We further show that this improvement is consistent across different illumination conditions and gaze directions and par- ticularly pronounced for the most challenging extreme head poses.
@inproceedings{zhang17_cvprw,
title = {It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation},
author = {Xucong Zhang and Yusuke Sugano and Mario Fritz and Andreas Bulling},
year = {2017},
booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
pages = {2299-2308},
doi = {10.1109/CVPRW.2017.284},
}

Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery
Xucong Zhang, Yusuke Sugano, Andreas Bulling
Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 193-203, 2017.
AbstractLinksBibTeXProject Best Paper Honourable Mention Award
Eye contact is an important non-verbal cue in social signal processing and promising as a measure of overt attention in human-object interactions and attentive user interfaces. However, robust detection of eye contact across different users, gaze targets, camera positions, and illumination conditions is notoriously challenging. We present a novel method for eye contact detection that combines a state-of-the-art appearance-based gaze estimator with a novel approach for unsupervised gaze target discovery, i.e. without the need for tedious and time-consuming manual data annotation. We evaluate our method in two real-world scenarios: detecting eye contact at the workplace, including on the main work display, from cameras mounted to target objects, as well as during everyday social interactions with the wearer of a head-mounted egocentric camera. We empirically evaluate the performance of our method in both scenarios and demonstrate its effectiveness for detecting eye contact independent of target object type and size, camera position, and user and recording environment.
@inproceedings{zhang17_uist,
title = {Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery},
author = {Xucong Zhang and Yusuke Sugano and Andreas Bulling},
year = {2017},
booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)},
pages = {193-203},
doi = {10.1145/3126594.3126614},
video = {https://www.youtube.com/watch?v=ccrS5XuhQpk},
}
2016

Concept for Using Eye Tracking in a Head-mounted Display to Adapt Rendering to the User's Current Visual Field
Daniel Pohl, Xucong Zhang, Andreas Bulling, Oliver Grau
Proc. of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST), pp. 323-324, 2016.
AbstractLinksBibTeXProject
With increasing spatial and temporal resolution in head-mounted displays (HMDs), using eye trackers to adapt rendering to the user is getting important to handle the rendering workload. Besides using methods like foveated rendering, we propose to use the current visual field for rendering, depending on the eye gaze. We use two effects for performance optimizations. First, we noticed a lens defect in HMDs, where depending on the distance of the eye gaze to the center, certain parts of the screen towards the edges are not visible anymore. Second, if the user looks up, he cannot see the lower parts of the screen anymore. For the invisible areas, we propose to skip rendering and to reuse the pixels colors from the previous frame. We provide a calibration routine to measure these two effects. We apply the current visual field to a renderer and get up to 2x speed-ups.
@inproceedings{pohl16_vrst,
title = {Concept for Using Eye Tracking in a Head-mounted Display to Adapt Rendering to the User's Current Visual Field},
author = {Daniel Pohl and Xucong Zhang and Andreas Bulling and Oliver Grau},
year = {2016},
booktitle = {Proc. of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST)},
pages = {323-324},
doi = {10.1145/2993369.2996300},
}

Combining Eye Tracking with Optimizations for Lens Astigmatism in Modern Wide-Angle HMDs
Daniel Pohl, Xucong Zhang, Andreas Bulling
Proc. IEEE Virtual Reality (VR), pp. 269-270, 2016.
AbstractLinksBibTeXProject
Virtual Reality has hit the consumer market with affordable head-mounted displays. When using these, it quickly becomes apparent that the resolution of the built-in display panels still needs to be highly increased. To overcome the resulting higher performance demands, eye tracking can be used for foveated rendering. However, as there are lens distortions in HMDs, there are more possibilities to increase the performance with smarter rendering approaches. We present a new system using optimizations for rendering considering lens astigmatism and combining this with foveated rendering through eye tracking. Depending on the current eye gaze, this delivers a rendering speed-up of up to 20%.
@inproceedings{pohl16_vr,
title = {Combining Eye Tracking with Optimizations for Lens Astigmatism in Modern Wide-Angle HMDs},
author = {Daniel Pohl and Xucong Zhang and Andreas Bulling},
year = {2016},
booktitle = {Proc. IEEE Virtual Reality (VR)},
pages = {269-270},
doi = {10.1109/VR.2016.7504757},
}

AggreGaze: Collective Estimation of Audience Attention on Public Displays
Yusuke Sugano, Xucong Zhang, Andreas Bulling
Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 821-831, 2016.
AbstractLinksBibTeXProject Best Paper Honourable Mention Award
Gaze is frequently explored in public display research given its importance for monitoring and analysing audience attention. However, current gaze-enabled public display interfaces require either special-purpose eye tracking equipment or explicit personal calibration for each individual user. We present AggreGaze, a novel method for estimating spatio-temporal audience attention on public displays. Our method requires only a single off-the-shelf camera attached to the display, does not require any personal calibration, and provides visual attention estimates across the full display. We achieve this by 1) compensating for errors of state-of-the-art appearance-based gaze estimation methods through on-site training data collection, and by 2) aggregating uncalibrated and thus inaccurate gaze estimates of multiple users into joint attention estimates. We propose different visual stimuli for this compensation: a standard 9-point calibration, moving targets, text and visual stimuli embedded into the display content, as well as normal video content. Based on a two-week deployment in a public space, we demonstrate the effectiveness of our method for estimating attention maps that closely resemble ground-truth audience gaze distributions.
@inproceedings{sugano16_uist,
title = {AggreGaze: Collective Estimation of Audience Attention on Public Displays},
author = {Yusuke Sugano and Xucong Zhang and Andreas Bulling},
year = {2016},
booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)},
pages = {821-831},
doi = {10.1145/2984511.2984536},
video = {https://www.youtube.com/watch?v=eFK39S_lgdg},
}

Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments
Marc Tonsen, Xucong Zhang, Yusuke Sugano, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 139-142, 2016.
AbstractLinksBibTeXProject
We present labelled pupils in the wild (LPW), a novel dataset of 66 high-quality, high-speed eye region videos for the development and evaluation of pupil detection algorithms. The videos in our dataset were recorded from 22 participants in everyday locations at about 95 FPS using a state-of-the-art dark-pupil head-mounted eye tracker. They cover people of different ethnicities and a diverse set of everyday indoor and outdoor illumination environments, as well as natural gaze direction distributions. The dataset also includes participants wearing glasses, contact lenses, and make-up. We bench- mark five state-of-the-art pupil detection algorithms on our dataset with respect to robustness and accuracy. We further study the influence of image resolution and vision aids as well as recording lo- cation (indoor, outdoor) on pupil detection performance. Our evaluations provide valuable insights into the general pupil detection problem and allow us to identify key challenges for robust pupil detection on head-mounted eye trackers.
@inproceedings{tonsen16_etra,
title = {Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments},
author = {Marc Tonsen and Xucong Zhang and Yusuke Sugano and Andreas Bulling},
year = {2016},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {139-142},
doi = {10.1145/2857491.2857520},
}
2015

Rendering of Eyes for Eye-Shape Registration and Gaze Estimation
Erroll Wood, Tadas Baltru{\v{s}}aitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, Andreas Bulling
Proc. IEEE International Conference on Computer Vision (ICCV), pp. 3756-3764, 2015.
AbstractLinksBibTeXProject
Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model's controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.
@inproceedings{wood15_iccv,
title = {Rendering of Eyes for Eye-Shape Registration and Gaze Estimation},
author = {Erroll Wood and Tadas Baltru{\v{s}}aitis and Xucong Zhang and Yusuke Sugano and Peter Robinson and Andreas Bulling},
year = {2015},
booktitle = {Proc. IEEE International Conference on Computer Vision (ICCV)},
pages = {3756-3764},
doi = {10.1109/ICCV.2015.428},
}

Appearance-based Gaze Estimation in the Wild
Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling
Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4511-4520, 2015.
AbstractLinksBibTeXProject
Appearance-based gaze estimation is believed to work well in real-world settings but existing datasets were collected under controlled laboratory conditions and methods were not evaluated across multiple datasets. In this work we study appearance-based gaze estimation in the wild. We present the MPIIGaze dataset that contains 213,659 images we collected from 15 participants during natural everyday laptop use over more than three months. Our dataset is significantly more variable than existing datasets with respect to appearance and illumination. We also present a method for in-the-wild appearance-based gaze estimation using multimodal convolutional neural networks, which significantly outperforms state-of-the art methods in the most challenging cross-dataset evaluation setting. We present an extensive evaluation of several state-of-the-art image-based gaze estimation algorithm on three current datasets, including our own. This evaluation provides clear insights and allows us identify key research challenges of gaze estimation in the wild.
@inproceedings{zhang15_cvpr,
title = {Appearance-based Gaze Estimation in the Wild},
author = {Zhang, Xucong and Sugano, Yusuke and Fritz, Mario and Bulling, Andreas},
year = {2015},
booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {4511-4520},
doi = {10.1109/CVPR.2015.7299081},
video = {https://www.youtube.com/watch?v=rw6LZA1USG8},
}