Biography
Zhiming Hu was a post-doctoral researcher in the Collaborative Artificial Intelligence group and Computational Biophysics and Biorobotics Group (with Syn Schmitt), at the University of Stuttgart, Germany from 2022 to 2025. He obtained his PhD degree in Computer Software and Theory from Peking University, China in 2022. He received his Bachelor's degree in Optical Engineering from Beijing Institute of Technology, China in 2017. His research interests include virtual reality, visual attention, human-computer interaction, eye tracking, and human intention prediction. He served as a reviewer for IEEE VR, ISMAR, TVCG, IMWUT, IJHCI, CVPR, ECCV, and ICCV.
Teaching
- 2024
- Machine Perception and Learning (Teaching Assistant)Master
- 2023
- Machine Perception and Learning (Teaching Assistant)Master
- 2022
- Machine Perception and Learning (Teaching Assistant)Master
Supervision
- 2024
- Understanding 3D Visualisation in VR with Eye Tracking (Research Project)Timo Hüttner§
- 2023
- Eye-body Correlation Analysis and Pose-based Gaze Prediction (M.Sc.)Jiahui Xu
§Co-supervised with Yao Wang
Publications
2026
HAGI++: Head-Assisted Gaze Imputation and Generation
Chuhan Jiao, Zhiming Hu, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2026.
Mobile eye-tracking is crucial for capturing human visual attention in real-world and XR settings, supporting research and human-computer interaction. Yet blinks, pupil-detection errors and lighting changes create missing values that hinder gaze analysis. We present HAGI++, a multi-modal diffusion-based imputation method that, for the first time, leverages integrated head-orientation sensors to exploit the natural correlation between head and eye movements. Using a transformer-based diffusion model, it learns cross-modal dependencies between eye and head data and can additionally incorporate wrist/hand motion when such wearable signals are available. Evaluations on the large-scale Nymeria, Ego-Exo4D and HOT3D datasets show that HAGI++ consistently outperforms traditional interpolation and deep-learning time-series imputation baselines. Statistical analysis confirms that its gaze-velocity distributions closely match real human behaviour, yielding realistic imputations. Even when 100% of gaze data are missing (pure gaze generation), HAGI++ exceeds methods that rely on the visual inputs and the methods rely on full-body motion capture by incorporating wrist motion from commercial wearables. Our approach enables more complete, accurate eye-gaze recordings in real-world contexts, enhancing gaze-based analysis and interaction across many applications.
Paper: jiao26_tvcg.pdf
@article{jiao26_tvcg,
title = {{{HAGI}}++: {{Head-Assisted Gaze Imputation}} and {{Generation}}},
author = {Jiao, Chuhan and Hu, Zhiming and Bulling, Andreas},
year = {2026},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
doi = {10.48550/arXiv.2511.02468},
shorttitle = {{{HAGI}}++},
}
2025
HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality
Zhiming Hu, Guanhua Zhang, Zheming Yin, Daniel Häufle, Syn Schmitt, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2025.
Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications. However, prior works on hand and head modelling in XR only explored a single modality or focused on specific applications. We present HaHeAE - a novel self-supervised method for learning generalisable joint representations of hand and head movements in XR. At the core of our method is an autoencoder (AE) that uses a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder to learn the joint semantic and stochastic representations of hand-head movements. It also features a diffusion-based decoder to reconstruct the original signals. Through extensive evaluations on three public XR datasets, we show that our method 1) significantly outperforms commonly used self-supervised methods by up to 74.1% in terms of reconstruction quality and is generalisable across users, activities, and XR environments, 2) enables new applications, including interpretable hand-head cluster identification and variable hand-head movement generation, and 3) can serve as an effective feature extractor for downstream tasks. Together, these results demonstrate the effectiveness of our method and underline the potential of self-supervised methods for jointly modelling hand-head behaviours in extended reality.
@article{hu25_tvcg,
title = {HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality},
author = {Hu, Zhiming and Zhang, Guanhua and Yin, Zheming and Häufle, Daniel and Schmitt, Syn and Bulling, Andreas},
year = {2025},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
doi = {10.1109/TVCG.2025.3576999},
}
HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination
Zhiming Hu, Daniel Häufle, Syn Schmitt, Andreas Bulling
Proc. ACM SIGGRAPH (SIGGRAPH), 2025.
We present HOIGaze - a novel learning-based approach for gaze estimation during hand-object interactions (HOI) in extended reality (XR). HOIGaze addresses the challenging HOI setting by building on one key insight: The eye, hand, and head movements are closely coordinated during HOIs and this coordination can be exploited to identify samples that are most useful for gaze estimator training - as such, effectively denoising the training data. This denoising approach is in stark contrast to previous gaze estimation methods that treated all training samples as equal. Specifically, we propose: 1) a novel hierarchical framework that first recognises the hand currently visually attended to and then estimates gaze direction based on the attended hand; 2) a new gaze estimator that uses cross-modal Transformers to fuse head and hand-object features extracted using a convolutional neural network and a spatio-temporal graph convolutional network; and 3) a novel eye-head coordination loss that upgrades training samples belonging to the coordinated eye-head movements. We evaluate HOIGaze on the HOT3D and Aria digital twin (ADT) datasets and show that it significantly outperforms state-of-theart methods, achieving an average improvement of 15.6% on HOT3D and 6.0% on ADT in mean angular error. To demonstrate the potential of our method, we further report significant performance improvements for the sample downstream task of eye-based activity recognition on ADT. Taken together, our results underline the significant information content available in eyehand-head coordination and, as such, open up an exciting new direction for learning-based gaze estimation.
@inproceedings{hu25_siggraph,
title = {{HOIGaze}: {Gaze} {Estimation} {During} {Hand}-{Object} {Interactions} in {Extended} {Reality} {Exploiting} {Eye}-{Hand}-{Head} {Coordination}},
author = {Hu, Zhiming and Häufle, Daniel and Schmitt, Syn and Bulling, Andreas},
year = {2025},
booktitle = {Proc. ACM SIGGRAPH (SIGGRAPH)},
doi = {https://doi.org/10.1145/3721238.3730692},
}
2024
DiffEyeSyn: Diffusion-based User-specific Eye Movement Synthesis
Chuhan Jiao, Guanhua Zhang, Zhiming Hu, Andreas Bulling
arXiv:2409.01240, pp. 1--22, 2024.
High-frequency components in eye gaze data contain user-specific information promising for various applications, but existing gaze modelling methods focus on low frequencies of typically not more than 30 Hz. We present DiffEyeSyn – the first computational method to synthesise high-frequency gaze data, including eye movement characteristics specific to individual users. The key idea is to consider the high-frequency, user-specific information as a special type of noise in eye movement data. This perspective reshapes eye movement synthesis into the task of injecting this user-specific noise into any given eye movement sequence. We formulate this injection task as a conditional diffusion process in which the synthesis is conditioned on user-specific embeddings extracted from the gaze data using pre-trained models for user authentication. We propose user identity guidance – a novel loss function that allows our model to preserve user identity while generating human-like eye movements in the spatial domain. Experiment results on two public high-frequency eye movement biometric datasets show that our synthetic eye movements are indistinguishable from real human eye movements. Furthermore, we demonstrate that DiffEyeSyn can be used to synthesise eye gaze data at scale and for different downstream tasks, such as gaze data imputation and gaze data super-resolution. As such, our work lays the methodological foundations for personalised eye movement synthesis that has significant application potential, such as for character animation, eye movement biometrics, or gaze-based activity and context recognition.
Paper Access: https://arxiv.org/abs/2409.01240
@techreport{jiao24_arxiv_2,
title = {DiffEyeSyn: Diffusion-based User-specific Eye Movement Synthesis},
author = {Chuhan Jiao and Guanhua Zhang and Zhiming Hu and Andreas Bulling},
year = {2024},
pages = {1--22},
url = {https://arxiv.org/abs/2409.01240},
}
DisMouse: Disentangling Information from Mouse Movement Data
Guanhua Zhang, Zhiming Hu, Andreas Bulling
Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 1--13, 2024.
Mouse movement data contain rich information about users, performed tasks, and user interfaces, but separating the respective components remains challenging and unexplored. As a first step to address this challenge, we propose DisMouse – the first method to disentangle user-specific and user-independent information and stochastic variations from mouse movement data. At the core of our method is an autoencoder trained in a semi-supervised fashion, consisting of a self-supervised denoising diffusion process and a supervised contrastive user identification module. Through evaluations on three datasets, we show that DisMouse 1) captures complementary information of mouse input, hence providing an interpretable framework for modelling mouse movements, 2) can be used to produce refined features, thus enabling various applications such as personalised and variable mouse data generation, and 3) generalises across different datasets. Taken together, our results underline the significant potential of disentangled representation learning for explainable, controllable, and generalised mouse behaviour modelling.
@inproceedings{zhang24_uist,
title = {DisMouse: Disentangling Information from Mouse Movement Data},
author = {Guanhua Zhang and Zhiming Hu and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)},
pages = {1--13},
doi = {https://doi.org/10.1145/3654777.3676411},
}
GazeMotion: Gaze-guided Human Motion Forecasting
Zhiming Hu, Syn Schmitt, Daniel Häufle, Andreas Bulling
Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1--6, 2024.
AbstractLinksBibTeXProject Oral Presentation
We present GazeMotion, a novel method for human motion forecasting that combines information on past human poses with human eye gaze. Inspired by evidence from behavioural sciences showing that human eye and body movements are closely coordinated, GazeMotion first predicts future eye gaze from past gaze, then fuses predicted future gaze and past poses into a gaze-pose graph, and finally uses a residual graph convolutional network to forecast body motion. We extensively evaluate our method on the MoGaze, ADT, and GIMO benchmark datasets and show that it outperforms state-of-the-art methods by up to 7.4% improvement in mean per joint position error. Using head direction as a proxy to gaze, our method still achieves an average improvement of 5.5%. We finally report an online user study showing that our method also outperforms prior methods in terms of perceived realism. These results show the significant information content available in eye gaze for human motion forecasting as well as the effectiveness of our method in exploiting this information.
@inproceedings{hu24_iros,
title = {GazeMotion: Gaze-guided Human Motion Forecasting},
author = {Zhiming Hu and Syn Schmitt and Daniel Häufle and Andreas Bulling},
year = {2024},
booktitle = {Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages = {1--6},
video = {https://youtu.be/I-ecIvRqOCY?si=kK8SE0r-JadwOKLt},
}
HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes
Zhiming Hu, Zheming Yin, Daniel Häufle, Syn Schmitt, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), pp. 1--11, 2024.
AbstractLinksBibTeXProject Best Journal Paper Award
We present HOIMotion – a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes. Human motion forecasting is important in many augmented reality applications but most existing methods have only used past body poses to predict future motion. HOIMotion first uses an encoder-residual graph convolutional network (GCN) and multi-layer perceptrons to extract features from body poses and egocentric 3D object bounding boxes, respectively. Our method then fuses pose and object features into a novel pose-object graph and uses a residual-decoder GCN to forecast future body motion. We extensively evaluate our method on the Aria digital twin (ADT) and MoGaze datasets and show that HOIMotion consistently outperforms state-of-the-art methods by a large margin of up to 8.7% on ADT and 7.2% on MoGaze in terms of mean per joint position error. Complementing these evaluations, we report a human study (N=20) that shows that the improvements achieved by our method result in forecasted poses being perceived as both more precise and more realistic than those of existing methods. Taken together, these results reveal the significant information content available in egocentric 3D object bounding boxes for human motion forecasting and the effectiveness of our method in exploiting this information.
Paper: hu24_ismar.pdf
@article{hu24_ismar,
title = {HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes},
author = {Zhiming Hu and Zheming Yin and Daniel Häufle and Syn Schmitt and Andreas Bulling},
year = {2024},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
pages = {1--11},
doi = {10.1109/TVCG.2024.3456161},
}
Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses
Zhiming Hu, Jiahui Xu, Syn Schmitt, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), pp. 1--12, 2024.
Human eye gaze plays a significant role in many virtual and augmented reality (VR/AR) applications, such as gaze-contingent rendering, gaze-based interaction, or eye-based activity recognition. However, prior works on gaze analysis and prediction have only explored eye-head coordination and were limited to human-object interactions. We first report a comprehensive analysis of eye-body coordination in various human-object and human-human interaction activities based on four public datasets collected in real-world (MoGaze), VR (ADT), as well as AR (GIMO and EgoBody) environments. We show that in human-object interactions, e.g. pick and place, eye gaze exhibits strong correlations with full-body motion while in human-human interactions, e.g. chat and teach, a person’s gaze direction is correlated with the body orientation towards the interaction partner. Informed by these analyses we then present Pose2Gaze – a novel eye-body coordination model that uses a convolutional neural network and a spatio-temporal graph convolutional neural network to extract features from head direction and full-body poses, respectively, and then uses a convolutional neural network to predict eye gaze. We compare our method with state-of-the-art methods that predict eye gaze only from head movements and show that Pose2Gaze outperforms these baselines with an average improvement of 24.0% on MoGaze, 10.1% on ADT, 21.3% on GIMO, and 28.6% on EgoBody in mean angular error, respectively. We also show that our method significantly outperforms prior methods in the sample downstream task of eye-based activity recognition. These results underline the significant information content available in eye-body coordination during daily activities and open up a new direction for gaze prediction.
Paper: hu24_tvcg.pdf
@article{hu24_tvcg,
title = {Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses},
author = {Zhiming Hu and Jiahui Xu and Syn Schmitt and Andreas Bulling},
year = {2024},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
pages = {1--12},
doi = {10.1109/TVCG.2024.3412190},
}
DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images
Chuhan Jiao, Yao Wang, Guanhua Zhang, Mihai Bâce, Zhiming Hu, Andreas Bulling
arXiv:2403.17477, pp. 1--13, 2024.
We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360° images based on a conditional score-based denoising diffusion model. Generating human gaze on 360° images is important for various human-computer interaction and computer graphics applications, e.g. for creating large-scale eye tracking datasets or for realistic animation of virtual humans. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting crucial parts of natural gaze behaviour. Our method uses features extracted from 360° images as condition and uses two transformers to model the temporal and spatial dependencies of continuous human gaze. We evaluate DiffGaze on two 360° image benchmarks for gaze sequence generation as well as scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks on both benchmarks. We also report a 21-participant user study showing that our method generates gaze sequences that are indistinguishable from real human sequences. Taken together, our evaluations not only demonstrate the effectiveness of DiffGaze but also point towards a new generation of methods that faithfully model the rich spatial and temporal nature of natural human gaze behaviour.
Paper Access: https://arxiv.org/abs/2403.17477
@techreport{jiao24_arxiv,
title = {DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images},
author = {Chuhan Jiao and Yao Wang and Guanhua Zhang and Mihai B{\^a}ce and Zhiming Hu and Andreas Bulling},
year = {2024},
pages = {1--13},
url = {https://arxiv.org/abs/2403.17477},
}
VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour
Yao Wang, Yue Jiang, Zhiming Hu, Constantin Ruhdorfer, Mihai Bâce, Andreas Bulling
Proc. ACM on Human-Computer Interaction (PACM HCI), 8 (ETRA), pp. 1--18, 2024.
Question answering has recently been proposed as a promising means to assess the recallability of information visualisations. However, prior works are yet to study the link between visually encoding a visualisation in memory and recall performance. To fill this gap, we propose VisRecall++ – a novel 40-participant recallability dataset that contains gaze data on 200 visualisations and five question types, such as identifying the title, and finding extreme values.We measured recallability by asking participants questions after they observed the visualisation for 10 seconds.Our analyses reveal several insights, such as saccade amplitude, number of fixations, and fixation duration significantly differ between high and low recallability groups.Finally, we propose GazeRecallNet – a novel computational method to predict recallability from gaze behaviour that outperforms several baselines on this task.Taken together, our results shed light on assessing recallability from gaze behaviour and inform future work on recallability-based visualisation optimisation.
@article{wang24_etra,
title = {VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour},
author = {Yao Wang and Yue Jiang and Zhiming Hu and Constantin Ruhdorfer and Mihai Bâce and Andreas Bulling},
year = {2024},
journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)},
volume = {8},
number = {ETRA},
pages = {1--18},
doi = {10.1145/3655613},
}
PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party Computation
Mayar Elfares, Pascal Reisert, Zhiming Hu, Wenwu Tang, Ralf Küsters, Andreas Bulling
Proc. ACM on Human-Computer Interaction (PACM HCI), 8 (ETRA), pp. 1--23, 2024.
Latest gaze estimation methods require large-scale training data but their collection and exchange pose significant privacy risks. We propose PrivatEyes - the first privacy-enhancing training approach for appearance-based gaze estimation based on federated learning (FL) and secure multi-party computation (MPC). PrivatEyes enables training gaze estimators on multiple local datasets across different users and server-based secure aggregation of the individual estimators' updates. PrivatEyes guarantees that individual gaze data remains private even if a majority of the aggregating servers is malicious. We also introduce a new data leakage attack DualView that shows that PrivatEyes limits the leakage of private training data more effectively than previous approaches. Evaluations on the MPIIGaze, MPIIFaceGaze, GazeCapture, and NVGaze datasets further show that the improved privacy does not lead to a lower gaze estimation accuracy or substantially higher computational costs - both of which are on par with its non-secure counterparts.
doi: 10.1145/3655606
Paper: elfares24_etra.pdf
@article{elfares24_etra,
title = {PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party Computation},
author = {Mayar Elfares and Pascal Reisert and Zhiming Hu and Wenwu Tang and Ralf Küsters and Andreas Bulling},
year = {2024},
journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)},
volume = {8},
number = {ETRA},
pages = {1--23},
doi = {10.1145/3655606},
}
Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour
Guanhua Zhang, Zhiming Hu, Mihai Bâce, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--17, 2024.
The mouse is a pervasive input device used for a wide range of interactive applications. However, computational modelling of mouse behaviour typically requires time-consuming design and extraction of handcrafted features, or approaches that are application-specific. We instead propose Mouse2Vec – a novel self-supervised method designed to learn semantic representations of mouse behaviour that are reusable across users and applications. Mouse2Vec uses a Transformer-based encoder-decoder architecture, which is specifically geared for mouse data: During pretraining, the encoder learns an embedding of input mouse trajectories while the decoder reconstructs the input and simultaneously detects mouse click events. We show that the representations learned by our method can identify interpretable mouse behaviour clusters and retrieve similar mouse trajectories. We also demonstrate on three sample downstream tasks that the representations can be practically used to augment mouse data for training supervised methods and serve as an effective feature extractor.
@inproceedings{zhang24_chi,
title = {Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour},
author = {Guanhua Zhang and Zhiming Hu and Mihai B{\^a}ce and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--17},
doi = {10.1145/3613904.3642141},
}
SalChartQA: Question-driven Saliency on Information Visualisations
Yao Wang, Weitian Wang, Abdullah Abdelhafez, Mayar Elfares, Zhiming Hu, Mihai Bâce, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--14, 2024.
Understanding the link between visual attention and user's needs when visually exploring information visualisations is under-explored due to a lack of large and diverse datasets to facilitate these analyses. To fill this gap, we introduce SalChartQA – a novel crowd-sourced dataset that uses the BubbleView interface as a proxy for human gaze and a question-answering (QA) paradigm to induce different information needs in users. SalChartQA contains 74,340 answers to 6,000 questions on 3,000 visualisations. Informed by our analyses demonstrating the tight correlation between the question and visual saliency, we propose the first computational method to predict question-driven saliency on information visualisations. Our method outperforms state-of-the-art saliency models, improving several metrics, such as the correlation coefficient and the Kullback-Leibler divergence. These results show the importance of information needs for shaping attention behaviour and paving the way for new applications, such as task-driven optimisation of visualisations or explainable AI in chart question-answering.
Paper: wang24_chi.pdf
Supplementary: wang24_chi_sup.pdf
@inproceedings{wang24_chi,
title = {SalChartQA: Question-driven Saliency on Information Visualisations},
author = {Yao Wang and Weitian Wang and Abdullah Abdelhafez and Mayar Elfares and Zhiming Hu and Mihai B{\^a}ce and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--14},
doi = {10.1145/3613904.3642942},
}
GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction
Haodong Yan, Zhiming Hu, Syn Schmitt, Andreas Bulling
Proc. 32nd Pacific Conference on Computer Graphics and Application (PG), pp. 1--10, 2024.
Human motion prediction is important for many virtual and augmented reality (VR/AR) applications such as collision avoidance
and realistic avatar generation. Existing methods have synthesised body motion only from observed past motion, despite the fact
that human eye gaze is known to correlate strongly with body movements and is readily available in recent VR/AR headsets.
We present GazeMoDiff – a novel gaze-guided denoising diffusion model to generate stochastic human motions. Our method
first uses a gaze encoder and a motion encoder to extract the gaze and motion features respectively, then employs a graph
attention network to fuse these features, and finally injects the gaze-motion features into a noise prediction network via a
cross-attention mechanism to progressively generate multiple reasonable human motions in the future. Extensive experiments on
the MoGaze and GIMO datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin in
terms of multi-modal final displacement error (17.3% on MoGaze and 13.3% on GIMO). We further conducted a human study
(N=21) and validated that the motions generated by our method were perceived as both more precise and more realistic than
those of prior methods. Taken together, these results reveal the significant information content available in eye gaze for stochastic
human motion prediction as well as the effectiveness of our method in exploiting this information.
Paper: yan24_pg.pdf
@inproceedings{yan24_pg,
title = {GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction},
author = {Yan, Haodong and Hu, Zhiming and Schmitt, Syn and Bulling, Andreas},
year = {2024},
booktitle = {Proc. 32nd Pacific Conference on Computer Graphics and Application (PG)},
pages = {1--10},
}
2023
SUPREYES: SUPer Resolution for EYES Using Implicit Neural Representation Learning
Chuhan Jiao, Zhiming Hu, Mihai Bâce, Andreas Bulling
Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 1--13, 2023.
We introduce SUPREYES – a novel self-supervised method to increase the spatio-temporal resolution of gaze data recorded using low(er)-resolution eye trackers. Despite continuing advances in eye tracking technology, the vast majority of current eye trackers – particularly mobile ones and those integrated into mobile devices – suffer from low-resolution gaze data, thus fundamentally limiting their practical usefulness. SUPREYES learns a continuous implicit neural representation from low-resolution gaze data to up-sample the gaze data to arbitrary resolutions. We compare our method with commonly used interpolation methods on arbitrary scale super-resolution and demonstrate that SUPREYES outperforms these baselines by a significant margin. We also test on
the sample downstream task of gaze-based user identification and show that our method improves the performance of original low-resolution gaze data and outperforms other baselines. These results are promising as they open up a new direction for increasing eye tracking fidelity as well as enabling new gaze-based applications without the need for new eye tracking equipment.
Paper: jiao23_uist.pdf
@inproceedings{jiao23_uist,
title = {SUPREYES: SUPer Resolution for EYES Using Implicit Neural Representation Learning},
author = {Jiao, Chuhan and Hu, Zhiming and B{\^a}ce, Mihai and Bulling, Andreas},
year = {2023},
booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)},
pages = {1--13},
doi = {10.1145/3586183.3606780},
}
Exploring Natural Language Processing Methods for Interactive Behaviour Modelling
Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling
Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1--22, 2023.
AbstractLinksBibTeXProject Best Student Paper Nomination
Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.
Paper: zhang23_interact.pdf
@inproceedings{zhang23_interact,
title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling},
author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas},
year = {2023},
booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)},
pages = {1--22},
publisher = {Springer},
}
Federated Learning for Appearance-based Gaze Estimation in the Wild
Mayar Elfares, Zhiming Hu, Pascal Reisert, Andreas Bulling, Ralf Küsters
Proceedings of The 1st Gaze Meets ML workshop, PMLR, pp. 20--36, 2023.
Gaze estimation methods have significantly matured in recent years, but the large number of eye images required to train deep learning models poses significant privacy risks. In addition, the heterogeneous data distribution across different users can significantly hinder the training process. In this work, we propose the first federated learning approach for gaze estimation to preserve the privacy of gaze data. We further employ pseudo-gradient optimisation to adapt our federated learning approach to the divergent model updates to address the heterogeneous nature of in-the-wild gaze data in collaborative setups. We evaluate our approach on a real-world dataset (MPIIGaze) and show that our work enhances the privacy guarantees of conventional appearance-based gaze estimation methods, handles the convergence issues of gaze estimators, and significantly outperforms vanilla federated learning by 15.8% (from a mean error of 10.63 degrees to 8.95 degrees). As such, our work paves the way to develop privacy-aware collaborative learning setups for gaze estimation while maintaining the model’s performance.
Paper: elfares23_gmml.pdf
Paper Access: https://proceedings.mlr.press/v210/elfares23a.html
@inproceedings{elfares23_gmml,
title = {Federated Learning for Appearance-based Gaze Estimation in the Wild},
author = {Elfares, Mayar and Hu, Zhiming and Reisert, Pascal and Bulling, Andreas and K{\"u}sters, Ralf},
editor = {Lourentzou, Ismini and Wu, Joy and Kashyap, Satyananda and Karargyris, Alexandros and Celi, Leo Anthony and Kawas, Ban and Talathi, Sachin},
year = {2023},
booktitle = {Proceedings of The 1st Gaze Meets ML workshop, PMLR},
volume = {210},
pages = {20--36},
url = {https://proceedings.mlr.press/v210/elfares23a.html},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
}
2022
Intentional Head-Motion Assisted Locomotion for Reducing Cybersickness
Zehui Lin, Xiang Gu, Sheng Li, Zhiming Hu, Guoping Wang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2022.
We present an efficient locomotion technique that can reduce cybersickness through aligning the visual and vestibular induced self-motion illusion. Our locomotion technique stimulates proprioception consistent with the visual sense by intentional head motion, which includes both the head’s translational movement and yaw rotation. A locomotion event is triggered by the hand-held controller together with an intended physical head motion simultaneously. Based on our method, we further explore the connections between the level of cybersickness and the velocity of self motion through a series of experiments. We first conduct Experiment 1 to investigate the cybersickness induced by different translation velocities using our method and then conduct Experiment 2 to investigate the cybersickness induced by different angular velocities. Our user studies from these two experiments reveal a new finding on the correlation between translation/angular velocities and the level of cybersickness. The cybersickness is greatest at the lowest velocity using our method, and the statistical analysis also indicates a possible U-shaped relation between the translation/angular velocity and cybersickness degree. Finally, we conduct Experiment 3 to evaluate the performances of our method and other commonly-used locomotion approaches, i.e., joystick-based steering and teleportation. The results show that our method can significantly reduce cybersickness compared with the joystick-based steering and obtain a higher presence compared with the teleportation. These advantages demonstrate that our method can be an optional locomotion solution for immersive VR applications using commercially available HMD suites only.
Paper: lin22_tvcg.pdf
@article{lin22_tvcg,
title = {Intentional Head-Motion Assisted Locomotion for Reducing Cybersickness},
author = {Zehui Lin and Xiang Gu and Sheng Li and Zhiming Hu and Guoping Wang},
year = {2022},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
doi = {10.1109/TVCG.2022.3160232},
}
2021
EHTask: Recognizing User Tasks from Eye and Head Movements in Immersive Virtual Reality
Zhiming Hu, Andreas Bulling, Sheng Li, Guoping Wang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 29 (4), pp. 1992--2004, 2021.
Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e. Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask – a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% vs. 62.8%) and on a real-world dataset (61.9% vs. 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.
Paper: hu21_tvcg_2.pdf
@article{hu21_tvcg_2,
title = {EHTask: Recognizing User Tasks from Eye and Head Movements in Immersive Virtual Reality},
author = {Zhiming Hu and Andreas Bulling and Sheng Li and Guoping Wang},
year = {2021},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
volume = {29},
number = {4},
pages = {1992--2004},
doi = {10.1109/TVCG.2021.3138902},
}
FixationNet: Forecasting Eye Fixations in Task-Oriented Virtual Environments
Zhiming Hu, Andreas Bulling, Sheng Li, Guoping Wang
IEEE Transactions on Visualization and Computer Graphics (TVCG), 27 (5), pp. 2681--2690, 2021.
AbstractLinksBibTeXProject TVCG Best Journal Nominees Award
Human visual attention in immersive virtual reality (VR) is key for many important applications, such as content design, gaze-contingent rendering, or gaze-based interaction. However, prior works typically focused on free-viewing conditions that have limited relevance for practical applications. We first collect eye tracking data of 27 participants performing a visual search task in four immersive VR environments. Based on this dataset, we provide a comprehensive analysis of the collected data and reveal correlations between users' eye fixations and other factors, i.e. users' historical gaze positions, task-related objects, saliency information of the VR content, and users' head rotation velocities. Based on this analysis, we propose FixationNet – a novel learning-based model to forecast users' eye fixations in the near future in VR. We evaluate the performance of our model for free-viewing and task-oriented settings and show that it outperforms the state of the art by a large margin of 19.8% (from a mean error of 2.93° to 2.35°) in free-viewing and of 15.1% (from 2.05° to 1.74°) in task-oriented situations. As such, our work provides new insights into task-oriented attention in virtual environments and guides future work on this important topic in VR research.
@article{hu21_tvcg,
title = {FixationNet: Forecasting Eye Fixations in Task-Oriented Virtual Environments},
author = {Zhiming Hu and Andreas Bulling and Sheng Li and Guoping Wang},
year = {2021},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
volume = {27},
number = {5},
pages = {2681--2690},
doi = {10.1109/TVCG.2021.3067779},
url = {https://cranehzm.github.io/FixationNet.html},
}
Eye Fixation Forecasting in Task-Oriented Virtual Reality
Proceedings of the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, pp. 707-708, 2021.
In immersive virtual reality (VR), users' visual attention is crucial for many important applications, including VR content design, gaze-based interaction, and gaze-contingent rendering. Especially, information on users' future eye fixations is key for intelligent user interfaces and has significant relevance for many areas, such as visual attention enhancement, dynamic event triggering, and human-computer interaction. However, previous works typically focused on free-viewing conditions and paid less attention to task-oriented attention. This paper aims at forecasting users' eye fixations in task-oriented virtual reality. To this end, a VR eye tracking dataset that corresponds to different users performing a visual search task in immersive virtual environments is built. A comprehensive analysis of users' eye fixations is performed based on the collected data. The analysis reveals that eye fixations are correlated with users' historical gaze positions, task-related objects, saliency information of the VR content, and head rotation velocities. Based on this analysis, a novel learning-based model is proposed to forecast users' eye fixations in the near future in immersive virtual environments.
Paper: hu21_cvr.pdf
@inproceedings{hu21_cvr,
title = {Eye Fixation Forecasting in Task-Oriented Virtual Reality},
author = {Zhiming Hu},
year = {2021},
booktitle = {Proceedings of the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops},
pages = {707-708},
doi = {10.1109/VRW52623.2021.00236},
}
Research progress of user task prediction and algorithm analysis (in Chinese)
Zhiming Hu, Sheng Li, Meng Gai
Journal of Graphics, 42 (3), pp. 367-375, 2021.
Users’ cognitive behaviors are dramatically influenced by the specific tasks assigned to them. Information on users’ tasks can be applied to many areas, such as human behavior analysis and intelligent human-computer interfaces. It can be used as the input of intelligent systems and enable the systems to automatically adjust their functions according to different tasks. User task prediction refers to the prediction of users’ tasks at hand based on the characteristics of his or her eye movements, the characteristics of scene content, and other related information. User task prediction is a popular research topic in vision research, and researchers have proposed many successful task prediction algorithms. However, the algorithms proposed in prior works mainly focus on a particular scene, and comparison and analysis are absent for these algorithms. This paper presented a review of prior works on task prediction in scenes of images, videos, and real world, and detailed existing task prediction algorithms. Based on a real-world task dataset, this paper evaluated the performances of existing algorithms and conducted the corresponding analysis and discussion. As such, this work can provide meaningful insights for future works on this important topic.
@article{hu21_jg,
title = {Research progress of user task prediction and algorithm analysis (in Chinese)},
author = {Zhiming Hu and Sheng Li and Meng Gai},
year = {2021},
journal = {Journal of Graphics},
volume = {42},
number = {3},
pages = {367-375},
doi = {10.11996/JG.j.2095-302X.2021030367},
}
2020
Gaze Analysis and Prediction in Virtual Reality
Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, pp. 543--544, 2020.
In virtual reality (VR) systems, users’ gaze information has gained importance in recent years. It can be applied to many aspects, including VR content design, eye-movement based interaction, gaze-contingent rendering, etc. In this context, it becomes increasingly important to understand users’ gaze behaviors in virtual reality and to predict users’ gaze positions. This paper presents research in gaze behavior analysis and gaze position prediction in virtual reality. Specifically, this paper focuses on static virtual scenes and dynamic virtual scenes under free-viewing conditions. Users’ gaze data in virtual scenes are collected and statistical analysis is performed on the recorded data. The analysis reveals that users’ gaze positions are correlated with their head rotation velocities and the salient regions of the content. In dynamic scenes, users’ gaze positions also have strong correlations with the positions of dynamic objects. A data-driven eye-head coordination model is proposed for realtime gaze prediction in static scenes and a CNN-based model is derived for predicting gaze positions in dynamic scenes.
Paper: hu20_cvr.pdf
@inproceedings{hu20_cvr,
title = {Gaze Analysis and Prediction in Virtual Reality},
author = {Zhiming Hu},
year = {2020},
booktitle = {Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops},
pages = {543--544},
doi = {10.1109/VRW50115.2020.00123},
}
DGaze: CNN-Based Gaze Prediction in Dynamic Scenes
Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, Dinesh Manocha
IEEE Transactions on Visualization and Computer Graphics, 26 (5), pp. 1902--1911, 2020.
We conduct novel analyses of users' gaze behaviors in dynamic virtual scenes and, based on our analyses, we present a novel CNN-based model called DGaze for gaze prediction in HMD-based applications. We first collect 43 users' eye tracking data in 5 dynamic scenes under free-viewing conditions. Next, we perform statistical analysis of our data and observe that dynamic object positions, head rotation velocities, and salient regions are correlated with users' gaze positions. Based on our analysis, we present a CNN-based model (DGaze) that combines object position sequence, head velocity sequence, and saliency features to predict users' gaze positions. Our model can be applied to predict not only realtime gaze positions but also gaze positions in the near future and can achieve better performance than prior method. In terms of realtime prediction, DGaze achieves a 22.0% improvement over prior method in dynamic scenes and obtains an improvement of 9.5% in static scenes, based on using the angular distance as the evaluation metric. We also propose a variant of our model called DGaze_ET that can be used to predict future gaze positions with higher precision by combining accurate past gaze data gathered using an eye tracker. We further analyze our CNN architecture and verify the effectiveness of each component in our model. We apply DGaze to gaze-contingent rendering and a game, and also present the evaluation results from a user study.
Paper: hu20_tvcg.pdf
@article{hu20_tvcg,
title = {DGaze: CNN-Based Gaze Prediction in Dynamic Scenes},
author = {Zhiming Hu and Sheng Li and Congyi Zhang and Kangrui Yi and Guoping Wang and Dinesh Manocha},
year = {2020},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {26},
number = {5},
pages = {1902--1911},
doi = {10.1109/TVCG.2020.2973473},
}
Temporal continuity of visual attention for future gaze prediction in immersive virtual reality
Zhiming Hu, Sheng Li, Meng Gai
Virtual Reality and Intelligent Hardware (VRIH), 2 (2), pp. 142--152, 2020.
Eye tracking technology is receiving increased attention in the field of virtual reality. Specifically, future gaze prediction is crucial in pre-computation for many applications such as gaze-contingent rendering, advertisement placement, and content-based design. To explore future gaze prediction, it is necessary to analyze the temporal continuity of visual attention in immersive virtual reality. Methods In this paper, the concept of temporal continuity of visual attention is presented. Subsequently, an autocorrelation function method is proposed to evaluate the temporal continuity. Thereafter, the temporal continuity is analyzed in both free-viewing and task-oriented conditions. Results Specifically, in free-viewing conditions, the analysis of a free-viewing gaze dataset indicates that the temporal continuity performs well only within a short time interval. A task-oriented game scene condition was created and conducted to collect users' gaze data. An analysis of the collected gaze data finds the temporal continuity has a similar performance with that of the free-viewing conditions. Temporal continuity can be applied to future gaze prediction and if it is good, users' current gaze positions can be directly utilized to predict their gaze positions in the future. Conclusions The current gaze's future prediction performances are further evaluated in both free-viewing and task-oriented conditions and discover that the current gaze can be efficiently applied to the task of short-term future gaze prediction. The task of long-term gaze prediction still remains to be explored.
Paper: hu20_vrih.pdf
@article{hu20_vrih,
title = {Temporal continuity of visual attention for future gaze prediction in immersive virtual reality},
author = {Zhiming Hu and Sheng Li and Meng Gai},
year = {2020},
journal = {Virtual Reality and Intelligent Hardware (VRIH)},
volume = {2},
number = {2},
pages = {142--152},
doi = {10.1016/j.vrih.2020.01.002},
publisher = {Elsevier},
}
2019
SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction
Zhiming Hu, Congyi Zhang, Sheng Li, Guoping Wang, Dinesh Manocha
IEEE Transactions on Visualization and Computer Graphics (TVCG), 25 (5), pp. 2002--2010, 2019.
We present a novel, data-driven eye-head coordination model that can be used for realtime gaze prediction for immersive HMD-based applications without any external hardware or eye tracker. Our model (SGaze) is computed by generating a large dataset that corresponds to different users navigating in virtual worlds with different lighting conditions. We perform statistical analysis on the recorded data and observe a linear correlation between gaze positions and head rotation angular velocities. We also find that there exists a latency between eye movements and head movements. SGaze can work as a software-based realtime gaze predictor and we formulate a time related function between head movement and eye movement and use that for realtime gaze position prediction. We demonstrate the benefits of SGaze for gaze-contingent rendering and evaluate the results with a user study.
Paper: hu19_tvcg.pdf
@article{hu19_tvcg,
title = {SGaze: A Data-Driven Eye-Head Coordination Model for Realtime Gaze Prediction},
author = {Zhiming Hu and Congyi Zhang and Sheng Li and Guoping Wang and Dinesh Manocha},
year = {2019},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
volume = {25},
number = {5},
pages = {2002--2010},
doi = {10.1109/TVCG.2019.2899187},
}
