Publications
2026
Is This It? Benchmarking Scanpath Metrics for Information Display
Yao Wang, Junichi Nagasawa, Danqing Shi, Chuhan Jiao, Yue Jiang, Andreas Bulling
Proc. Eye Tracking Metrics Workshop (METR), 2026.
AbstractLinksBibTeXProject
Scanpath prediction is a fundamental task in human visual attention research, aiming to simulate user viewing behaviour for given stimuli. While scanpath prediction methods have matured in natural scenes, recent research has expanded to information displays, such as graphical user interfaces and data visualisations. However, there is currently no consensus on which scanpath metrics to use for evaluation, raising concerns regarding the validity and comparability of the proposed methods. This paper benchmarks ten commonly used scanpath metrics across the MASSVIS and UEyes datasets by comparing model predictions with empirical gaze data. We evaluate these metrics with subjective expert ratings of scanpath similarity. Our analysis reveals that vector-based and region-based metrics align more closely with expert ratings than pixel-based and recurrence-based metrics. Based on these findings, we provide best practices for evaluating visual scanpaths in information displays, emphasising the urgent need for appropriate metrics to ensure the validity of future research.
@inproceedings{wang26_metr,
title = {Is This It? {{Benchmarking Scanpath Metrics}} for {{Information Display}}},
author = {Wang, Yao and Nagasawa, Junichi and Shi, Danqing and Jiao, Chuhan and Jiang, Yue and Bulling, Andreas},
year = {2026},
booktitle = {Proc. Eye Tracking Metrics Workshop (METR)},
number = {ETRA},
doi = {10.1145/3797246.3805691},
}
DiffGaze: A Diffusion Model for Modelling Fine-grained Human Gaze Behaviour on 360° Images
Chuhan Jiao, Yao Wang, Guanhua Zhang, Mihai Bâce, Zhiming Hu, Andreas Bulling
ACM Transactions on Interactive Intelligent Systems (TiiS), pp. 1--13, 2026.
AbstractLinksBibTeXProject
Modelling human gaze behaviour on 360° images is important for various human-computer interaction applications. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting fine-grained gaze behaviour such as saccadic eye movements that can be captured by commercial eye-trackers. We introduce a more challenging task—fine-grained gaze sequence generation. This task aims to generate eye-tracker-like gaze data for given stimuli. We propose DiffGaze, a diffusion-based method for generating realistic and diverse fine-grained human gaze sequences conditioned on 360◦ images. We evaluate DiffGaze on two 360◦ image benchmarks for fine-grained gaze sequence generation as well as two downstream tasks, scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms the fine-grained gaze generation baselines in all tasks on both benchmarks. We also report a 21-participant survey study showing that our method generates gaze sequences that are indistinguishable from real human sequences. Taken together, our evaluations not only demonstrate the effectiveness of DiffGaze but also point towards a new generation of methods that faithfully model the rich spatial and temporal nature of natural human gaze behaviour.
@article{jiao25_tiis,
title = {DiffGaze: A Diffusion Model for Modelling Fine-grained Human Gaze Behaviour on 360° Images},
author = {Chuhan Jiao and Yao Wang and Guanhua Zhang and Mihai B{\^a}ce and Zhiming Hu and Andreas Bulling},
year = {2026},
journal = {ACM Transactions on Interactive Intelligent Systems (TiiS)},
pages = {1--13},
doi = {10.1145/3772075},
}
2025
Towards a Better Understanding of Graph Perception in Immersive Environments
Lin Zhang, Yao Wang, Ying Zhang, Wilhelm Kerle-Malcharek, Karsten Klein, Falk Schreiber, Andreas Bulling
Proc. 33rd International Symposium on Graph Drawing and Network Visualization (GD), pp. 1--19, 2025.
AbstractLinksBibTeXProject
As Immersive Analytics (IA) increasingly uses Virtual Reality (VR) for stereoscopic 3D (S3D) graph visualisation, it is crucial to understand how users perceive network structures in these immersive environments. However, little is known about how humans read S3D graphs during task solving, and how gaze behaviour indicates task performance. To address this gap, we report a user study with 18 participants asked to perform three analytical tasks on S3D graph visualisations in a VR environment. Our findings reveal systematic relationships between network structural properties and gaze behaviour. Based on these insights, we contribute a comprehensive eye tracking methodology for analysing human perception in immersive environments and establish eye tracking as a valuable tool for objectively evaluating cognitive load in S3D graph visualisation.
@inproceedings{zhang25_gd,
title = {Towards a Better Understanding of Graph Perception in Immersive Environments},
author = {Zhang, Lin and Wang, Yao and Zhang, Ying and Kerle-Malcharek, Wilhelm and Klein, Karsten and Schreiber, Falk and Bulling, Andreas},
year = {2025},
booktitle = {Proc. 33rd International Symposium on Graph Drawing and Network Visualization (GD)},
pages = {1--19},
doi = {https://doi.org/10.4230/LIPIcs.GD.2025.11},
}
Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attention
Minsuk Chang, Yao Wang, Huichen Will Wang, Yuanhong Zhou, Andreas Bulling, Cindy Xiong Bearfield
Proc. International Conference on Visualisation (VIS), pp. 1--11, 2025.
AbstractLinksBibTeXProject
Accounting for individual differences can improve the effectiveness of visualization design. While the role of visual attention in visualization interpretation is well recognized, existing work often overlooks how this behavior varies based on visual literacy levels. Based on data from a 235-participant user study covering three visualization tests (mini-VLAT, CALVI, and SGL), we show that distinct attention patterns in visual data exploration can correlate with participants' literacy levels: While experts generally show a strong attentional focus, novices focus less and explore more. We then propose two computational models leveraging these insights: Lit2Sal - a novel visual saliency model that predicts observer attention given their visualization literacy level, and Sal2Lit - a model to predict visual literacy from human visual attention data. Our quantitative and qualitative evaluation demonstrates that Lit2Sal outperforms state-of-the-art saliency models with literacy-aware considerations. Sal2Lit predicts literacy with 86% accuracy using a single attention map, providing a time-efficient supplement to literacy assessment that only takes less than a minute. Taken together, our unique approach to consider individual differences in salience models and visual attention in literacy assessments paves the way for new directions in personalized visual data communication to enhance understanding.
@inproceedings{chang25_vis,
title = {Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attention},
author = {Chang, Minsuk and Wang, Yao and Wang, Huichen Will and Zhou, Yuanhong and Bulling, Andreas and Bearfield, Cindy Xiong},
year = {2025},
booktitle = {Proc. International Conference on Visualisation (VIS)},
pages = {1--11},
doi = {https://doi.ieeecomputersociety.org/10.1109/TVCG.2025.3634815},
}
AttentionLeak: What Does Human Attention Reveal About Information Visualisation?
Malte Sönnichsen, Mayar Elfares, Yao Wang, Ralf Küsters, Alina Roitberg, Andreas Bulling
Proc. International Conference on Document Analysis and Recognition (ICDAR), pp. 77--95, 2025.
AbstractLinksBibTeXProject
In scenarios where direct access to displayed content, such as secured web pages or confidential documents, is restricted, eye-tracking data can serve as a side channel for information inference. Represented as human attention maps, eye tracking data is widely used in research, for example, to quantify how users explore visual information. In this work, we specifically focus on visual question-answering (VQA) scenarios to demonstrate, for the first time, that a rich amount of information can be leaked solely from human attention maps. Hence, we assume that an adversary only has access to the gaze attention maps and aims to derive a range of attributes about the image (e.g. the chart type), the question (e.g. question type), and the answer (e.g. the accuracy-based complexity). This information leakage could be the first step towards potentially more complex insights about human perception and cognition. Our experiments demonstrate that deriving attributes is feasible, and simultaneously predicting multiple attributes improves the success rate for attributes that are difficult to infer. This paper highlights potential threats, encouraging the community to address these concerns and develop appropriate privacy-preserving solutions.
@inproceedings{sonnichsen25_icdar,
title = {AttentionLeak: What Does Human Attention Reveal About Information Visualisation?},
author = {Sönnichsen, Malte and Elfares, Mayar and Wang, Yao and Küsters, Ralf and Roitberg, Alina and Bulling, Andreas},
year = {2025},
booktitle = {Proc. International Conference on Document Analysis and Recognition (ICDAR)},
pages = {77--95},
doi = {10.1007/978-3-032-04627-7_5},
}
ChartOptimiser: Task-driven Optimisation of Chart Designs
Yao Wang, Danqing Shi, Jiarong Pan, Zhiming Hu, Antti Oulasvirta, Andreas Bulling
arXiv:2504.10180, pp. 1--20, 2025.
AbstractLinksBibTeXProject
Effective chart design is essential for satisfying viewers' information needs, such as retrieving values from a chart or comparing two values.
However, creating effective charts is challenging and time-consuming due to the large design space and the inter-dependencies between individual design parameters.
To address this challenge, we propose ChartOptimiser – a Bayesian approach for task-driven optimisation of charts, such as bar charts.
At the core of ChartOptimiser is a novel objective function to automatically optimise an eight-dimensional design space combining four perceptual metrics: visual saliency, text legibility, colour preference, and white space ratio.
Through empirical evaluation on 12 bar charts and four common analytical tasks – finding the extreme value, retrieving a value, comparing two values, and computing a derived value – we show that ChartOptimiser outperforms existing design baselines concerning task-solving ease, visual aesthetics, and chart clarity.
We also discuss two practical applications of ChartOptimiser: generating charts for accessibility and content localisation.
Taken together, ChartOptimiser opens up an exciting new research direction in automated chart design where charts are optimised for users' information needs, preferences, and contexts.
@techreport{wang25_arxiv,
title = {ChartOptimiser: Task-driven Optimisation of Chart Designs},
author = {Yao Wang and Danqing Shi and Jiarong Pan and Zhiming Hu and Antti Oulasvirta and Andreas Bulling},
year = {2025},
pages = {1--20},
url = {https://arxiv.org/abs/2504.10180},
}
Grid Labeling: Crowdsourcing Task-Specific Importance from Visualizations
Minsuk Chang, Yao Wang, Huichen Will Wang, Andreas Bulling, Cindy Xiong Bearfield
Proc. 27th Annual Conference on Data Visualization (EuroVis), pp. 1--6, 2025.
AbstractLinksBibTeXProject
Knowing where people look in visualizations is key to effective design. Yet, existing research primarily focuses on task-agnostic saliency models, although visual attention is inherently task-dependent. Collecting task-relevant importance data remains a resource-intensive challenge. To address this, we introduce Grid Labeling, a novel annotation method for collecting task-specific importance data to enhance saliency prediction models. Grid Labeling dynamically segments visualizations into Adaptive Grids, enabling efficient, low-effort annotation while adapting to visualization structure. We conducted a human-subject study comparing Grid Labeling with existing annotation methods, ImportAnnots, and BubbleView across multiple metrics. Results show that Grid Labeling produces the least noisy data and the highest inter-participant agreement with fewer participants while requiring less physical (e.g., clicks or mouse movements) and cognitive effort.
@inproceedings{chang25_eurovis,
title = {Grid Labeling: Crowdsourcing Task-Specific Importance from Visualizations},
author = {Minsuk Chang and Yao Wang and Huichen Will Wang and Andreas Bulling and Cindy Xiong Bearfield},
year = {2025},
booktitle = {Proc. 27th Annual Conference on Data Visualization (EuroVis)},
pages = {1--6},
doi = {10.2312/evs.20251092},
}
RelEYEance: Gaze-based Assessment of Users’ AI-reliance at Run-time
Zekun Wu, Yao Wang, Markus Langer, Anna Maria Feit
Proc. ACM on Human-Computer Interaction (PACM HCI), 9 (ETRA16), pp. 1--18, 2025.
AbstractLinksBibTeXProject
In time-critical detection tasks,such as drone monitoring, a key condition for users to effectively leverage AI assistance is to find an appropriate trade-off between making fast decisions and verifying AI suggestions, which we refer to as appropriate user reliance. However, assessing such reliance is often oversimplified by focusing solely on task outcomes, potentially overlooking whether users properly verify AI messages. We collected eye-tracking data from an AI-assisted monitoring task and developed a gaze-based reliance model: RelEYEance, to assess the extent of user reliance on AI-suggested alarms. We found that gaze patterns related to verification behaviors distinguish between appropriate reliance, over-reliance, and under-reliance, influencing task performance. We validated our model in a second user study, showing it can reliably detect users' over- and under-reliance at run-time, which could be used e.g. for issuing intervention messages. The results demonstrate the potential for real-time human-AI reliance assessment, facilitating adaptive reliance calibration.
@article{wu25_etra,
title = {RelEYEance: Gaze-based Assessment of Users’ AI-reliance at Run-time},
author = {Zekun Wu and Yao Wang and Markus Langer and Anna Maria Feit},
year = {2025},
journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)},
volume = {9},
number = {ETRA16},
pages = {1--18},
doi = {10.1145/3725841},
}
ChartQC: Question Classification from Human Attention Data on Charts
Takumi Nishiyasu, Tobias Kostorz, Yao Wang, Yoichi Sato, Andreas Bulling
Proc. Eye Tracking and Visualization Workshop (ETVIS), pp. 1--6, 2025.
AbstractLinksBibTeXProject
Understanding how humans interact with information visualizations is crucial for improving user experience and designing effective visualization systems. While previous studies have focused on task-agnostic visual attention, the relationship between attention patterns and visual analytical tasks remains underexplored. This paper investigates how attention data on charts can be used to classify question types, providing insights into question-driven gaze behaviors. We propose ChartQC, a question classification model leveraging spatial feature alignment in chart images and visual attention data. By aligning spatial features, our approach strengthens the integration of visual and attentional cues, improving classification accuracy. These findings help deepen the understanding of user perception in charts and provide a basis for future research on interactive visual analysis.
@inproceedings{nishiyasu25_etvis,
title = {ChartQC: Question Classification from Human Attention Data on Charts},
author = {Takumi Nishiyasu and Tobias Kostorz and Yao Wang and Yoichi Sato and Andreas Bulling},
year = {2025},
booktitle = {Proc. Eye Tracking and Visualization Workshop (ETVIS)},
pages = {1--6},
doi = {10.1145/3715669.3725883},
}
Chartist: Task-driven Eye Movement Control for Chart Reading
Danqing Shi, Yao Wang, Yunpeng Bai, Andreas Bulling, Antti Oulasvirta
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--14, 2025.
AbstractLinksBibTeXProject
To design data visualizations that are easy to comprehend, we need to understand how people with different interests read them. Computational models of modeling scanpaths on charts could complement empirical studies by offering estimates of user performance inexpensively; however, previous models have been limited to gaze patterns and overlooked the effects of tasks. Here, we contribute a model that simulates how users move their eyes to extract information from the chart in order to solve analytical tasks, including retrieve value, filter, and find extreme. Our insight is a two-level hierarchical control structure. At the high level, the model uses a LLM to comprehend information gained so far and uses this representation to select a goal for the lower-level controllers, which in turn move the eyes according to a sampling policy learned via reinforcement learning. The model can accurately predict task-driven scanpaths and reproduce the human-like statistical summary across tasks.
@inproceedings{shi25_chi,
title = {Chartist: Task-driven Eye Movement Control for Chart Reading},
author = {Shi, Danqing and Wang, Yao and Bai, Yunpeng and Bulling, Andreas and Oulasvirta, Antti},
year = {2025},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--14},
doi = {10.1145/3706598.3713128},
}
2024
VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour
Yao Wang, Yue Jiang, Zhiming Hu, Constantin Ruhdorfer, Mihai Bâce, Andreas Bulling
Proc. ACM on Human-Computer Interaction (PACM HCI), 8 (ETRA), pp. 1--18, 2024.
AbstractLinksBibTeXProject
Question answering has recently been proposed as a promising means to assess the recallability of information visualisations. However, prior works are yet to study the link between visually encoding a visualisation in memory and recall performance. To fill this gap, we propose VisRecall++ – a novel 40-participant recallability dataset that contains gaze data on 200 visualisations and five question types, such as identifying the title, and finding extreme values.We measured recallability by asking participants questions after they observed the visualisation for 10 seconds.Our analyses reveal several insights, such as saccade amplitude, number of fixations, and fixation duration significantly differ between high and low recallability groups.Finally, we propose GazeRecallNet – a novel computational method to predict recallability from gaze behaviour that outperforms several baselines on this task.Taken together, our results shed light on assessing recallability from gaze behaviour and inform future work on recallability-based visualisation optimisation.
@article{wang24_etra,
title = {VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze Behaviour},
author = {Yao Wang and Yue Jiang and Zhiming Hu and Constantin Ruhdorfer and Mihai Bâce and Andreas Bulling},
year = {2024},
journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)},
volume = {8},
number = {ETRA},
pages = {1--18},
doi = {10.1145/3655613},
}
Saliency3D: a 3D Saliency Dataset Collected on Screen
Yao Wang, Qi Dai, Mihai Bâce, Karsten Klein, Andreas Bulling
Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA), pp. 1--9, 2024.
AbstractLinksBibTeXProject
While visual saliency has recently been studied in 3D, the experimental setup for collecting 3D saliency data can be expensive and cumbersome. To address this challenge, we propose a novel experimental design that utilizes an eye tracker on a screen to collect 3D saliency data. Our experimental design reduces the cost and complexity of 3D saliency dataset collection. We first collect gaze data on a screen, then we map them to 3D saliency data through perspective transformation. Using this method, we collect a 3D saliency dataset (49,276 fixations) comprising 10 participants looking at sixteen objects. Moreover, we examine the viewing preferences for objects and discuss our findings in this study. Our results indicate potential preferred viewing directions and a correlation between salient features and the variation in viewing directions.
@inproceedings{wang24_etras,
title = {Saliency3D: a 3D Saliency Dataset Collected on Screen},
author = {Yao Wang and Qi Dai and Mihai B{\^a}ce and Karsten Klein and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM International Symposium on Eye Tracking Research and Applications (ETRA)},
pages = {1--9},
doi = {10.1145/3649902.3653350},
}
SalChartQA: Question-driven Saliency on Information Visualisations
Yao Wang, Weitian Wang, Abdullah Abdelhafez, Mayar Elfares, Zhiming Hu, Mihai Bâce, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--14, 2024.
AbstractLinksBibTeXProject
Understanding the link between visual attention and user's needs when visually exploring information visualisations is under-explored due to a lack of large and diverse datasets to facilitate these analyses. To fill this gap, we introduce SalChartQA – a novel crowd-sourced dataset that uses the BubbleView interface as a proxy for human gaze and a question-answering (QA) paradigm to induce different information needs in users. SalChartQA contains 74,340 answers to 6,000 questions on 3,000 visualisations. Informed by our analyses demonstrating the tight correlation between the question and visual saliency, we propose the first computational method to predict question-driven saliency on information visualisations. Our method outperforms state-of-the-art saliency models, improving several metrics, such as the correlation coefficient and the Kullback-Leibler divergence. These results show the importance of information needs for shaping attention behaviour and paving the way for new applications, such as task-driven optimisation of visualisations or explainable AI in chart question-answering.
@inproceedings{wang24_chi,
title = {SalChartQA: Question-driven Saliency on Information Visualisations},
author = {Yao Wang and Weitian Wang and Abdullah Abdelhafez and Mayar Elfares and Zhiming Hu and Mihai B{\^a}ce and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--14},
doi = {10.1145/3613904.3642942},
}
2023
Scanpath Prediction on Information Visualisations
Yao Wang, Mihai Bâce, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 30 (7), pp. 3902--3914, 2023.
AbstractLinksBibTeXProject
We propose Unified Model of Saliency and Scanpaths (UMSS) – a model that learns to predict multi-duration saliency and scanpaths (i.e. sequences of eye fixations) on information visualisations. Although scanpaths provide rich information about the importance of different visualisation elements during the visual exploration process, prior work has been limited to predicting aggregated attention statistics, such as visual saliency. We present in-depth analyses of gaze behaviour for different information visualisation elements (e.g. Title, Label, Data) on the popular MASSVIS dataset. We show that while, overall, gaze patterns are surprisingly consistent across visualisations and viewers, there are also structural differences in gaze dynamics for different elements. Informed by our analyses, UMSS first predicts multi-duration element-level saliency maps, then probabilistically samples scanpaths from them. Extensive experiments on MASSVIS show that our method consistently outperforms state-of-the-art methods with respect tto several, widely used scanpath and saliency evaluation metrics. Our method achieves a relative improvement in sequence score of 11.5 % for scanpath prediction, and a relative improvement in Pearson correlation coefficient of up to 23.6 % for saliency prediction. These results are auspicious and point towards richer user models and simulations of visual attention on visualisations without the need for any eye tracking equipment.
@article{wang23_tvcg,
title = {Scanpath Prediction on Information Visualisations},
author = {Yao Wang and Mihai Bâce and Andreas Bulling},
year = {2023},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
volume = {30},
number = {7},
pages = {3902--3914},
doi = {10.1109/TVCG.2023.3242293},
}
Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention
Ekta Sood, Lei Shi, Matteo Bortoletto, Yao Wang, Philipp Müller, Andreas Bulling
Proc. Annual Meeting of the Cognitive Science Society (CogSci), pp. 3639--3646, 2023.
AbstractLinksBibTeXProject
We present a novel method for saliency prediction that leverages a cognitive model of visual attention as an inductive bias. This approach is in stark contrast to recent purely data-driven saliency models that achieve performance improvements mainly by increased capacity, resulting in high computational costs and the need for large-scale training datasets. We demonstrate that by using a cognitive model, our method achieves competitive performance to the state of the art across several natural image datasets while only requiring a fraction of the parameters. Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization. We further provide augmented versions of the full MSCOCO dataset with synthetic gaze data using the cognitive model, which we used to pre-train our method. Our results are highly promising and underline the significant potential of bridging between cognitive and data-driven models, potentially also beyond attention.
@inproceedings{sood23_cogsci,
title = {Improving Neural Saliency Prediction with a Cognitive Model of Human Visual Attention},
author = {Ekta Sood and Lei Shi and Matteo Bortoletto and Yao Wang and Philipp Müller and Andreas Bulling},
year = {2023},
booktitle = {Proc. Annual Meeting of the Cognitive Science Society (CogSci)},
pages = {3639--3646},
}
2022
VisRecall: Quantifying Information Visualisation Recallability via Question Answering
Yao Wang, Chuhan Jiao, Mihai Bâce, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), 28 (12), pp. 4995-5005, 2022.
AbstractLinksBibTeXProject
Despite its importance for assessing the effectiveness of communicating information visually, fine-grained recallability of information visualisations has not been studied quantitatively so far. In this work, we propose a question-answering paradigm to study visualisation recallability and present VisRecall - a novel dataset consisting of 200 visualisations that are annotated with crowd-sourced human (N = 305) recallability scores obtained from 1,000 questions of five question types. Furthermore, we present the first computational method to predict recallability of different visualisation elements, such as the title or specific data values. We report detailed analyses of our method on VisRecall and demonstrate that it outperforms several baselines in overall recallability and FE-, F-, RV-, and U-question recallability. Our work makes fundamental contributions towards a new generation of methods to assist designers in optimising visualisations.
@article{wang22_tvcg,
title = {VisRecall: Quantifying Information Visualisation Recallability via Question Answering},
author = {Yao Wang and Chuhan Jiao and Mihai Bâce and Andreas Bulling},
year = {2022},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
volume = {28},
number = {12},
pages = {4995-5005},
doi = {10.1109/TVCG.2022.3198163},
}
Impact of Gaze Uncertainty on AOIs in Information Visualisations
Yao Wang, Maurice Koch, Mihai Bâce, Daniel Weiskopf, Andreas Bulling
ETRA Workshop on Eye Tracking and Visualization (ETVIS), pp. 1--6, 2022.
AbstractLinksBibTeXProject
Gaze-based analysis of areas of interest (AOIs) is widely used in information visualisation research to understand how people explore visualisations or assess the quality of visualisations concerning key characteristics such as memorability. However, nearby AOIs in visualisations amplify the uncertainty caused by the gaze estimation error, which strongly influences the mapping between gaze samples or fixations and different AOIs. We contribute a novel investigation into gaze uncertainty and quantify its impact on AOI-based analysis on visualisations using two novel metrics: the Flipping Candidate Rate (FCR) and Hit Any AOI Rate (HAAR). Our analysis of 40 real-world visualisations, including human gaze and AOI annotations, shows that gaze uncertainty frequently and significantly impacts the analysis conducted in AOI-based studies. Moreover, we analysed four visualisation types and found that bar and scatter plots are usually designed in a way that causes more uncertainty than line and pie plots in gaze-based analysis.
@inproceedings{wang22_etvis,
title = {Impact of Gaze Uncertainty on AOIs in Information Visualisations},
author = {Yao Wang and Maurice Koch and Mihai B{\^a}ce and Daniel Weiskopf and Andreas Bulling},
year = {2022},
booktitle = {ETRA Workshop on Eye Tracking and Visualization (ETVIS)},
pages = {1--6},
doi = {10.1145/3517031.3531166},
}
2020
Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images
Zizhuang Wei, Yao Wang, Hongwei Yi, Yisong Chen, Guoping Wang
Applied Sciences, 10 (4), pp. 1275, 2020.
AbstractLinksBibTeXProject
Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error.
@article{wei20_as,
title = {Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images},
author = {Wei, Zizhuang and Wang, Yao and Yi, Hongwei and Chen, Yisong and Wang, Guoping},
year = {2020},
journal = {Applied Sciences},
volume = {10},
number = {4},
pages = {1275},
doi = {10.3390/app10041275},
}
2018
Large-Scale Structure from Motion with Semantic Constraints of Aerial Images
Yu Chen, Yao Wang, Peng Lu, Yisong Chen, Guoping Wang
Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 347-359, 2018.
AbstractLinksBibTeXProject
Structure from Motion (SfM) and semantic segmentation are two branches of computer vision. However, few previous methods integrate the two branches together. SfM is limited by the precision of traditional feature detecting method, especially in complicated scenes. As the research field of semantic segmentation thrives, we could gain semantic information of high confidence in each specific task with little effort. By utilizing semantic segmentation information, our paper presents a new way to boost the accuracy of feature point matching. Besides, with the semantic constraints taken from the result of semantic segmentation, a new bundle adjustment method with equality constraint is proposed. By exploring the sparsity of equality constraint, it indicates that constrained bundle adjustment can be solved by Sequential Quadratic Programming (SQP) efficiently. The proposed approach achieves state of the art accuracy, and, by grouping the descriptors together by their semantic labels, the speed of putative matches is slightly boosted. Moreover, our approach demonstrates a potential of automatic labeling of semantic segmentation. In a nutshell, our work strongly verifies that SfM and semantic segmentation benefit from each other.
@inproceedings{chen18_prcv,
title = {Large-Scale Structure from Motion with Semantic Constraints of Aerial Images},
author = {Chen, Yu and Wang, Yao and Lu, Peng and Chen, Yisong and Wang, Guoping},
year = {2018},
booktitle = {Proc. Chinese Conference on Pattern Recognition and Computer Vision (PRCV)},
pages = {347-359},
doi = {10.1007/978-3-030-03398-9_30},
}
Sobel Heuristic Kernel for Aerial Semantic Segmentation
Tao Hu, Yao Wang, Yisong Chen, Peng Lu, Heng Wang, Guoping Wang
IEEE International Conference on Image Processing (ICIP), pp. 3074-3078, 2018.
AbstractLinksBibTeXProject
Misclassification in semantic segmentation mostly occurs in the pixels around the semantic contour. In this work, we address the task of aerial image segmentation by borrowing the kernel prior from classical edge detecting operator. We propose a module called Sobel Heuristic Kernel(SHK). Our work makes several main contributions and experimentally shows good performance. To the best of our knowledge, we are the first to combine traditional edge detection method and deep learning method in semantic segmentation. Our SHK module reaches state of the art in the Inria Aerial Image Labeling dataset.
@inproceedings{hu18_icip,
title = {Sobel Heuristic Kernel for Aerial Semantic Segmentation},
author = {Hu, Tao and Wang, Yao and Chen, Yisong and Lu, Peng and Wang, Heng and Wang, Guoping},
year = {2018},
booktitle = {IEEE International Conference on Image Processing (ICIP)},
pages = {3074-3078},
doi = {10.1109/ICIP.2018.8451170},
}
