Alert button
Picture for Shixia Liu

Shixia Liu

Alert button

Foundation Models Meet Visualizations: Challenges and Opportunities

Oct 09, 2023
Weikai Yang, Mengchen Liu, Zheng Wang, Shixia Liu

Figure 1 for Foundation Models Meet Visualizations: Challenges and Opportunities
Figure 2 for Foundation Models Meet Visualizations: Challenges and Opportunities
Figure 3 for Foundation Models Meet Visualizations: Challenges and Opportunities

Recent studies have indicated that foundation models, such as BERT and GPT, excel in adapting to a variety of downstream tasks. This adaptability has established them as the dominant force in building artificial intelligence (AI) systems. As visualization techniques intersect with these models, a new research paradigm emerges. This paper divides these intersections into two main areas: visualizations for foundation models (VIS4FM) and foundation models for visualizations (FM4VIS). In VIS4FM, we explore the primary role of visualizations in understanding, refining, and evaluating these intricate models. This addresses the pressing need for transparency, explainability, fairness, and robustness. Conversely, within FM4VIS, we highlight how foundation models can be utilized to advance the visualization field itself. The confluence of foundation models and visualizations holds great promise, but it also comes with its own set of challenges. By highlighting these challenges and the growing opportunities, this paper seeks to provide a starting point for continued exploration in this promising avenue.

* Submitted to Computational Visual Media 
Viaarxiv icon

A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision

Aug 09, 2023
Changjian Chen, Yukai Guo, Fengyuan Tian, Shilong Liu, Weikai Yang, Zhaowei Wang, Jing Wu, Hang Su, Hanspeter Pfister, Shixia Liu

Figure 1 for A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision
Figure 2 for A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision
Figure 3 for A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision
Figure 4 for A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision

Existing model evaluation tools mainly focus on evaluating classification models, leaving a gap in evaluating more complex models, such as object detection. In this paper, we develop an open-source visual analysis tool, Uni-Evaluator, to support a unified model evaluation for classification, object detection, and instance segmentation in computer vision. The key idea behind our method is to formulate both discrete and continuous predictions in different tasks as unified probability distributions. Based on these distributions, we develop 1) a matrix-based visualization to provide an overview of model performance; 2) a table visualization to identify the problematic data subsets where the model performs poorly; 3) a grid visualization to display the samples of interest. These visualizations work together to facilitate the model evaluation from a global overview to individual samples. Two case studies demonstrate the effectiveness of Uni-Evaluator in evaluating model performance and making informed improvements.

* Accepted to IEEE VIS 2023 
Viaarxiv icon

TimePool: Visually Answer "Which and When" Questions On Univariate Time Series

Aug 01, 2023
Tinghao Feng, Yueqi Hu, Jing Yang, Tom Polk, Ye Zhao, Shixia Liu, Zhaocong Yang

Figure 1 for TimePool: Visually Answer "Which and When" Questions On Univariate Time Series
Figure 2 for TimePool: Visually Answer "Which and When" Questions On Univariate Time Series

When exploring time series datasets, analysts often pose "which and when" questions. For example, with world life expectancy data over one hundred years, they may inquire about the top 10 countries in life expectancy and the time period when they achieved this status, or which countries have had longer life expectancy than Ireland and when. This paper proposes TimePool, a new visualization prototype, to address this need for univariate time series analysis. It allows users to construct interactive "which and when" queries and visually explore the results for insights.

Viaarxiv icon

Visual Analytics For Machine Learning: A Data Perspective Survey

Jul 15, 2023
Junpeng Wang, Shixia Liu, Wei Zhang

Figure 1 for Visual Analytics For Machine Learning: A Data Perspective Survey
Figure 2 for Visual Analytics For Machine Learning: A Data Perspective Survey
Figure 3 for Visual Analytics For Machine Learning: A Data Perspective Survey
Figure 4 for Visual Analytics For Machine Learning: A Data Perspective Survey

The past decade has witnessed a plethora of works that leverage the power of visualization (VIS) to interpret machine learning (ML) models. The corresponding research topic, VIS4ML, keeps growing at a fast pace. To better organize the enormous works and shed light on the developing trend of VIS4ML, we provide a systematic review of these works through this survey. Since data quality greatly impacts the performance of ML models, our survey focuses specifically on summarizing VIS4ML works from the data perspective. First, we categorize the common data handled by ML models into five types, explain the unique features of each type, and highlight the corresponding ML models that are good at learning from them. Second, from the large number of VIS4ML works, we tease out six tasks that operate on these types of data (i.e., data-centric tasks) at different stages of the ML pipeline to understand, diagnose, and refine ML models. Lastly, by studying the distribution of 143 surveyed papers across the five data types, six data-centric tasks, and their intersections, we analyze the prospective research directions and envision future research trends.

* 20 pages 
Viaarxiv icon

Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles

Aug 20, 2022
Jun Yuan, Mengchen Liu, Fengyuan Tian, Shixia Liu

Figure 1 for Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles
Figure 2 for Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles
Figure 3 for Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles
Figure 4 for Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles

Recent advances in artificial intelligence largely benefit from better neural network architectures. These architectures are a product of a costly process of trial-and-error. To ease this process, we develop ArchExplorer, a visual analysis method for understanding a neural architecture space and summarizing design principles. The key idea behind our method is to make the architecture space explainable by exploiting structural distances between architectures. We formulate the pairwise distance calculation as solving an all-pairs shortest path problem. To improve efficiency, we decompose this problem into a set of single-source shortest path problems. The time complexity is reduced from O(kn^2N) to O(knN). Architectures are hierarchically clustered according to the distances between them. A circle-packing-based architecture visualization has been developed to convey both the global relationships between clusters and local neighborhoods of the architectures in each cluster. Two case studies and a post-analysis are presented to demonstrate the effectiveness of ArchExplorer in summarizing design principles and selecting better-performing architectures.

* 11 pages, 11 figures; accepted for IEEE VIS 2022 
Viaarxiv icon

A Unified Understanding of Deep NLP Models for Text Classification

Jun 19, 2022
Zhen Li, Xiting Wang, Weikai Yang, Jing Wu, Zhengyan Zhang, Zhiyuan Liu, Maosong Sun, Hui Zhang, Shixia Liu

Figure 1 for A Unified Understanding of Deep NLP Models for Text Classification
Figure 2 for A Unified Understanding of Deep NLP Models for Text Classification
Figure 3 for A Unified Understanding of Deep NLP Models for Text Classification
Figure 4 for A Unified Understanding of Deep NLP Models for Text Classification

The rapid development of deep natural language processing (NLP) models for text classification has led to an urgent need for a unified understanding of these models proposed individually. Existing methods cannot meet the need for understanding different models in one framework due to the lack of a unified measure for explaining both low-level (e.g., words) and high-level (e.g., phrases) features. We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. We model the intra- and inter-word information at each layer measuring the importance of a word to the final prediction as well as the relationships between words, such as the formation of phrases. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples. Two case studies on classification tasks and comparison between models demonstrate that DeepNLPVis can help users effectively identify potential problems caused by samples and model architectures and then make informed improvements.

* 14 pages, 10 figures 
Viaarxiv icon

Diagnosing Ensemble Few-Shot Classifiers

Jun 09, 2022
Weikai Yang, Xi Ye, Xingxing Zhang, Lanxi Xiao, Jiazhi Xia, Zhongyuan Wang, Jun Zhu, Hanspeter Pfister, Shixia Liu

Figure 1 for Diagnosing Ensemble Few-Shot Classifiers
Figure 2 for Diagnosing Ensemble Few-Shot Classifiers
Figure 3 for Diagnosing Ensemble Few-Shot Classifiers
Figure 4 for Diagnosing Ensemble Few-Shot Classifiers

The base learners and labeled samples (shots) in an ensemble few-shot classifier greatly affect the model performance. When the performance is not satisfactory, it is usually difficult to understand the underlying causes and make improvements. To tackle this issue, we propose a visual analysis method, FSLDiagnotor. Given a set of base learners and a collection of samples with a few shots, we consider two problems: 1) finding a subset of base learners that well predict the sample collections; and 2) replacing the low-quality shots with more representative ones to adequately represent the sample collections. We formulate both problems as sparse subset selection and develop two selection algorithms to recommend appropriate learners and shots, respectively. A matrix visualization and a scatterplot are combined to explain the recommended learners and shots in context and facilitate users in adjusting them. Based on the adjustment, the algorithm updates the recommendation results for another round of improvement. Two case studies are conducted to demonstrate that FSLDiagnotor helps build a few-shot classifier efficiently and increases the accuracy by 12% and 21%, respectively.

* Accepted in IEEE TVCG 
Viaarxiv icon

Real-Time Visual Analysis of High-Volume Social Media Posts

Aug 06, 2021
Johannes Knittel, Steffen Koch, Tan Tang, Wei Chen, Yingcai Wu, Shixia Liu, Thomas Ertl

Figure 1 for Real-Time Visual Analysis of High-Volume Social Media Posts
Figure 2 for Real-Time Visual Analysis of High-Volume Social Media Posts
Figure 3 for Real-Time Visual Analysis of High-Volume Social Media Posts
Figure 4 for Real-Time Visual Analysis of High-Volume Social Media Posts

Breaking news and first-hand reports often trend on social media platforms before traditional news outlets cover them. The real-time analysis of posts on such platforms can reveal valuable and timely insights for journalists, politicians, business analysts, and first responders, but the high number and diversity of new posts pose a challenge. In this work, we present an interactive system that enables the visual analysis of streaming social media data on a large scale in real-time. We propose an efficient and explainable dynamic clustering algorithm that powers a continuously updated visualization of the current thematic landscape as well as detailed visual summaries of specific topics of interest. Our parallel clustering strategy provides an adaptive stream with a digestible but diverse selection of recent posts related to relevant topics. We also integrate familiar visual metaphors that are highly interlinked for enabling both explorative and more focused monitoring tasks. Analysts can gradually increase the resolution to dive deeper into particular topics. In contrast to previous work, our system also works with non-geolocated posts and avoids extensive preprocessing such as detecting events. We evaluated our dynamic clustering algorithm and discuss several use cases that show the utility of our system.

* IEEE VIS 2021, to appear in IEEE Transactions on Visualization & Computer Graphics 
Viaarxiv icon

Interactive Steering of Hierarchical Clustering

Sep 21, 2020
Weikai Yang, Xiting Wang, Jie Lu, Wenwen Dou, Shixia Liu

Figure 1 for Interactive Steering of Hierarchical Clustering
Figure 2 for Interactive Steering of Hierarchical Clustering
Figure 3 for Interactive Steering of Hierarchical Clustering
Figure 4 for Interactive Steering of Hierarchical Clustering

Hierarchical clustering is an important technique to organize big data for exploratory data analysis. However, existing one-size-fits-all hierarchical clustering methods often fail to meet the diverse needs of different users. To address this challenge, we present an interactive steering method to visually supervise constrained hierarchical clustering by utilizing both public knowledge (e.g., Wikipedia) and private knowledge from users. The novelty of our approach includes 1) automatically constructing constraints for hierarchical clustering using knowledge (knowledge-driven) and intrinsic data distribution (data-driven), and 2) enabling the interactive steering of clustering through a visual interface (user-driven). Our method first maps each data item to the most relevant items in a knowledge base. An initial constraint tree is then extracted using the ant colony optimization algorithm. The algorithm balances the tree width and depth and covers the data items with high confidence. Given the constraint tree, the data items are hierarchically clustered using evolutionary Bayesian rose tree. To clearly convey the hierarchical clustering results, an uncertainty-aware tree visualization has been developed to enable users to quickly locate the most uncertain sub-hierarchies and interactively improve them. The quantitative evaluation and case study demonstrate that the proposed approach facilitates the building of customized clustering trees in an efficient and effective manner.

* Accepted for IEEE Transactions on Visualization and Computer Graphics (TVCG) 
Viaarxiv icon

Visual Analysis of Discrimination in Machine Learning

Jul 30, 2020
Qianwen Wang, Zhenhua Xu, Zhutian Chen, Yong Wang, Shixia Liu, Huamin Qu

Figure 1 for Visual Analysis of Discrimination in Machine Learning
Figure 2 for Visual Analysis of Discrimination in Machine Learning
Figure 3 for Visual Analysis of Discrimination in Machine Learning
Figure 4 for Visual Analysis of Discrimination in Machine Learning

The growing use of automated decision-making in critical applications, such as crime prediction and college admission, has raised questions about fairness in machine learning. How can we decide whether different treatments are reasonable or discriminatory? In this paper, we investigate discrimination in machine learning from a visual analytics perspective and propose an interactive visualization tool, DiscriLens, to support a more comprehensive analysis. To reveal detailed information on algorithmic discrimination, DiscriLens identifies a collection of potentially discriminatory itemsets based on causal modeling and classification rules mining. By combining an extended Euler diagram with a matrix-based visualization, we develop a novel set visualization to facilitate the exploration and interpretation of discriminatory itemsets. A user study shows that users can interpret the visually encoded information in DiscriLens quickly and accurately. Use cases demonstrate that DiscriLens provides informative guidance in understanding and reducing algorithmic discrimination.

Viaarxiv icon