Alert button
Picture for Zihan Yan

Zihan Yan

Alert button

Relightable and Animatable Neural Avatar from Sparse-View Video

Aug 17, 2023
Zhen Xu, Sida Peng, Chen Geng, Linzhan Mou, Zihan Yan, Jiaming Sun, Hujun Bao, Xiaowei Zhou

Figure 1 for Relightable and Animatable Neural Avatar from Sparse-View Video
Figure 2 for Relightable and Animatable Neural Avatar from Sparse-View Video
Figure 3 for Relightable and Animatable Neural Avatar from Sparse-View Video
Figure 4 for Relightable and Animatable Neural Avatar from Sparse-View Video

This paper tackles the challenge of creating relightable and animatable neural avatars from sparse-view (or even monocular) videos of dynamic humans under unknown illumination. Compared to studio environments, this setting is more practical and accessible but poses an extremely challenging ill-posed problem. Previous neural human reconstruction methods are able to reconstruct animatable avatars from sparse views using deformed Signed Distance Fields (SDF) but cannot recover material parameters for relighting. While differentiable inverse rendering-based methods have succeeded in material recovery of static objects, it is not straightforward to extend them to dynamic humans as it is computationally intensive to compute pixel-surface intersection and light visibility on deformed SDFs for inverse rendering. To solve this challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to approximate the world space distances under arbitrary human poses. Specifically, we estimate coarse distances based on a parametric human model and compute fine distances by exploiting the local deformation invariance of SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently estimate the surface intersection and light visibility. This allows us to develop the first system to recover animatable and relightable neural avatars from sparse view (or monocular) inputs. Experiments demonstrate that our approach is able to produce superior results compared to state-of-the-art methods. Our code will be released for reproducibility.

* Project page: https://zju3dv.github.io/relightable_avatar 
Viaarxiv icon

XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis

Jan 25, 2023
Yingchaojie Feng, Xingbo Wang, Bo Pan, Kam Kwai Wong, Yi Ren, Shi Liu, Zihan Yan, Yuxin Ma, Huamin Qu, Wei Chen

Figure 1 for XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
Figure 2 for XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
Figure 3 for XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
Figure 4 for XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis

Natural language interfaces (NLIs) enable users to flexibly specify analytical intentions in data visualization. However, diagnosing the visualization results without understanding the underlying generation process is challenging. Our research explores how to provide explanations for NLIs to help users locate the problems and further revise the queries. We present XNLI, an explainable NLI system for visual data analysis. The system introduces a Provenance Generator to reveal the detailed process of visual transformations, a suite of interactive widgets to support error adjustments, and a Hint Generator to provide query revision hints based on the analysis of user queries and interactions. Two usage scenarios of XNLI and a user study verify the effectiveness and usability of the system. Results suggest that XNLI can significantly enhance task accuracy without interrupting the NLI-based analysis process.

* 14 pages, 7 figures. A preprint version of a publication at IEEE Transactions on Visualization and Computer Graphics (TVCG), 2023 
Viaarxiv icon

TC-SfM: Robust Track-Community-Based Structure-from-Motion

Jun 13, 2022
Lei Wang, Linlin Ge, Shan Luo, Zihan Yan, Zhaopeng Cui, Jieqing Feng

Figure 1 for TC-SfM: Robust Track-Community-Based Structure-from-Motion
Figure 2 for TC-SfM: Robust Track-Community-Based Structure-from-Motion
Figure 3 for TC-SfM: Robust Track-Community-Based Structure-from-Motion
Figure 4 for TC-SfM: Robust Track-Community-Based Structure-from-Motion

Structure-from-Motion (SfM) aims to recover 3D scene structures and camera poses based on the correspondences between input images, and thus the ambiguity caused by duplicate structures (i.e., different structures with strong visual resemblance) always results in incorrect camera poses and 3D structures. To deal with the ambiguity, most existing studies resort to additional constraint information or implicit inference by analyzing two-view geometries or feature points. In this paper, we propose to exploit high-level information in the scene, i.e., the spatial contextual information of local regions, to guide the reconstruction. Specifically, a novel structure is proposed, namely, {\textit{track-community}}, in which each community consists of a group of tracks and represents a local segment in the scene. A community detection algorithm is used to partition the scene into several segments. Then, the potential ambiguous segments are detected by analyzing the neighborhood of tracks and corrected by checking the pose consistency. Finally, we perform partial reconstruction on each segment and align them with a novel bidirectional consistency cost function which considers both 3D-3D correspondences and pairwise relative camera poses. Experimental results demonstrate that our approach can robustly alleviate reconstruction failure resulting from visually indistinguishable structures and accurately merge the partial reconstructions.

Viaarxiv icon

Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors

Nov 22, 2021
Zihan Yan, Li Liu, Xin Li, William K. Cheung, Youmin Zhang, Qun Liu, Guoyin Wang

Figure 1 for Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors
Figure 2 for Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors
Figure 3 for Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors
Figure 4 for Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors

Social network alignment aims at aligning person identities across social networks. Embedding based models have been shown effective for the alignment where the structural proximity preserving objective is typically adopted for the model training. With the observation that ``overly-close'' user embeddings are unavoidable for such models causing alignment inaccuracy, we propose a novel learning framework which tries to enforce the resulting embeddings to be more widely apart among the users via the introduction of carefully implanted pseudo anchors. We further proposed a meta-learning algorithm to guide the updating of the pseudo anchor embeddings during the learning process. The proposed intervention via the use of pseudo anchors and meta-learning allows the learning framework to be applicable to a wide spectrum of network alignment methods. We have incorporated the proposed learning framework into several state-of-the-art models. Our experimental results demonstrate its efficacy where the methods with the pseudo anchors implanted can outperform their counterparts without pseudo anchors by a fairly large margin, especially when there only exist very few labeled anchors.

* IEEE Transactions on Knowledge and Data Engineering 2021  
Viaarxiv icon