Alert button
Picture for Zixing Song

Zixing Song

Alert button

WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering

May 08, 2023
Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, Irwin King

Figure 1 for WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering
Figure 2 for WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering
Figure 3 for WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering

Maximizing the user-item engagement based on vectorized embeddings is a standard procedure of recent recommender models. Despite the superior performance for item recommendations, these methods however implicitly deprioritize the modeling of user-wise similarity in the embedding space; consequently, identifying similar users is underperforming, and additional processing schemes are usually required otherwise. To avoid thorough model re-training, we propose WSFE, a model-agnostic and training-free representation encoder, to be flexibly employed on the fly for effective user segmentation. Underpinned by the optimal transport theory, the encoded representations from WSFE present a matched user-wise similarity/distance measurement between the realistic and embedding space. We incorporate WSFE into six state-of-the-art recommender models and conduct extensive experiments on six real-world datasets. The empirical analyses well demonstrate the superiority and generality of WSFE to fuel multiple downstream tasks with diverse underlying targets in recommendation.

Viaarxiv icon

Spectral Feature Augmentation for Graph Contrastive Learning and Beyond

Dec 02, 2022
Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, Irwin King

Figure 1 for Spectral Feature Augmentation for Graph Contrastive Learning and Beyond
Figure 2 for Spectral Feature Augmentation for Graph Contrastive Learning and Beyond
Figure 3 for Spectral Feature Augmentation for Graph Contrastive Learning and Beyond
Figure 4 for Spectral Feature Augmentation for Graph Contrastive Learning and Beyond

Although augmentations (e.g., perturbation of graph edges, image crops) boost the efficiency of Contrastive Learning (CL), feature level augmentation is another plausible, complementary yet not well researched strategy. Thus, we present a novel spectral feature argumentation for contrastive learning on graphs (and images). To this end, for each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power iteration regime which enjoys two valuable byproducts (under mere one or two iterations): (i) it partially balances spectrum of the feature map, and (ii) it injects the noise into rebalanced singular values of the feature map (spectral augmentation). For two views, we align these rebalanced feature maps as such an improved alignment step can focus more on less dominant singular values of matrices of both views, whereas the spectral augmentation does not affect the spectral angle alignment (singular vectors are not perturbed). We derive the analytical form for: (i) the incomplete power iteration to capture its spectrum-balancing effect, and (ii) the variance of singular values augmented implicitly by the noise. We also show that the spectral augmentation improves the generalization bound. Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines, and is complementary with other augmentation strategies and compatible with various contrastive losses.

* This paper has been published with the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI 2023) 
Viaarxiv icon

ConcreteGraph: A Data Augmentation Method Leveraging the Properties of Concept Relatedness Estimation

Jun 25, 2022
Yueen Ma, Zixing Song, Chirui Chang, Yue Yu, Irwin King

Figure 1 for ConcreteGraph: A Data Augmentation Method Leveraging the Properties of Concept Relatedness Estimation
Figure 2 for ConcreteGraph: A Data Augmentation Method Leveraging the Properties of Concept Relatedness Estimation
Figure 3 for ConcreteGraph: A Data Augmentation Method Leveraging the Properties of Concept Relatedness Estimation
Figure 4 for ConcreteGraph: A Data Augmentation Method Leveraging the Properties of Concept Relatedness Estimation

The concept relatedness estimation (CRE) task is to determine whether two given concepts are related. Although existing methods for the semantic textual similarity (STS) task can be easily adapted to this task, the CRE task has some unique properties that can be leveraged to augment the datasets for addressing its data scarcity problem. In this paper, we construct a graph named ConcreteGraph (Concept relatedness estimation Graph) to take advantage of the CRE properties. For the sampled new concept pairs from the ConcreteGraph, we add an additional step of filtering out the new concept pairs with low quality based on simple yet effective quality thresholding. We apply the ConcreteGraph data augmentation on three Transformer-based models to show its efficacy. Detailed ablation study for quality thresholding further shows that even a limited amount of high-quality data is more beneficial than a large quantity of unthresholded data. This paper is the first one to work on the WORD dataset and the proposed ConcreteGraph can boost the accuracy of the Transformers by more than 2%. All three Transformers, with the help of ConcreteGraph, can outperform the current state-of-theart method, Concept Interaction Graph (CIG), on the CNSE and CNSS datasets.

* 8 pages, acl format 
Viaarxiv icon

ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization

Jun 16, 2022
Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, Jieping Ye

Figure 1 for ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization
Figure 2 for ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization
Figure 3 for ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization
Figure 4 for ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization

Graph Neural Networks (GNNs) have attracted much attention due to their ability in learning representations from graph-structured data. Despite the successful applications of GNNs in many domains, the optimization of GNNs is less well studied, and the performance on node classification heavily suffers from the long-tailed node degree distribution. This paper focuses on improving the performance of GNNs via normalization. In detail, by studying the long-tailed distribution of node degrees in the graph, we propose a novel normalization method for GNNs, which is termed ResNorm (\textbf{Res}haping the long-tailed distribution into a normal-like distribution via \textbf{norm}alization). The $scale$ operation of ResNorm reshapes the node-wise standard deviation (NStd) distribution so as to improve the accuracy of tail nodes (\textit{i}.\textit{e}., low-degree nodes). We provide a theoretical interpretation and empirical evidence for understanding the mechanism of the above $scale$. In addition to the long-tailed distribution issue, over-smoothing is also a fundamental issue plaguing the community. To this end, we analyze the behavior of the standard shift and prove that the standard shift serves as a preconditioner on the weight matrix, increasing the risk of over-smoothing. With the over-smoothing issue in mind, we design a $shift$ operation for ResNorm that simulates the degree-specific parameter strategy in a low-cost manner. Extensive experiments have validated the effectiveness of ResNorm on several node classification benchmark datasets.

Viaarxiv icon

COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning

Jun 13, 2022
Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, Irwin King

Figure 1 for COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
Figure 2 for COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
Figure 3 for COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
Figure 4 for COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning

Graph contrastive learning (GCL) improves graph representation learning, leading to SOTA on various downstream tasks. The graph augmentation step is a vital but scarcely studied step of GCL. In this paper, we show that the node embedding obtained via the graph augmentations is highly biased, somewhat limiting contrastive models from learning discriminative features for downstream tasks. Thus, instead of investigating graph augmentation in the input space, we alternatively propose to perform augmentations on the hidden features (feature augmentation). Inspired by so-called matrix sketching, we propose COSTA, a novel COvariance-preServing feaTure space Augmentation framework for GCL, which generates augmented features by maintaining a "good sketch" of original features. To highlight the superiority of feature augmentation with COSTA, we investigate a single-view setting (in addition to multi-view one) which conserves memory and computations. We show that the feature augmentation with COSTA achieves comparable/better results than graph augmentation based models.

* This paper is accepted by the ACM KDD 2022 
Viaarxiv icon

A Survey on Deep Semi-supervised Learning

Feb 28, 2021
Xiangli Yang, Zixing Song, Irwin King, Zenglin Xu

Figure 1 for A Survey on Deep Semi-supervised Learning
Figure 2 for A Survey on Deep Semi-supervised Learning
Figure 3 for A Survey on Deep Semi-supervised Learning
Figure 4 for A Survey on Deep Semi-supervised Learning

Deep semi-supervised learning is a fast-growing field with a range of practical applications. This paper provides a comprehensive survey on both fundamentals and recent advances in deep semi-supervised learning methods from model design perspectives and unsupervised loss functions. We first present a taxonomy for deep semi-supervised learning that categorizes existing methods, including deep generative methods, consistency regularization methods, graph-based methods, pseudo-labeling methods, and hybrid methods. Then we offer a detailed comparison of these methods in terms of the type of losses, contributions, and architecture differences. In addition to the past few years' progress, we further discuss some shortcomings of existing methods and provide some tentative heuristic solutions for solving these open problems.

* 24 pages, 6 figures 
Viaarxiv icon

Graph-based Semi-supervised Learning: A Comprehensive Review

Feb 26, 2021
Zixing Song, Xiangli Yang, Zenglin Xu, Irwin King

Figure 1 for Graph-based Semi-supervised Learning: A Comprehensive Review
Figure 2 for Graph-based Semi-supervised Learning: A Comprehensive Review
Figure 3 for Graph-based Semi-supervised Learning: A Comprehensive Review
Figure 4 for Graph-based Semi-supervised Learning: A Comprehensive Review

Semi-supervised learning (SSL) has tremendous value in practice due to its ability to utilize both labeled data and unlabelled data. An important class of SSL methods is to naturally represent data as graphs such that the label information of unlabelled samples can be inferred from the graphs, which corresponds to graph-based semi-supervised learning (GSSL) methods. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large scale data. Focusing on this class of methods, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. This makes our paper distinct from recent surveys that cover an overall picture of SSL methods while neglecting fundamental understanding of GSSL methods. In particular, a major contribution of this paper lies in a new generalized taxonomy for GSSL, including graph regularization and graph embedding methods, with the most up-to-date references and useful resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with insights into this rapidly growing field.

Viaarxiv icon