A Heterogeneous Graph Based Framework for Multimodal Neuroimaging Fusion Learning

Here, we present a Heterogeneous Graph neural network for Multimodal neuroimaging fusion learning (HGM). Traditional GNN-based models usually assume the brain network is a homogeneous graph with single type of nodes and edges. However, vast literatures have shown the heterogeneity of the human brain especially between the two hemispheres. Homogeneous brain network is insufficient to model the complicated brain state. Therefore, in this work we firstly model the brain network as heterogeneous graph with multi-type nodes (i.e., left and right hemispheric nodes) and multi-type edges (i.e., intra- and inter-hemispheric edges). Besides, we also propose a self-supervised pre-training strategy based on heterogeneou brain network to address the overfitting problem due to the complex model and small sample size. Our results on two datasets show the superiority of proposed model over other multimodal methods for disease prediction task. Besides, ablation experiments show that our model with pre-training strategy can alleviate the problem of limited training sample size.

Deep Open Snake Tracker for Vessel Tracing

Vessel tracing by modeling vascular structures in 3D medical images with centerlines and radii can provide useful information for vascular health. Existing algorithms have been developed but there are certain persistent problems such as incomplete or inaccurate vessel tracing, especially in complicated vascular beds like the intracranial arteries. We propose here a deep learning based open curve active contour model (DOST) to trace vessels in 3D images. Initial curves were proposed from a centerline segmentation neural network. Then data-driven machine knowledge was used to predict the stretching direction and vessel radius of the initial curve, while the active contour model (as human knowledge) maintained smoothness and intensity fitness of curves. Finally, considering the nonloop topology of most vasculatures, individually traced vessels were connected into a tree topology by applying a minimum spanning tree algorithm on a global connection graph. We evaluated DOST on a Time-of-Flight (TOF) MRA intracranial artery dataset and demonstrated its superior performance over existing segmentation-based and tracking-based vessel tracing methods. In addition, DOST showed strong adaptability on different imaging modalities (CTA, MR T1 SPACE) and vascular beds (coronary arteries).