Benefiting from the powerful expressive capability of graphs, graph-based approaches have achieved impressive performance in various biomedical applications. Most existing methods tend to define the adjacency matrix among samples manually based on meta-features, and then obtain the node embeddings for downstream tasks by Graph Representation Learning (GRL). However, it is not easy for these approaches to generalize to unseen samples. Meanwhile, the complex correlation between modalities is also ignored. As a result, these factors inevitably yield the inadequacy of providing valid information about the patient's condition for a reliable diagnosis. In this paper, we propose an end-to-end Multimodal Graph Learning framework (MMGL) for disease prediction. To effectively exploit the rich information across multi-modality associated with diseases, amodal-attentional multi-modal fusion is proposed to integrate the features of each modality by leveraging the correlation and complementarity between the modalities. Furthermore, instead of defining the adjacency matrix manually as existing methods, the latent graph structure can be captured through a novel way of adaptive graph learning. It could be jointly optimized with the prediction model, thus revealing the intrinsic connections among samples. Unlike the previous transductive methods, our model is also applicable to the scenario of inductive learning for those unseen data. An extensive group of experiments on two disease prediction problems is then carefully designed and presented, demonstrating that MMGL obtains more favorable performances. In addition, we also visualize and analyze the learned graph structure to provide more reliable decision support for doctors in real medical applications and inspiration for disease research.