It is always an attractive task to discover knowledge for various learning problems; however, this knowledge discovery and maintenance process usually suffers from noise, incompleteness or knowledge domain mismatch. Thus, robust knowledge discovery by removing the noisy features or samples, complementing incomplete data, and mitigating the distribution difference becomes the key. Along this line of research, low-rank modeling is widely-used to solve these challenges. This survey covers the topic of: (1) robust knowledge recovery, (2) robust knowledge transfer, (3) robust knowledge fusion, centered around several major applications. First of all, we deliver a unified formulation for robust knowledge discovery based on a given dataset. Second, we discuss robust knowledge transfer and fusion given multiple datasets with different knowledge flows, followed by practical challenges, model variations, and remarks. Finally, we highlight future research of robust knowledge discovery for incomplete, unbalance, large-scale data analysis. This would benefit AI community from literature review to future direction.