We present a detailed study on Variational Autoencoders (VAEs) for anomalous jet tagging. By taking in low-level jet constituents' information, and only training with background jets in an unsupervised manner, the VAE is able to encode important information for reconstructing jets, while learning an expressive posterior distribution in the latent space. When using VAE as an anomaly detector, we present two approaches to detect anomalies: directly comparing in the input space or, instead, working in the latent space. Different anomaly metrics were examined. Results of the tagging performance for different jet types and over a large kinematic range are shown. In order to facilitate general search approaches such as bump-hunt, mass-decorrelated VAEs based on distance correlation regularization are also examined. Confronted with the problem of mis-assigning lower likelihood to out-of-distributions samples, we explore one potential solution -- Outlier Exposure (OE). OE, in the context of jet tagging, is employed to facilitate two goals: increasing sensitivity of outlier detection and decorrelating jet mass. We observe excellent results from both aspects. Code implementation can be found in \href{https://github.com/taolicheng/VAE-Jet}{Github}.