To mitigate the inspector's workload and improve the quality of the product, computer vision-based anomaly detection (AD) techniques are gradually deployed in real-world industrial scenarios. Recent anomaly analysis benchmarks progress to generative models. The aim is to model the defect-free distribution so that anomalies can be classified as out-of-distribution samples. Nevertheless, there are two disturbing factors that need researchers and deployers to prioritize: (i) the simplistic prior latent distribution inducing limited expressive capability; (ii) the collapsed mutual-dependent features resulting in poor generalization. In this paper, we propose a novel Patch-wise Wasserstein AutoEncoder (P-WAE) architecture to alleviate those challenges. In particular, a patch-wise variational inference model coupled with solving the jigsaw puzzle is designed, which is a simple yet effective way to increase the expressiveness and complexity of the latent manifold. This alleviates the blurry reconstruction problem. In addition, the Hilbert-Schmidt Independence Criterion (HSIC) bottleneck is introduced to constrain the over-regularization representation. Comprehensive experiments, conducted on the MVTec AD dataset, demonstrate the superior performance of our propo