Multi-aspect user preferences are attracting wider attention in recommender systems, as they enable more detailed understanding of users' evaluations of items. Previous studies show that incorporating multi-aspect preferences can greatly improve the performance and explainability of recommendation. However, as recommendation is essentially a ranking problem, there is no principled solution for ranking multiple aspects collectively to enhance the recommendation. In this work, we derive a multi-aspect ranking criterion. To maintain the dependency among different aspects, we propose to use a vectorized representation of multi-aspect ratings and develop a probabilistic multivariate tensor factorization framework (PMTF). The framework naturally leads to a probabilistic multi-aspect ranking criterion, which generalizes the single-aspect ranking to a multivariate fashion. Experiment results on a large multi-aspect review rating dataset confirmed the effectiveness of our solution.