In this paper we propose and study an optimization problem over a matrix group orbit that we call \emph{Group Orbit Optimization} (GOO). We prove that GOO can be used to induce matrix decomposition techniques such as singular value decomposition (SVD), LU decomposition, QR decomposition, Schur decomposition and Cholesky decomposition, etc. This gives rise to a unified framework for matrix decomposition and allows us to bridge these matrix decomposition methods. Moreover, we generalize GOO for tensor decomposition. As a concrete application of GOO, we devise a new data decomposition method over a special linear group to normalize point cloud data. Experiment results show that our normalization method is able to obtain recovery well from distortions like shearing, rotation and squeezing.