Human emotion recognition is an important direction in the field of biometric and information forensics. However, most existing human emotion research are based on the single RGB view. In this paper, we introduce a RGBD video-emotion dataset and a RGBD face-emotion dataset for research. To our best knowledge, this may be the first RGBD video-emotion dataset. We propose a new supervised nonlinear multi-view laplacian eigenmaps (MvLE) approach and a multihidden-layer out-of-sample network (MHON) for RGB-D humanemotion recognition. To get better representations of RGB view and depth view, MvLE is used to map the training set of both views from original space into the common subspace. As RGB view and depth view lie in different spaces, a new distance metric bag of neighbors (BON) used in MvLE can get the similar distributions of the two views. Finally, MHON is used to get the low-dimensional representations of test data and predict their labels. MvLE can deal with the cases that RGB view and depth view have different size of features, even different number of samples and classes. And our methods can be easily extended to more than two views. The experiment results indicate the effectiveness of our methods over some state-of-art methods.