Building's energy consumption prediction is a major concern in the recent years and many efforts have been achieved in order to improve the energy management of buildings. In particular, the prediction of energy consumption in building is essential for the energy operator to build an optimal operating strategy, which could be integrated to building's energy management system (BEMS). This paper proposes a prediction model for building energy consumption using support vector machine (SVM). Data-driven model, for instance, SVM is very sensitive to the selection of training data. Thus the relevant days data selection method based on Dynamic Time Warping is used to train SVM model. In addition, to encompass thermal inertia of building, pseudo dynamic model is applied since it takes into account information of transition of energy consumption effects and occupancy profile. Relevant days data selection and whole training data model is applied to the case studies of Ecole des Mines de Nantes, France Office building. The results showed that support vector machine based on relevant data selection method is able to predict the energy consumption of building with a high accuracy in compare to whole data training. In addition, relevant data selection method is computationally cheaper (around 8 minute training time) in contrast to whole data training (around 31 hour for weekend and 116 hour for working days) and reveals realistic control implementation for online system as well.