Traffic flows are set of packets transferring between a client and a server with the same set of source and destination IP and port numbers. Traffic classification is referred to as the task of categorizing traffic flows into application-aware classes such as chats, streaming, VoIP, etc. Classification can be used for several purposes including policy enforcement and control or QoS management. In this paper, we introduce a novel end-to-end traffic classification method to distinguish between traffic classes including VPN traffic. Classification of VPN traffic is not trivial using traditional classification approaches due to its encrypted nature. We utilize two well-known neural networks, namely multi-layer perceptron and recurrent neural network focused on two metrics: class scores and distance from the center of the classes. Such approaches combined extraction, selection, and classification functionality into a single end-to-end system to systematically learn the non-linear relationship between input and predicted performance. Therefore, we could distinguish VPN traffics from Non-VPN traffics by rejecting the unrelated features of the VPN class. Moreover, obtain the application of Non-VPN traffics at the same time. The approach is evaluated using the general traffic dataset ISCX VPN-nonVPN and the acquired real dataset. The results of the analysis demonstrate that our proposed model fulfills the realistic project's criterion for precision.