Emotion estimation is an active field of research that has an important impact on the interaction between human and computer. Among the different modality to assess emotion, electroencephalogram (EEG) representing the electrical brain activity presented motivating results during the last decade. Emotion estimation from EEG could help in the diagnosis or rehabilitation of certain diseases. In this paper, we propose a dual method considering the physiological knowledge defined by specialists combined with novel deep learning (DL) models initially dedicated to computer vision. The joint learning has been enhanced with model saliency analysis. To present a global approach, the model has been evaluated on four publicly available datasets and achieves similar results to the state-of-theart approaches and outperforming results for two of the proposed datasets with a lower standard deviation that reflects higher stability. For sake of reproducibility, the codes and models proposed in this paper are available at github.com/VDelv/Emotion-EEG.