Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based Photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer 4 questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption color variance to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. Furthermore, we conclude that the availability of multiple convolutional kernels in the skin-region is necessary for the method to arrive at a flexible channel combination through the spatial operation, but does not provide the same advantages as a multi-site measurement with a knowledge based PPG extraction method. Finally, we show that a hybrid of knowledge based color-channel combination (pre-processing) and CNN is possible and enables an improved motion robustness.