This study presents a novel, deep, fully convolutional architecture which is optimized for the task of EEG-based neonatal seizure detection. Architectures of different depths were designed and tested; varying network depth impacts convolutional receptive fields and the corresponding learned feature complexity. Two deep convolutional networks are compared with a shallow SVM-based neonatal seizure detector, which relies on the extraction of hand-crafted features. On a large clinical dataset, of over 800 hours of multichannel unedited EEG, containing 1389 seizure events, the deep 11-layer architecture significantly outperforms the shallower architectures, improving the AUC90 from 82.6% to 86.8%. Combining the end-to-end deep architecture with the feature-based shallow SVM further improves the AUC90 to 87.6%. The fusion of classifiers of different depths gives greatly improved performance and reduced variability, making the combined classifier more clinically reliable.