Light Field Saliency Detection with Deep Convolutional Networks

CNN-based methods have been proven to work well for saliency detection on RGB images owing to the outstanding feature representation abilities of CNNs. However, their performance will degrade when detecting multiple saliency regions in highly cluttered or similar backgrounds. To address these problems, in this paper we resort to light field imaging, which records the color intensity of each pixel as well as the directions of incoming light rays, and thus can improve performance for saliency detection owing to the usage of both spatial and angular patterns encoded in light field images. However, it is not trivial to use CNN-based methods for saliency detection on light field images because these methods are not specifically designed for processing light field inputs and current light field datasets are not sufficiently large to train CNNs. To overcome these issues, we first present a new Lytro Illum dataset, which contains 640 light fields and their corresponding micro-lens images, central-viewing images as well as ground-truth saliency maps. Comparing to the current light field saliency datasets~\cite{Li14, Zhang17}, the new dataset is larger, of higher quality, contains more variations and more types of light field inputs, which is suitable for training deeper networks as well as better benchmarking algorithms. Furthermore, we propose a novel end-to-end CNN-based framework for light field saliency detection as well as its several variants. We systematically study the impact of different variants and compare light field saliency with regular 2D saliency on the performance of the proposed network. We also conduct extensive experimental comparisons, which indicate that our network significantly outperforms state-of-the-art methods on the proposed dataset and has desired generalization abilities on other existing datasets.