Location-aware Upsampling for Semantic Segmentation

Many successful learning targets such as minimizing dice loss and cross-entropy loss have enabled unprecedented breakthroughs in segmentation tasks. Beyond these semantic metrics, this paper aims to introduce location supervision into semantic segmentation. Based on this idea, we present a Location-aware Upsampling (LaU) that adaptively refines the interpolating coordinates with trainable offsets. Then, location-aware losses are established by encouraging pixels to move towards well-classified locations. An LaU is offset prediction coupled with interpolation, which is trained end-to-end to generate confidence score at each position from coarse to fine. Guided by location-aware losses, the new module can replace its plain counterpart (\textit{e.g.}, bilinear upsampling) in a plug-and-play manner to further boost the leading encoder-decoder approaches. Extensive experiments validate the consistent improvement over the state-of-the-art methods on benchmark datasets. Our code is available at https://github.com/HolmesShuan/Location-aware-Upsampling-for-Semantic-Segmentation

SpatialFlow: Bridging All Tasks for Panoptic Segmentation

The newly proposed panoptic segmentation task, which aims to encompass the tasks of instance segmentation (for things) and semantic segmentation (for stuff), is an essential step toward real-world vision systems and has attracted a lot of attention in the vision community. Recently, several works have been proposed for this task. Most of them focused on unifying two tasks by sharing the backbone but ignored to highlight the significance of fully interweaving features between tasks, such as providing the spatial context of objects to both semantic and instance segmentation. However, being well aware of locations of objects is fundamental to many vision tasks, e.g., object detection, instance segmentation, semantic segmentation. In this paper, we propose object spatial information flows, which can bridge all tasks together by delivering the spatial context from the box regression task to others. Based on these flows, we present a location-aware and unified framework for panoptic segmentation, {\em SpatialFlow}. The spatial information flows in {\em SpatialFlow} can provide clues for segmenting both things and stuff and help networks better understand the whole image. Moreover, instead of endowing Mask R-CNN with a stuff segmentation branch on a shared backbone, we design four parallel sub-networks for sub-tasks, which facilitate the feature integration among different tasks. We perform a detail ablation study on MS-COCO and Cityscapes panoptic benchmarks. Extensive experiments show that SpatialFlow achieves state-of-the-art results and can boost the performance of things and stuff segmentation at the same time.

A System-Level Solution for Low-Power Object Detection

Object detection has made impressive progress in recent years with the help of deep learning. However, state-of-the-art algorithms are both computation and memory intensive. Though many lightweight networks are developed for a trade-off between accuracy and efficiency, it is still a challenge to make it practical on an embedded device. In this paper, we present a system-level solution for efficient object detection on a heterogeneous embedded device. The detection network is quantized to low bits and allows efficient implementation with shift operators. In order to make the most of the benefits of low-bit quantization, we design a dedicated accelerator with programmable logic. Inside the accelerator, a hybrid dataflow is exploited according to the heterogeneous property of different convolutional layers. We adopt a straightforward but resource-friendly column-prior tiling strategy to map the computation-intensive convolutional layers to the accelerator that can support arbitrary feature size. Other operations can be performed on the low-power CPU cores, and the entire system is executed in a pipelined manner. As a case study, we evaluate our object detection system on a real-world surveillance video with input size of 512x512, and it turns out that the system can achieve an inference speed of 18 fps at the cost of 6.9W (with display) with an mAP of 66.4 verified on the PASCAL VOC 2012 dataset.

Compact Global Descriptor for Neural Networks

Long-range dependencies modeling, widely used in capturing spatiotemporal correlation, has shown to be effective in CNN dominated computer vision tasks. Yet neither stacks of convolutional operations to enlarge receptive fields nor recent nonlocal modules is computationally efficient. In this paper, we present a generic family of lightweight global descriptors for modeling the interactions between positions across different dimensions (e.g., channels, frames). This descriptor enables subsequent convolutions to access the informative global features with negligible computational complexity and parameters. Benchmark experiments show that the proposed method can complete state-of-the-art long-range mechanisms with a significant reduction in extra computing cost. Code available at https://github.com/HolmesShuan/Compact-Global-Descriptor.