Sign-Coded Exposure Sensing for Noise-Robust High-Speed Imaging

We present a novel Fourier camera, an in-hardware optical compression of high-speed frames employing pixel-level sign-coded exposure where pixel intensities temporally modulated as positive and negative exposure are combined to yield Hadamard coefficients. The orthogonality of Walsh functions ensures that the noise is not amplified during high-speed frame reconstruction, making it a much more attractive option for coded exposure systems aimed at very high frame rate operation. Frame reconstruction is carried out by a single-pass demosaicking of the spatially multiplexed Walsh functions in a lattice arrangement, significantly reducing the computational complexity. The simulation prototype confirms the improved robustness to noise compared to the binary-coded exposure patterns, such as one-hot encoding and pseudo-random encoding. Our hardware prototype demonstrated the reconstruction of 4kHz frames of a moving scene lit by ambient light only.

Time-Ordered Recent Event (TORE) Volumes for Event Cameras

Event cameras are an exciting, new sensor modality enabling high-speed imaging with extremely low-latency and wide dynamic range. Unfortunately, most machine learning architectures are not designed to directly handle sparse data, like that generated from event cameras. Many state-of-the-art algorithms for event cameras rely on interpolated event representations - obscuring crucial timing information, increasing the data volume, and limiting overall network performance. This paper details an event representation called Time-Ordered Recent Event (TORE) volumes. TORE volumes are designed to compactly store raw spike timing information with minimal information loss. This bio-inspired design is memory efficient, computationally fast, avoids time-blocking (i.e. fixed and predefined frame rates), and contains "local memory" from past data. The design is evaluated on a wide range of challenging tasks (e.g. event denoising, image reconstruction, classification, and human pose estimation) and is shown to dramatically improve state-of-the-art performance. TORE volumes are an easy-to-implement replacement for any algorithm currently utilizing event representations.

Event Probability Mask (EPM) and Event Denoising Convolutional Neural Network (EDnCNN) for Neuromorphic Cameras

This paper presents a novel method for labeling real-world neuromorphic camera sensor data by calculating the likelihood of generating an event at each pixel within a short time window, which we refer to as "event probability mask" or EPM. Its applications include (i) objective benchmarking of event denoising performance, (ii) training convolutional neural networks for noise removal called "event denoising convolutional neural network" (EDnCNN), and (iii) estimating internal neuromorphic camera parameters. We provide the first dataset (DVSNOISE20) of real-world labeled neuromorphic camera events for noise removal.

Event Probability Mask (EPM) and Event Denoising Convolutional NeuralNetwork (EDnCNN) for Neuromorphic Cameras

This paper presents a novel method for labeling real-world neuromorphic camera sensor data by calculating the likelihood of generating an event at each pixel within a short time window, which we refer to as "event probability mask" or EPM. Its applications include (i) objective benchmarking of event denoising performance, (ii) training convolutional neural networks for noise removal called "event denoising convolutional neural network" (EDnCNN), and (iii) estimating internal neuromorphic camera parameters. We provide the first dataset (DVSNOISE20) of real-world labeled neuromorphic camera events for noise removal.