Roadmap on Deep Learning for Microscopy

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Convolutional Neural Networks for Real-Time Localization and Classification in Feedback Digital Microscopy

We present an adapted single-shot convolutional neural network (YOLOv2) for the real-time localization and classification of particles in optical microscopy. As compared to previous works, we focus on the real-time detection capabilities of the system to allow for manipulation of microscopic objects in large heterogeneous ensembles with the help of feedback control. The network is capable of localizing and classifying several hundreds of microscopic objects even at very low signal-to-noise ratios for images as large as 416x416 pixels with an inference time of about 10 ms. We demonstrate the real-time detection performance by manipulating active particles propelled by laser-induced self-thermophoresis. In order to make our framework readily available for others, we provide all scripts and source code. The network is implemented in Python/Keras using the TensorFlow backend. A C library supporting GPUs is provided for the real-time inference.