DermSynth3D: Synthesis of in-the-wild Annotated Dermatology Images

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermoscopy images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images

Despite continued advancement in recent years, deep neural networks still rely on large amounts of training data to avoid overfitting. However, labeled training data for real-world applications such as healthcare is limited and difficult to access given longstanding privacy, and strict data sharing policies. By manipulating image datasets in the pixel or feature space, existing data augmentation techniques represent one of the effective ways to improve the quantity and diversity of training data. Here, we look to advance augmentation techniques by building upon the emerging success of text-to-image diffusion probabilistic models in augmenting the training samples of our macroscopic skin disease dataset. We do so by enabling fine-grained control of the image generation process via input text prompts. We demonstrate that this generative data augmentation approach successfully maintains a similar classification accuracy of the visual classifier even when trained on a fully synthetic skin disease dataset. Similar to recent applications of generative models, our study suggests that diffusion models are indeed effective in generating high-quality skin images that do not sacrifice the classifier performance, and can improve the augmentation of training datasets after curation.

Detection and Longitudinal Tracking of Pigmented Skin Lesions in 3D Total-Body Skin Textured Meshes

We present an automated approach to detect and longitudinally track skin lesions on 3D total-body skin surfaces scans. The acquired 3D mesh of the subject is unwrapped to a 2D texture image, where a trained region convolutional neural network (R-CNN) localizes the lesions within the 2D domain. These detected skin lesions are mapped back to the 3D surface of the subject and, for subjects imaged multiple times, the anatomical correspondences among pairs of meshes and the geodesic distances among lesions are leveraged in our longitudinal lesion tracking algorithm. We evaluated the proposed approach using three sources of data. Firstly, we augmented the 3D meshes of human subjects from the public FAUST dataset with a variety of poses, textures, and images of lesions. Secondly, using a handheld structured light 3D scanner, we imaged a mannequin with multiple synthetic skin lesions at selected location and with varying shapes, sizes, and colours. Finally, we used 3DBodyTex, a publicly available dataset composed of 3D scans imaging the colored (textured) skin of 200 human subjects. We manually annotated locations that appeared to the human eye to contain a pigmented skin lesion as well as tracked a subset of lesions occurring on the same subject imaged in different poses. Our results, on test subjects annotated by three human annotators, suggest that the trained R-CNN detects lesions at a similar performance level as the human annotators. Our lesion tracking algorithm achieves an average accuracy of 80% when identifying corresponding pairs of lesions across subjects imaged in different poses. As there currently is no other large-scale publicly available dataset of 3D total-body skin lesions, we publicly release the 10 mannequin meshes and over 25,000 3DBodyTex manual annotations, which we hope will further research on total-body skin lesion analysis.

Visual Diagnosis of Dermatological Disorders: Human and Machine Performance

Skin conditions are a global health concern, ranking the fourth highest cause of nonfatal disease burden when measured as years lost due to disability. As diagnosing, or classifying, skin diseases can help determine effective treatment, dermatologists have extensively researched how to diagnose conditions from a patient's history and the lesion's visual appearance. Computer vision researchers are attempting to encode this diagnostic ability into machines, and several recent studies report machine level performance comparable with dermatologists. This report reviews machine approaches to classify skin images and consider their performance when compared to human dermatologists. Following an overview of common image modalities, dermatologists' diagnostic approaches and common tasks, and publicly available datasets, we discuss approaches to machine skin lesion classification. We then review works that directly compare human and machine performance. Finally, this report addresses the limitations and sources of errors in image-based skin disease diagnosis, applicable to both machines and dermatologists in a teledermatology setting.

Fully Convolutional Networks to Detect Clinical Dermoscopic Features

We use a pretrained fully convolutional neural network to detect clinical dermoscopic features from dermoscopy skin lesion images. We reformulate the superpixel classification task as an image segmentation problem, and extend a neural network architecture originally designed for image classification to detect dermoscopic features. Specifically, we interpolate the feature maps from several layers in the network to match the size of the input, concatenate the resized feature maps, and train the network to minimize a smoothed negative F1 score. Over the public validation leaderboard of the 2017 ISIC/ISBI Lesion Dermoscopic Feature Extraction Challenge, our approach achieves 89.3% AUROC, the highest averaged score when compared to the other two entries. Results over the private test leaderboard are still to be announced.