Deep Learning and Handheld Augmented Reality Based System for Optimal Data Collection in Fault Diagnostics Domain

Compared to current AI or robotic systems, humans navigate their environment with ease, making tasks such as data collection trivial. However, humans find it harder to model complex relationships hidden in the data. AI systems, especially deep learning (DL) algorithms, impressively capture those complex relationships. Symbiotically coupling humans and computational machines' strengths can simultaneously minimize the collected data required and build complex input-to-output mapping models. This paper enables this coupling by presenting a novel human-machine interaction framework to perform fault diagnostics with minimal data. Collecting data for diagnosing faults for complex systems is difficult and time-consuming. Minimizing the required data will increase the practicability of data-driven models in diagnosing faults. The framework provides instructions to a human user to collect data that mitigates the difference between the data used to train and test the fault diagnostics model. The framework is composed of three components: (1) a reinforcement learning algorithm for data collection to develop a training dataset, (2) a deep learning algorithm for diagnosing faults, and (3) a handheld augmented reality application for data collection for testing data. The proposed framework has provided above 100\% precision and recall on a novel dataset with only one instance of each fault condition. Additionally, a usability study was conducted to gauge the user experience of the handheld augmented reality application, and all users were able to follow the provided steps.

Physics-Infused Fuzzy Generative Adversarial Network for Robust Failure Prognosis

Prognostics aid in the longevity of fielded systems or products. Quantifying the system's current health enable prognosis to enhance the operator's decision-making to preserve the system's health. Creating a prognosis for a system can be difficult due to (a) unknown physical relationships and/or (b) irregularities in data appearing well beyond the initiation of a problem. Traditionally, three different modeling paradigms have been used to develop a prognostics model: physics-based (PbM), data-driven (DDM), and hybrid modeling. Recently, the hybrid modeling approach that combines the strength of both PbM and DDM based approaches and alleviates their limitations is gaining traction in the prognostics domain. In this paper, a novel hybrid modeling approach for prognostics applications based on combining concepts from fuzzy logic and generative adversarial networks (GANs) is outlined. The FuzzyGAN based method embeds a physics-based model in the aggregation of the fuzzy implications. This technique constrains the output of the learning method to a realistic solution. Results on a bearing problem showcases the efficacy of adding a physics-based aggregation in a fuzzy logic model to improve GAN's ability to model health and give a more accurate system prognosis.

Semantic Map Injected GAN Training for Image-to-Image Translation

Image-to-image translation is the recent trend to transform images from one domain to another domain using generative adversarial network (GAN). The existing GAN models perform the training by only utilizing the input and output modalities of transformation. In this paper, we perform the semantic injected training of GAN models. Specifically, we train with original input and output modalities and inject a few epochs of training for translation from input to semantic map. Lets refer the original training as the training for the translation of input image into target domain. The injection of semantic training in the original training improves the generalization capability of the trained GAN model. Moreover, it also preserves the categorical information in a better way in the generated image. The semantic map is only utilized at the training time and is not required at the test time. The experiments are performed using state-of-the-art GAN models over CityScapes and RGB-NIR stereo datasets. We observe the improved performance in terms of the SSIM, FID and KID scores after injecting semantic training as compared to original training.

Fuzzy Generative Adversarial Networks

Generative Adversarial Networks (GANs) are well-known tools for data generation and semi-supervised classification. GANs, with less labeled data, outperform Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) in classification across various tasks, this shows promise for developing GANs capable of trespassing into the domain of semi-supervised regression. However, developing GANs for regression introduce two major challenges: (1) inherent instability in the GAN formulation and (2) performing regression and achieving stability simultaneously. This paper introduces techniques that show improvement in the GANs' regression capability through mean absolute error (MAE) and mean squared error (MSE). We bake a differentiable fuzzy logic system at multiple locations in a GAN because fuzzy logic systems have demonstrated high efficacy in classification and regression settings. The fuzzy logic takes the output of either or both the generator and the discriminator to either or both predict the output, $y$, and evaluate the generator's performance. We outline the results of applying the fuzzy logic system to CGAN and summarize each approach's efficacy. This paper shows that adding a fuzzy logic layer can enhance GAN's ability to perform regression; the most desirable injection location is problem-specific, and we show this through experiments over various datasets. Besides, we demonstrate empirically that the fuzzy-infused GAN is competitive with DNNs.

IH-GAN: A Conditional Generative Model for Implicit Surface-Based Inverse Design of Cellular Structures

Variable-density cellular structures can overcome connectivity and manufacturability issues of topologically-optimized, functionally graded structures, particularly when those structures are represented as discrete density maps. One na\"ive approach to creating variable-density cellular structures is simply replacing the discrete density map with an unselective type of unit cells having corresponding densities. However, doing so breaks the desired mechanical behavior, as equivalent density alone does not guarantee equivalent mechanical properties. Another approach uses homogenization methods to estimate each pre-defined unit cell's effective properties and remaps the unit cells following a scaling law. However, a scaling law merely mitigates the problem by performing an indirect and inaccurate mapping from the material property space to single-type unit cells. In contrast, we propose a deep generative model that resolves this problem by automatically learning an accurate mapping and generating diverse cellular unit cells conditioned on desired properties (i.e., Young's modulus and Poisson's ratio). We demonstrate our method via the use of implicit function-based unit cells and conditional generative adversarial networks. Results show that our method can 1) generate various unit cells that satisfy given material properties with high accuracy (relative error <5%), 2) create functionally graded cellular structures with high-quality interface connectivity (98.7% average overlap area at interfaces), and 3) improve the structural performance over the conventional topology-optimized variable-density structure (84.4% reduction in concentrated stress and extra 7% reduction in displacement).

Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters

This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal data is processed through a thresholding and Kalman filter approach to detect and track the bounding box. Training and testing data are prepared by combining a set of original experiments conducted in a motion capture environment and publicly available UAV image data. The new pipeline compares favorably to existing methods and demonstrates promising tracking and localization capacity of sample experiments.

Interpretable machine learning models: a physics-based view

To understand changes in physical systems and facilitate decisions, explaining how model predictions are made is crucial. We use model-based interpretability, where models of physical systems are constructed by composing basic constructs that explain locally how energy is exchanged and transformed. We use the port Hamiltonian (p-H) formalism to describe the basic constructs that contain physically interpretable processes commonly found in the behavior of physical systems. We describe how we can build models out of the p-H constructs and how we can train them. In addition we show how we can impose physical properties such as dissipativity that ensure numerical stability of the training process. We give examples on how to build and train models for describing the behavior of two physical systems: the inverted pendulum and swarm dynamics.

Hybrid modeling: Applications in real-time diagnosis

Reduced-order models that accurately abstract high fidelity models and enable faster simulation is vital for real-time, model-based diagnosis applications. In this paper, we outline a novel hybrid modeling approach that combines machine learning inspired models and physics-based models to generate reduced-order models from high fidelity models. We are using such models for real-time diagnosis applications. Specifically, we have developed machine learning inspired representations to generate reduced order component models that preserve, in part, the physical interpretation of the original high fidelity component models. To ensure the accuracy, scalability and numerical stability of the learning algorithms when training the reduced-order models we use optimization platforms featuring automatic differentiation. Training data is generated by simulating the high-fidelity model. We showcase our approach in the context of fault diagnosis of a rail switch system. Three new model abstractions whose complexities are two orders of magnitude smaller than the complexity of the high fidelity model, both in the number of equations and simulation time are shown. The numerical experiments and results demonstrate the efficacy of the proposed hybrid modeling approach.