A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks

Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. Research communities with a history of using DE models may view DNN-based differential equation solvers (DNN-DEs) as a faster and transferable alternative to current numerical methods. However, there is a lack of systematic surveys detailing the use of DNN-DE methods across physical application domains and a generalized taxonomy to guide future research. This paper surveys and classifies previous works and provides an educational tutorial for senior practitioners, professionals, and graduate students in engineering and computer science. First, we propose a taxonomy to navigate domains of DE systems studied under the umbrella of DNN-DE. Second, we examine the theory and performance of the Physics Informed Neural Network (PINN) to demonstrate how the influential DNN-DE architecture mathematically solves a system of equations. Third, to reinforce the key ideas of solving and discovery of DEs using DNN, we provide a tutorial using DeepXDE, a Python package for developing PINNs, to develop DNN-DEs for solving and discovering a classic DE, the linear transport equation.

Eco-PiNN: A Physics-informed Neural Network for Eco-toll Estimation

The eco-toll estimation problem quantifies the expected environmental cost (e.g., energy consumption, exhaust emissions) for a vehicle to travel along a path. This problem is important for societal applications such as eco-routing, which aims to find paths with the lowest exhaust emissions or energy need. The challenges of this problem are three-fold: (1) the dependence of a vehicle's eco-toll on its physical parameters; (2) the lack of access to data with eco-toll information; and (3) the influence of contextual information (i.e. the connections of adjacent segments in the path) on the eco-toll of road segments. Prior work on eco-toll estimation has mostly relied on pure data-driven approaches and has high estimation errors given the limited training data. To address these limitations, we propose a novel Eco-toll estimation Physics-informed Neural Network framework (Eco-PiNN) using three novel ideas, namely, (1) a physics-informed decoder that integrates the physical laws of the vehicle engine into the network, (2) an attention-based contextual information encoder, and (3) a physics-informed regularization to reduce overfitting. Experiments on real-world heavy-duty truck data show that the proposed method can greatly improve the accuracy of eco-toll estimation compared with state-of-the-art methods.

Spatiotemporal Data Mining: A Survey

Spatiotemporal data mining aims to discover interesting, useful but non-trivial patterns in big spatial and spatiotemporal data. They are used in various application domains such as public safety, ecology, epidemiology, earth science, etc. This problem is challenging because of the high societal cost of spurious patterns and exorbitant computational cost. Recent surveys of spatiotemporal data mining need update due to rapid growth. In addition, they did not adequately survey parallel techniques for spatiotemporal data mining. This paper provides a more up-to-date survey of spatiotemporal data mining methods. Furthermore, it has a detailed survey of parallel formulations of spatiotemporal data mining.

MC-DGCNN: A Novel DNN Architecture for Multi-Category Point Set Classification

Point set classification aims to build a representation learning model that distinguishes between spatial and categorical configurations of point set data. This problem is societally important since in many applications domains such as immunology, and microbial ecology. This problem is challenging since the interactions between different categories of points are not always equal; as a result, the representation learning model must selectively learn the most relevant multi-categorical relationships. The related works are limited (1) in learning the importance of different multi-categorical relationships, especially for high-order interactions, and (2) do not fully exploit the spatial distribution of points beyond simply measuring relative distance or applying a feed-forward neural network to coordinates. To overcome these limitations, we leverage the dynamic graph convolutional neural network (DGCNN) architecture to design a novel multi-category DGCNN (MC-DGCNN), contributing location representation and point pair attention layers for multi-categorical point set classification. MC-DGCNN has the ability to identify the categorical importance of each point pair and extends this to N-way spatial relationships, while still preserving all the properties and benefits of DGCNN (e.g., differentiability). Experimental results show that the proposed architecture is computationally efficient and significantly outperforms current deep learning architectures on real-world datasets.

Towards Comparative Physical Interpretation of Spatial Variability Aware Neural Networks: A Summary of Results

Given Spatial Variability Aware Neural Networks (SVANNs), the goal is to investigate mathematical (or computational) models for comparative physical interpretation towards their transparency (e.g., simulatibility, decomposability and algorithmic transparency). This problem is important due to important use-cases such as reusability, debugging, and explainability to a jury in a court of law. Challenges include a large number of model parameters, vacuous bounds on generalization performance of neural networks, risk of overfitting, sensitivity to noise, etc., which all detract from the ability to interpret the models. Related work on either model-specific or model-agnostic post-hoc interpretation is limited due to a lack of consideration of physical constraints (e.g., mass balance) and properties (e.g., second law of geography). This work investigates physical interpretation of SVANNs using novel comparative approaches based on geographically heterogeneous features. The proposed approach on feature-based physical interpretation is evaluated using a case-study on wetland mapping. The proposed physical interpretation improves the transparency of SVANN models and the analytical results highlight the trade-off between model transparency and model performance (e.g., F1-score). We also describe an interpretation based on geographically heterogeneous processes modeled as partial differential equations (PDEs).

Vehicle Emissions Prediction with Physics-Aware AI Models: Preliminary Results

Given an on-board diagnostics (OBD) dataset and a physics-based emissions prediction model, this paper aims to develop an accurate and computational-efficient AI (Artificial Intelligence) method that predicts vehicle emissions. The problem is of societal importance because vehicular emissions lead to climate change and impact human health. This problem is challenging because the OBD data does not contain enough parameters needed by high-order physics models. Conversely, related work has shown that low-order physics models have poor predictive accuracy when using available OBD data. This paper uses a divergent window co-occurrence pattern detection method to develop a spatiotemporal variability-aware AI model for predicting emission values from the OBD datasets. We conducted a case study using real-world OBD data from a local public transportation agency. Results show that the proposed AI method has approximately 65% improved predictive accuracy than a non-AI low-order physics model and is approximately 35% more accurate than a baseline model.

Statistically-Robust Clustering Techniques for Mapping Spatial Hotspots: A Survey

Mapping of spatial hotspots, i.e., regions with significantly higher rates or probability density of generating certain events (e.g., disease or crime cases), is a important task in diverse societal domains, including public health, public safety, transportation, agriculture, environmental science, etc. Clustering techniques required by these domains differ from traditional clustering methods due to the high economic and social costs of spurious results (e.g., false alarms of crime clusters). As a result, statistical rigor is needed explicitly to control the rate of spurious detections. To address this challenge, techniques for statistically-robust clustering have been extensively studied by the data mining and statistics communities. In this survey we present an up-to-date and detailed review of the models and algorithms developed by this field. We first present a general taxonomy of the clustering process with statistical rigor, covering key steps of data and statistical modeling, region enumeration and maximization, significance testing, and data update. We further discuss different paradigms and methods within each of key steps. Finally, we highlight research gaps and potential future directions, which may serve as a stepping stone in generating new ideas and thoughts in this growing field and beyond.

Towards Spatial Variability Aware Deep Neural Networks (SVANN): A Summary of Results

Spatial variability has been observed in many geo-phenomena including climatic zones, USDA plant hardiness zones, and terrestrial habitat types (e.g., forest, grasslands, wetlands, and deserts). However, current deep learning methods follow a spatial-one-size-fits-all(OSFA) approach to train single deep neural network models that do not account for spatial variability. In this work, we propose and investigate a spatial-variability aware deep neural network(SVANN) approach, where distinct deep neural network models are built for each geographic area. We evaluate this approach using aerial imagery from two geographic areas for the task of mapping urban gardens. The experimental results show that SVANN provides better performance than OSFA in terms of precision, recall,and F1-score to identify urban gardens.

Adversarial Attacks on Reinforcement Learning based Energy Management Systems of Extended Range Electric Delivery Vehicles

Adversarial examples are firstly investigated in the area of computer vision: by adding some carefully designed ''noise'' to the original input image, the perturbed image that cannot be distinguished from the original one by human, can fool a well-trained classifier easily. In recent years, researchers also demonstrated that adversarial examples can mislead deep reinforcement learning (DRL) agents on playing video games using image inputs with similar methods. However, although DRL has been more and more popular in the area of intelligent transportation systems, there is little research investigating the impacts of adversarial attacks on them, especially for algorithms that do not take images as inputs. In this work, we investigated several fast methods to generate adversarial examples to significantly degrade the performance of a well-trained DRL- based energy management system of an extended range electric delivery vehicle. The perturbed inputs are low-dimensional state representations and close to the original inputs quantified by different kinds of norms. Our work shows that, to apply DRL agents on real-world transportation systems, adversarial examples in the form of cyber-attack should be considered carefully, especially for applications that may lead to serious safety issues.

A Physics Model-Guided Online Bayesian Framework for Energy Management of Extended Range Electric Delivery Vehicles

Increasing the fuel economy of hybrid electric vehicles (HEVs) and extended range electric vehicles (EREVs) through optimization-based energy management strategies (EMS) has been an active research area in transportation. However, it is difficult to apply optimization-based EMS to current in-use EREVs because insufficient knowledge is known about future trips, and because such methods are computationally expensive for large-scale deployment. As a result, most past research has been validated on standard driving cycles or on recorded high-resolution data from past real driving cycles. This paper improves an in-use rule-based EMS that is used in a delivery vehicle fleet equipped with two-way vehicle-to-cloud connectivity. A physics model-guided online Bayesian framework is described and validated on large number of in-use driving samples of EREVs used for last-mile package delivery. The framework includes: a database, a preprocessing module, a vehicle model and an online Bayesian algorithm module. It uses historical 0.2 Hz resolution trip data as input and outputs an updated parameter to the engine control logic on the vehicle to reduce fuel consumption on the next trip. The key contribution of this work is a framework that provides an immediate solution for fuel use reduction of in-use EREVs. The framework was also demonstrated on real-world EREVs delivery vehicles operating on actual routes. The results show an average of 12.8% fuel use reduction among tested vehicles for 155 real delivery trips. The presented framework is extendable to other EREV applications including passenger vehicles, transit buses, and other vocational vehicles whose trips are similar day-to-day.