Deep neural network (DNN) has been increasingly applied to microscopic demand analysis. While DNN often outperforms traditional multinomial logit (MNL) model, it is unclear whether we can obtain interpretable economic information from DNN-based choice model beyond prediction accuracy. This paper provides an empirical method of numerically extracting valuable economic information such as choice probability, probability derivatives (or elasticities), and marginal rates of substitution. Using a survey collected in Singapore, we find that when the economic information is aggregated over population or models, DNN models can reveal roughly S-shaped choice probability curves, inverse bell-shaped driving probability derivatives regarding costs and time, and reasonable median value of time (VOT). However at the disaggregate level, choice probability curves of DNN models can be non-monotonically decreasing with costs and highly sensitive to the particular estimation; derivatives of choice probabilities regarding costs and time can be positive at some region; VOT can be infinite, undefined, zero, or arbitrarily large. Some of these patterns can be seen as counter-intuitive, while others can potentially be regarded as advantages of DNN for its flexibility to reflect certain behavior peculiarities. These patterns broadly relate to two theoretical challenges of DNN, irregularity of its probability space and large estimation errors. Overall, this study provides a practical guidance of using DNN for demand analysis with two suggestions: First, researchers can use numerical methods to obtain behaviorally intuitive choice probabilities, probability derivatives, and reasonable VOT. Second, given the large estimation errors and irregularity of the probability space of DNN, researchers should always ensemble either over population or individual models to obtain stable economic information.