Deep neural network models are becoming popular and have used in various tasks such as computer vision, speech recognition, and natural language processing. It is often the case that the training phase of a model is executed in one environment, while the inference phase is executed in another environment. This is because the optimization characteristics for each phase significantly differ. Therefore, it is critical to efficiently compile a trained model for inferencing on different environments. To represent neural network models, users often use Open Neural Network Exchange (ONNX) which is an open standard format for machine learning interoperability. We are developing a compiler for rewriting a model in ONNX into a standalone binary that is executable on different target hardwares such as x86 machines, IBM Power Systems, and IBM System Z. The compiler was written using Multi-level Intermediate Representation (MLIR), a modern compiler infrastructure. In particular, we introduce two internal representations: ONNX IR for representing ONNX operators, and Kernel IR for efficiently lowering ONNX operators into LLVM bitcode. In this paper, we will discuss the overall structure of our compiler and give some practical examples of converting ONNX operators and models. We also cover several issues related to endianness. Our framework is publicly available as an open source project under the ONNX project.