LSTM language model is an essential component of industrial ASR systems. One important challenge of training an LSTM language model is how to scale out the learning process to leverage big data. Conventional approach such as block momentum provides a blockwise model update filtering (BMUF) process to stabilize the learning process, and achieves almost linear speedups with no degradation for speech recognition with DNNs and LSTMs. However, it needs to calculate the global average of all nodes and when the number of computing nodes is large, the communication latency is a big problem. For this reason, BMUF is not suitable under restricted network conditions. In this paper, we present a decentralized BMUF process, in which the model is split into different components, and each component is updated by communicating to some randomly chosen neighbor nodes with the same component, followed by a BMUF-like process. We apply this method to several LSTM language modeling tasks. Experimental results show that our approach achieves consistently better performance than the conventional BMUF. In particular, we obtain a lower perplexity than the single-GPU baseline on the wiki-text-103 benchmark using 4 GPUs. In addition, no performance degradation is incurred when scaling to 8 and 16 GPUs. Last but not least, our approach has a much simpler network topology than the centralized topology with a superior performance.