Deeper neural networks are hard to train. Inspired by the elastic collision model in physics, we present a universal structure that could be integrated into the existing network structures to speed up the training process and eventually increase its generalization ability. We apply our structure to the Convolutional Neural Networks(CNNs) to form a new structure, which we term the "Inter-layer Collision" (IC) structure. The IC structure provides the deeper layer a better representation of the input features. We evaluate the IC structure on CIFAR10 and Imagenet by integrating it into the existing state-of-the-art CNNs. Our experiment shows that the proposed IC structure can effectively increase the accuracy and convergence speed.