Multi-object tracking (MOT) has important applications in monitoring, logistics, and other fields. This paper develops a real-time multi-object tracking and prediction system in rugged environments. A 3D object detection algorithm based on Lidar-camera fusion is designed to detect the target objects. Based on the Hungarian algorithm, this paper designs a 3D multi-object tracking algorithm with an adaptive threshold to realize the stable matching and tracking of the objects. We combine Memory Augmented Neural Networks (MANN) and Kalman filter to achieve 3D trajectory prediction on rugged terrains. Besides, we realize a new dynamic SLAM by using the results of multi-object tracking to remove dynamic points for better SLAM performance and static map. To verify the effectiveness of the proposed multi-object tracking and prediction system, several simulations and physical experiments are conducted. The results show that the proposed system can track dynamic objects and provide future trajectory and a more clean static map in real-time.