The quantification of uncertainty on fluid flow in porous media is often hampered by multi-scale heterogeneity and insufficient site characterization. Monte-Carlo simulation (MCS), which runs numerical simulations for a large number of realization of input parameters , becomes infeasible when simulation cost is expensive or the degree of uncertainty is large. Many deep-neural-network-based methods are developed in order to replace the numerical flow simulation, but previous studies focused only on generating several snapshots of outputs at the fixed time steps, and lack to reflect the time dependent property of simulation data. Recently, the convolutional long short term memory (ConvLSTM) is utilized to deal with time series image data. Here, we propose to combine SegNet with ConvLSTM layers for the surrogate modeling of numerical flow simulation. The results show that the proposed method improves the performance of SegNet based surrogate model remarkably when the output of the simulation is time series data.