Accurate and reliable state of charge (SoC) estimation becomes increasingly important to provide a stable and efficient environment for Lithium-ion batteries (LiBs) powered devices. Most data-driven SoC models are built for a fixed ambient temperature, which neglect the high sensitivity of LiBs to temperature and may cause severe prediction errors. Nevertheless, a systematic evaluation of the impact of temperature on SoC estimation and ways for a prompt adjustment of the estimation model to new temperatures using limited data have been hardly discussed. To solve these challenges, a novel SoC estimation method is proposed by exploiting temporal dynamics of measurements and transferring consistent estimation ability among different temperatures. First, temporal dynamics, which is presented by correlations between the past fluctuation and the future motion, is extracted using canonical variate analysis. Next, two models, including a reference SoC estimation model and an estimation ability monitoring model, are developed with temporal dynamics. The monitoring model provides a path to quantitatively evaluate the influences of temperature on SoC estimation ability. After that, once the inability of the reference SoC estimation model is detected, consistent temporal dynamics between temperatures are selected for transfer learning. Finally, the efficacy of the proposed method is verified through a benchmark. Our proposed method not only reduces prediction errors at fixed temperatures (e.g., reduced by 24.35% at -20{\deg}C, 49.82% at 25{\deg}C) but also improves prediction accuracies at new temperatures.