As the human, we can recognize the places across a wide range of changing environmental conditions such as those caused by weathers, seasons, and day-night cycles. We can excavate and memorize the stable semantic structure of different places and scenes. For example, we can recognize tree whether the bare tree in winter or lush tree in summer. Therefore, the intrinsic features that are corresponding to specific semantic contents and condition-invariant of appearance changes can improve the performance of long-term place recognition significantly. In this paper, we propose a novel intrinsic encoder that excavates the condition-invariant latent space of different places under drastic appearance changes. Our method excavates the space of intrinsic structure features by self-supervised cycle loss designed based on Generative Adversarial Network (GAN). Different from previous learning based place recognition methods that need paired training data of each place with appearance changes, we employ the weakly-supervised strategy to utilize unpaired set-based training data of different environmental conditions. We conduct comprehensive experiments on the standard datasets and show that our semi-supervised intrinsic encoder achieves excellent performance for place recognition under drastic appearance changes.