Generative adversarial networks (GANs) have shown outstanding performance on a broad range of computer vision problems, but often require enormous training data and computational resources. Several works propose a transfer learning scheme to handle this issue, but they are prone to overfitting or too restrictive to learn the distribution shift. In this paper, we find that simply fine-tuning the networks while freezing the lower layers of the discriminator surprisingly works well. The simple baseline, freeze $D$, significantly outperforms the prior methods in both unconditional and conditional GANs, under StyleGAN and SNGAN-projection architectures and Animal Face, Anime Face, Oxford Flower, CUB-200-2011, and Caltech-256 datasets. Code and results are available in https://github.com/sangwoomo/freezeD.