Self-supervised grasp learning, i.e., learning to grasp by trial and error, has made great progress. However, it is still time-consuming to train such a model and also a challenge to apply it in practice. This work presents an accelerating method of robotic grasp learning via pretraining with coarse affordance maps of objects to be grasped based on a quite small dataset. A model generated through pre-training is harnessed as an initialization policy to warmly start grasp learning so as to guide a robot to capture more effective rewards at the beginning of training. An object in its coarse affordance map is annotated with a single key point and thereby, the burden of labeling is greatly alleviated. Extensive experiments in simulation and on a real robot are conducted to evaluate the proposed method. The simulation results show that it can significantly accelerate grasp learning by nearly three times over a vanilla Deep Q-Network -based method. Its test on a real UR3 robot shows that it reaches a grasp success rate of 89.5% via only 500 times of grasp tries within about two hours, which is four times faster than its competitor. In addition, it enjoys an outstanding generalization ability to grasp prior-unseen novel objects. It outperforms some existing methods and has the potential to directly apply to a robot for real-world grasp learning tasks.