Force control is essential for medical robots when touching and contacting the patient's body. To increase the stability and efficiency in force control, an Adaption Module could be used to adjust the parameters for different contact situations. We propose an adaptive controller with an Adaption Module which can produce control parameters based on force feedback and real-time stiffness detection. We develop methods for learning the optimal policies by value iteration and using the data generated from those policies to train the Adaptive Module. We test this controller on different zones of a person's arm. All the parameters used in practice are learned from data. The experiments show that the proposed adaptive controller can exert various target forces on different zones of the arm with fast convergence and good stability.