Image captioning models are usually trained according to human annotated ground-truth captions, which could generate accurate but generic captions. To improve the distinctiveness of captioning models, we firstly propose a series of metrics that use large-scale vision-language pre-training model CLIP to evaluate the distinctiveness of captions. Then we propose a simple and effective training strategy which trains the model by comparison within similar image groups. We conduct extensive experiments on various existing models to demonstrate the wide applicability of our strategy and the consistency of metric based results with human evaluation. By comparing the performance of our best model with existing state-of-the-art models, we claim that our model achieves new state-of-the-art towards distinctiveness objective.