Alert button

Zero-shot Nuclei Detection via Visual-Language Pre-trained Models

Jun 30, 2023
Yongjian Wu, Yang Zhou, Jiya Saiyin, Bingzheng Wei, Maode Lai, Jianzhong Shou, Yubo Fan, Yan Xu

Figure 1 for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
Figure 2 for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
Figure 3 for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
Figure 4 for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models

Share this with someone who'll enjoy it:

Large-scale visual-language pre-trained models (VLPM) have proven their excellent performance in downstream object detection for natural scenes. However, zero-shot nuclei detection on H\&E images via VLPMs remains underexplored. The large gap between medical images and the web-originated text-image pairs used for pre-training makes it a challenging task. In this paper, we attempt to explore the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP) model, for zero-shot nuclei detection. Concretely, an automatic prompts design pipeline is devised based on the association binding trait of VLPM and the image-to-text VLPM BLIP, avoiding empirical manual prompts engineering. We further establish a self-training framework, using the automatically designed prompts to generate the preliminary results as pseudo labels from GLIP and refine the predicted boxes in an iterative manner. Our method achieves a remarkable performance for label-free nuclei detection, surpassing other comparison methods. Foremost, our work demonstrates that the VLPM pre-trained on natural image-text pairs exhibits astonishing potential for downstream tasks in the medical field as well. Code will be released at https://github.com/wuyongjianCODE/VLPMNuD.

* This article has been accepted by MICCAI 2023,but has not been fully edited. Content may change prior to final publication  
View paper onarxiv icon

Share this with someone who'll enjoy it: