Alert button

CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets

Feb 07, 2023
Zachary Novack, Saurabh Garg, Julian McAuley, Zachary C. Lipton

Figure 1 for CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
Figure 2 for CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
Figure 3 for CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets
Figure 4 for CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets

Share this with someone who'll enjoy it:

Open vocabulary models (e.g. CLIP) have shown strong performance on zero-shot classification through their ability generate embeddings for each class based on their (natural language) names. Prior work has focused on improving the accuracy of these models through prompt engineering or by incorporating a small amount of labeled downstream data (via finetuning). However, there has been little focus on improving the richness of the class names themselves, which can pose issues when class labels are coarsely-defined and uninformative. We propose Classification with Hierarchical Label Sets (or CHiLS), an alternative strategy for zero-shot classification specifically designed for datasets with implicit semantic hierarchies. CHiLS proceeds in three steps: (i) for each class, produce a set of subclasses, using either existing label hierarchies or by querying GPT-3; (ii) perform the standard zero-shot CLIP procedure as though these subclasses were the labels of interest; (iii) map the predicted subclass back to its parent to produce the final prediction. Across numerous datasets with underlying hierarchical structure, CHiLS leads to improved accuracy in situations both with and without ground-truth hierarchical information. CHiLS is simple to implement within existing CLIP pipelines and requires no additional training cost. Code is available at: https://github.com/acmi-lab/CHILS.

View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it: