CaGNetv2: From Pixel to Patch (accpeted by TNNLS)

Code for "From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentation".

Created by Zhangxuan Gu, Siyuan Zhou, Li Niu*, Zihan Zhao, Liqing Zhang*.

Paper Link: [arXiv]

Note

This work is an extension of our previous CaGNet [arXiv, github].

Visualization on Pascal-VOC

Introduction

Zero-shot learning has been actively studied for image classification task to relieve the burden of annotating image labels. Interestingly, semantic segmentation task requires more labor-intensive pixel-wise annotation, but zero-shot semantic segmentation has only attracted limited research interest. Thus, we focus on zero-shot semantic segmentation, which aims to segment unseen objects with only category-level semantic representations provided for unseen categories. In this paper, we propose a novel Context-aware feature Generation Network (CaGNetv2), which can synthesize context-aware pixel-wise visual features for unseen categories based on category-level semantic representations and pixel-wise contextual information. The synthesized features are used to finetune the classifier to enable segmenting unseen objects. Furthermore, we extend pixel-wise feature generation and finetuning to patch-wise feature generation and finetuning, which additionally considers inter-pixel relationship. Experimental results on Pascal-VOC, Pascal-Context, and COCO-stuff show that our method significantly outperforms the existing zero-shot semantic segmentation methods.

Experimental Results

We compare our CaGNetv2 with SPNet [github, paper] and ZS3Net [github, paper].

“ST” in the following tables stands for self-training mentioned in ZS3Net.

Our Results on Pascal-VOC dataset

Our Results on COCO-Stuff dataset

Our Results on Pascal-Context dataset

Please note that our reproduced results of SPNet on Pascal-VOC dataset are obtained using their released model and code with careful tuning, but still lower than their reported results.

Code

COMING SOON !