55 lines
4.3 KiB
Markdown
55 lines
4.3 KiB
Markdown
# DZGCoOp: Visual-Language Prompt Tuning with Zero-shot Guided Dual-branch Context Optimization
|
|
|
|
[](https://mygitea.rainbus.top/Rain-Bus/DZGCoOp)
|
|
|
|
## Overview of DZGCoOp
|
|
|
|
We introduce **Dual-branch Zero-shot Guidance Context Optimization (DZGCoOp)**, a prompt tuning framework that exploits the zero-shot prior of the frozen CLIP model to guide dual-branch prompt learning for few-shot image classification. Instead of ensembling multiple equally-weighted prompts, DZGCoOp assigns each branch an explicit role and adapts their contribution per sample.
|
|
|
|
DZGCoOp is composed of three core designs:
|
|
|
|
1. **Zero-shot Guidance Module**: LLM-generated class descriptions are encoded by the frozen text encoder to build semantic anchors, which together with the frozen visual encoder yield zero-shot text/visual features, a zero-shot probability distribution, and an instance-level confidence $\alpha$ — stable prior signals and an adaptive fusion weight, all computed without gradient updates.
|
|
2. **Dual-branch Multimodal Learning Module**: a strong branch (handcrafted-template initialization, stronger semantic guidance) preserves consistency with pretraining knowledge, while a weak branch (random initialization, weaker guidance) explores task-specific representations. Both text branches share deep text prompts and reuse a single prompted visual branch, and their logits are fused adaptively by the instance-level confidence: $\mathbf{l}^{\mathrm{fin}} = \alpha \cdot \mathbf{l}^{\mathrm{str}} + (1-\alpha) \cdot \mathbf{l}^{\mathrm{weak}}$.
|
|
3. **Guidance Losses and Optimization Strategy**: a zero-shot feature guidance loss aligns learnable visual/text features with the zero-shot counterparts, a zero-shot probability guidance loss (KL divergence) constrains the predicted distribution to the zero-shot prior, and an Epoch Weighted Averaging (EWA) strategy fuses per-epoch parameter snapshots with Gaussian weights for stable generalization.
|
|
|
|
A schematic overview of the DZGCoOp architecture can be found below:
|
|
|
|
|
|
|
|
## Main Contributions
|
|
|
|
- **Zero-shot Guidance**: the frozen CLIP prior is distilled into semantic anchors, zero-shot features, a probability prior and an instance-level confidence that drives adaptive fusion.
|
|
- **Asymmetric Dual-branch Learning**: complementary strong/weak branches with deep prompting replace equal-weight multi-prompt ensembling.
|
|
- **Confidence-based Dynamic Fusion**: per-sample adaptive weighting balances pretrained-knowledge consistency and task-specific exploration.
|
|
- **Extensive Empirical Validation**: thorough experiments on base-to-novel, cross-domain, and cross-dataset generalization benchmarks.
|
|
|
|
## Results
|
|
|
|
DZGCoOp is evaluated on 11 benchmark datasets:
|
|
|
|
- **Base-to-novel generalization**: average base accuracy **84.52%**, novel accuracy **76.54%**, and harmonic mean **80.33%**, surpassing MSGCoOp by **+2.23%** harmonic mean and MaPLe by **+1.78%**.
|
|
- **Cross-domain generalization**: average accuracy **60.66%** over four ImageNet variants, outperforming MSGCoOp by **+0.25%** and MaPLe by **+0.39%**.
|
|
- **Cross-dataset generalization**: average accuracy **65.99%** across 10 transfer datasets, a **+1.82%** gain over MSGCoOp.
|
|
- **Ablation studies** show the zero-shot guidance losses bring **+4.48%** novel-class accuracy over the IVLP baseline, and the dual-branch architecture with confidence fusion further improves the harmonic mean.
|
|
|
|
## Getting Started
|
|
|
|
1. [Dassl.ProGrad.pytorch](Dassl.ProGrad.pytorch/) is the modified toolbox of [Dassl.pytorch](https://github.com/KaiyangZhou/Dassl.pytorch).
|
|
2. [DZGCoOp](DZGCoOp/). To get the results in our paper, follow the [README.md](DZGCoOp/README.md) under [DZGCoOp/](DZGCoOp/) to set the environment.
|
|
|
|
## Citation
|
|
|
|
If you use our work, please cite:
|
|
```bibtex
|
|
@inproceedings{dzgcoop26,
|
|
title={Visual-Language Prompt Tuning with Zero-shot Guided Dual-branch Context Optimization},
|
|
author={Zhaolong Wang},
|
|
booktitle={arxiv},
|
|
year={2026}
|
|
}
|
|
```
|
|
|
|
|
|
## Acknowledgements
|
|
Our code is based on [Co-CoOp and CoOp](https://github.com/KaiyangZhou/CoOp), [MaPLe](https://github.com/muzairkhattak/multimodal-prompt-learning), [KgCoOp](https://github.com/htyao89/KgCoOp) repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.
|