DZGCoOp: Visual-Language Prompt Tuning with Zero-shot Guided Dual-branch Context Optimization

Overview of DZGCoOp

We introduce Dual-branch Zero-shot Guidance Context Optimization (DZGCoOp), a prompt tuning framework that exploits the zero-shot prior of the frozen CLIP model to guide dual-branch prompt learning for few-shot image classification. Instead of ensembling multiple equally-weighted prompts, DZGCoOp assigns each branch an explicit role and adapts their contribution per sample.

DZGCoOp is composed of three core designs:

Zero-shot Guidance Module: LLM-generated class descriptions are encoded by the frozen text encoder to build semantic anchors, which together with the frozen visual encoder yield zero-shot text/visual features, a zero-shot probability distribution, and an instance-level confidence \alpha — stable prior signals and an adaptive fusion weight, all computed without gradient updates.
Dual-branch Multimodal Learning Module: a strong branch (handcrafted-template initialization, stronger semantic guidance) preserves consistency with pretraining knowledge, while a weak branch (random initialization, weaker guidance) explores task-specific representations. Both text branches share deep text prompts and reuse a single prompted visual branch, and their logits are fused adaptively by the instance-level confidence: \mathbf{l}^{\mathrm{fin}} = \alpha \cdot \mathbf{l}^{\mathrm{str}} + (1-\alpha) \cdot \mathbf{l}^{\mathrm{weak}}.
Guidance Losses and Optimization Strategy: a zero-shot feature guidance loss aligns learnable visual/text features with the zero-shot counterparts, a zero-shot probability guidance loss (KL divergence) constrains the predicted distribution to the zero-shot prior, and an Epoch Weighted Averaging (EWA) strategy fuses per-epoch parameter snapshots with Gaussian weights for stable generalization.

A schematic overview of the DZGCoOp architecture can be found below:

Main Contributions

Zero-shot Guidance: the frozen CLIP prior is distilled into semantic anchors, zero-shot features, a probability prior and an instance-level confidence that drives adaptive fusion.
Asymmetric Dual-branch Learning: complementary strong/weak branches with deep prompting replace equal-weight multi-prompt ensembling.
Confidence-based Dynamic Fusion: per-sample adaptive weighting balances pretrained-knowledge consistency and task-specific exploration.
Extensive Empirical Validation: thorough experiments on base-to-novel, cross-domain, and cross-dataset generalization benchmarks.

Results

DZGCoOp is evaluated on 11 benchmark datasets:

Base-to-novel generalization: average base accuracy 84.52%, novel accuracy 76.54%, and harmonic mean 80.33%, surpassing MSGCoOp by +2.23% harmonic mean and MaPLe by +1.78%.
Cross-domain generalization: average accuracy 60.66% over four ImageNet variants, outperforming MSGCoOp by +0.25% and MaPLe by +0.39%.
Cross-dataset generalization: average accuracy 65.99% across 10 transfer datasets, a +1.82% gain over MSGCoOp.
Ablation studies show the zero-shot guidance losses bring +4.48% novel-class accuracy over the IVLP baseline, and the dual-branch architecture with confidence fusion further improves the harmonic mean.

Getting Started

Dassl.ProGrad.pytorch is the modified toolbox of Dassl.pytorch.
DZGCoOp. To get the results in our paper, follow the README.md under DZGCoOp/ to set the environment.

Citation

If you use our work, please cite:

@inproceedings{dzgcoop26,
    title={Visual-Language Prompt Tuning with Zero-shot Guided Dual-branch Context Optimization},
    author={Zhaolong Wang},
    booktitle={arxiv},
    year={2026}
}

Acknowledgements

Our code is based on Co-CoOp and CoOp, MaPLe, KgCoOp repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.

4.3 KiB Raw Blame History