12 lines
2.8 KiB
TeX
12 lines
2.8 KiB
TeX
\section{Related Work}
|
|
\label{sec:relw}
|
|
|
|
\noindent\textbf{Prompt learning for vision-language models.}
|
|
CoOp~\cite{zhou2022learning} first replaces hand-crafted templates with learnable continuous context vectors for CLIP. CoCoOp~\cite{zhou2022conditional} conditions the prompt on each input image to improve generalization to unseen classes. ProGrad~\cite{zhu2023prompt} restricts prompt updates to directions consistent with the zero-shot CLIP gradient, and KgCoOp~\cite{yao2023visual} regularizes learnable prompts toward the ``a photo of a \{class\}'' embeddings to mitigate forgetting. Beyond the text side, VPT~\cite{jia2022visual} inserts learnable tokens into vision transformers, and MaPLe~\cite{khattak2023maple} jointly prompts both encoders across depth by mapping deep text prompts to deep visual prompts. AAPL~\cite{kim2024aapl} disentangles attribute-related tokens from augmentation noise. Our prior MSGCoOp~\cite{wang2025msgcoop} ensembles $N$ parallel context vectors with LLM-generated semantic anchors and a diversity regularizer. These methods either fuse branch outputs with fixed equal weights or regularize branches only implicitly, and none exploits CLIP's zero-shot prediction itself to decide \emph{per sample} how much each branch should contribute. DZGCoOp differs in using the frozen encoders to produce an instance-level confidence that routes between an explicitly conservative branch and an explicitly exploratory one, while inheriting the deep cross-modal prompting of MaPLe.
|
|
|
|
\noindent\textbf{Knowledge preservation with zero-shot priors.}
|
|
Preserving pre-trained knowledge during few-shot adaptation is a central concern of prompt learning. KgCoOp~\cite{yao2023visual} and ProGrad~\cite{zhu2023prompt} anchor optimization to hand-crafted template embeddings; MSGCoOp~\cite{wang2025msgcoop} enriches such anchors with LLM-generated class descriptions~\cite{brown2020language,achiam2023gpt}. Distillation-style approaches additionally constrain output distributions. DZGCoOp extends this line along two axes: the zero-shot prior is used not only as a feature-space anchor but also as a distribution-level teacher via a KL guidance loss, and the zero-shot confidence actively participates in inference-time branch fusion rather than only in training-time regularization.
|
|
|
|
\noindent\textbf{Weight averaging.}
|
|
Stochastic Weight Averaging (SWA)~\cite{izmailov2018averaging} averages checkpoints sampled in a late phase of training and finds flatter minima. Prompt learning, however, exhibits strong epoch-dependent behavior: models trained longer overfit the source classes and transfer worse~\cite{wang2025msgcoop}. Our epoch-weighted averaging (EWA) generalizes uniform averaging with a Gaussian weighting schedule whose center and width select the favorable phase per transfer regime, subsuming ``early-checkpoint'' selection without extra inference cost.
|