Paper draft for the second work (thesis Ch.4 ZGDPE, renamed DZGCoOp): abstract, intro, related work, method (zero-shot guidance module, asymmetric dual-branch with deep prompts, confidence-guided dynamic fusion, guidance losses + EWA), experiments (base-to-novel, cross-domain, cross-dataset, ablations) and conclusion. Compiles to 8 pages with tectonic (no errors, no overfull boxes). Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
\section{Related Work}
|
||||
\label{sec:relw}
|
||||
|
||||
\noindent\textbf{Prompt learning for vision-language models.}
|
||||
CoOp~\cite{zhou2022learning} first replaces hand-crafted templates with learnable continuous context vectors for CLIP. CoCoOp~\cite{zhou2022conditional} conditions the prompt on each input image to improve generalization to unseen classes. ProGrad~\cite{zhu2023prompt} restricts prompt updates to directions consistent with the zero-shot CLIP gradient, and KgCoOp~\cite{yao2023visual} regularizes learnable prompts toward the ``a photo of a \{class\}'' embeddings to mitigate forgetting. Beyond the text side, VPT~\cite{jia2022visual} inserts learnable tokens into vision transformers, and MaPLe~\cite{khattak2023maple} jointly prompts both encoders across depth by mapping deep text prompts to deep visual prompts. AAPL~\cite{kim2024aapl} disentangles attribute-related tokens from augmentation noise. Our prior MSGCoOp~\cite{wang2025msgcoop} ensembles $N$ parallel context vectors with LLM-generated semantic anchors and a diversity regularizer. These methods either fuse branch outputs with fixed equal weights or regularize branches only implicitly, and none exploits CLIP's zero-shot prediction itself to decide \emph{per sample} how much each branch should contribute. DZGCoOp differs in using the frozen encoders to produce an instance-level confidence that routes between an explicitly conservative branch and an explicitly exploratory one, while inheriting the deep cross-modal prompting of MaPLe.
|
||||
|
||||
\noindent\textbf{Knowledge preservation with zero-shot priors.}
|
||||
Preserving pre-trained knowledge during few-shot adaptation is a central concern of prompt learning. KgCoOp~\cite{yao2023visual} and ProGrad~\cite{zhu2023prompt} anchor optimization to hand-crafted template embeddings; MSGCoOp~\cite{wang2025msgcoop} enriches such anchors with LLM-generated class descriptions~\cite{brown2020language,achiam2023gpt}. Distillation-style approaches additionally constrain output distributions. DZGCoOp extends this line along two axes: the zero-shot prior is used not only as a feature-space anchor but also as a distribution-level teacher via a KL guidance loss, and the zero-shot confidence actively participates in inference-time branch fusion rather than only in training-time regularization.
|
||||
|
||||
\noindent\textbf{Weight averaging.}
|
||||
Stochastic Weight Averaging (SWA)~\cite{izmailov2018averaging} averages checkpoints sampled in a late phase of training and finds flatter minima. Prompt learning, however, exhibits strong epoch-dependent behavior: models trained longer overfit the source classes and transfer worse~\cite{wang2025msgcoop}. Our epoch-weighted averaging (EWA) generalizes uniform averaging with a Gaussian weighting schedule whose center and width select the favorable phase per transfer regime, subsuming ``early-checkpoint'' selection without extra inference cost.
|
||||
Reference in New Issue
Block a user