\section{Method}
\label{sec:method}

We propose DZGCoOp, a prompt learning framework that exploits the frozen CLIP's zero-shot capability to guide an asymmetric dual-branch multimodal prompt network. Fig.~\ref{fig:framework} illustrates the overall pipeline: a zero-shot guidance stage builds priors from the frozen encoders, a dual-branch learning stage extracts deep-prompted visual and textual features and fuses them per instance, and a loss-and-optimization stage aligns the fused prediction with both labels and zero-shot priors. Throughout training, only prompt-related parameters are optimized.

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.92\textwidth]{fig/dzg_arch.pdf}
    \caption{Architecture of DZGCoOp. \textbf{Blue}: the zero-shot guidance module distills LLM-enriched semantic anchors, zero-shot visual features, a zero-shot probability distribution, and an instance-level confidence $\alpha$ from the frozen CLIP encoders. \textbf{Green}: the dual-branch multimodal learning module couples a conservative strong branch and an exploratory weak branch through shared deep prompt layers on both encoders and fuses their logits with $\alpha$. \textbf{Purple \& orange}: guidance losses and the epoch-weighted averaging (EWA) strategy. Only prompt parameters are updated.}
    \label{fig:framework}
\end{figure*}

\subsection{Preliminaries}
\label{sec:prelim}

CLIP consists of a vision encoder $\phi(\cdot)$ and a text encoder $\theta(\cdot)$ trained contrastively on web-scale image-text pairs. For a task with $N_c$ classes $\{\mathrm{cls}_1,\dots,\mathrm{cls}_{N_c}\}$, zero-shot classification encodes the template ``a photo of a [CLASS]'' into text features $\mathbf{w}_i^{\mathrm{txt}}$ and predicts via cosine similarity between $\mathbf{w}^{\mathrm{img}}=\phi(I)$ and each $\mathbf{w}_i^{\mathrm{txt}}$. Prompt learning replaces the fixed template with learnable context vectors while keeping $\phi$ and $\theta$ frozen. Following MaPLe~\cite{khattak2023maple}, we insert prompts at both the input layer (\emph{shallow}) and intermediate layers (\emph{deep}) of both encoders.

\subsection{Zero-Shot Guidance Module}
\label{sec:zsg}

The zero-shot guidance module distills four priors from the frozen CLIP, which are consumed later by the fusion rule and the guidance losses.

\noindent\textbf{Class-level priors.}
Following MSGCoOp~\cite{wang2025msgcoop}, a large language model~\cite{achiam2023gpt} generates multiple visually grounded descriptions of each class covering appearance, color, shape, and texture. The descriptions of class $\mathrm{cls}_i$ form a set $D_i$; encoding them with the frozen text encoder and averaging yields the \emph{semantic anchor}
\begin{equation}
    \mathbf{w}^{\mathrm{zst}}_i = \frac{1}{|D_i|} \sum_{d \in D_i} \theta(d),
    \label{eq:anchor}
\end{equation}
a consensus representation of the class in CLIP's semantic space. Since anchors are instance-independent, they are precomputed once before training.

\noindent\textbf{Instance-level priors.}
For an input image $I$, the frozen vision encoder yields the zero-shot visual feature $\mathbf{w}^{\mathrm{zsv}} = \phi(I)$. Cosine similarities against all anchors form the zero-shot logits $l^{\mathrm{zs}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{zsv}}, \mathbf{w}^{\mathrm{zst}}_i)$, which after softmax with temperature $\tau$ give the zero-shot probability distribution
\begin{equation}
    p^{\mathrm{zs}}_i = \frac{\exp(l^{\mathrm{zs}}_i / \tau)}{\sum_{j=1}^{N_c} \exp(l^{\mathrm{zs}}_j / \tau)}.
    \label{eq:pzs}
\end{equation}
The \emph{instance-level confidence} is the maximum zero-shot probability,
\begin{equation}
    \alpha = \max_{i}\, p^{\mathrm{zs}}_i \in [1/N_c,\, 1],
    \label{eq:alpha}
\end{equation}
which measures how deterministically the frozen CLIP ``knows'' this sample: $\alpha\!\to\!1$ indicates a confident zero-shot prediction, whereas $\alpha\!\to\!1/N_c$ indicates a near-uniform distribution. On large-scale benchmarks such as ImageNet, the average confidence on correct classes lies around $0.6$--$0.8$, making $\alpha$ a reliable per-sample signal. All computations in this module use frozen parameters and incur no gradient cost.

\subsection{Dual-Branch Multimodal Learning Module}
\label{sec:dual}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.92\linewidth]{fig/dzg_fusion.pdf}
    \caption{Zero-shot-confidence-guided dynamic fusion. The instance-level confidence $\alpha$ extracted from the frozen CLIP adaptively balances the conservative strong branch and the exploratory weak branch.}
    \label{fig:fusion}
\end{figure}

The learnable part of DZGCoOp is an asymmetric dual-branch network with deep cross-modal prompting.

\noindent\textbf{Deep cross-modal prompting.}
On the vision side, shallow visual prompts $\mathbf{v}^{\mathrm{vis}}$ are prepended to the patch tokens, and deep visual prompts $\text{prompt}_{\text{layer}_i}^{\mathrm{vis}}$ are injected into intermediate transformer layers, yielding a prompted encoder $\tilde{\phi}(\cdot;\cdot)$ whose backbone stays frozen. On the text side, shallow contexts $\mathbf{v}^{\mathrm{str}}$ and $\mathbf{v}^{\mathrm{weak}}$ feed the strong and weak branches respectively, while the deep text prompts $\text{prompt}_{\text{layer}_i}^{\mathrm{text}}$ are \emph{shared} between the two branches. The visual pathway is single and shared: the visual feature $\mathbf{w}^{\mathrm{vis}} = \tilde{\phi}(I; V^{\mathrm{vis}})$ serves both branches.

\noindent\textbf{Asymmetric branch design.}
The strong branch concatenates $\mathbf{v}^{\mathrm{str}}$ (initialized from the hand-crafted template ``a photo of a'') with the class token and encodes it as $\mathbf{w}^{\mathrm{str}}_i = \tilde{\theta}(\mathrm{cls}_i; V^{\mathrm{str}})$; the weak branch uses randomly initialized $\mathbf{v}^{\mathrm{weak}}$ and yields $\mathbf{w}^{\mathrm{weak}}_i = \tilde{\theta}(\mathrm{cls}_i; V^{\mathrm{weak}})$. The strong branch thus starts aligned with pre-trained knowledge and is trained under a strong anchoring constraint, so it \emph{preserves} zero-shot generalization; the weak branch starts from a random point under a weak constraint, so it \emph{explores} task-specific semantics that the template cannot express. Compared with implicit diversity regularization in multi-prompt ensembles~\cite{wang2025msgcoop}, this initialization-plus-guidance asymmetry assigns the two branches explicit and complementary roles.

\noindent\textbf{Confidence-guided dynamic fusion.}
Each branch scores the image against its class features,
\begin{equation}
    l^{\mathrm{str}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{vis}}, \mathbf{w}^{\mathrm{str}}_i), \quad
    l^{\mathrm{weak}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{vis}}, \mathbf{w}^{\mathrm{weak}}_i),
\end{equation}
and the final logit vector is an instance-level convex combination (Fig.~\ref{fig:fusion}):
\begin{equation}
    \mathbf{l}^{\mathrm{fin}} = \alpha \cdot \mathbf{l}^{\mathrm{str}} + (1 - \alpha) \cdot \mathbf{l}^{\mathrm{weak}}.
    \label{eq:fusion}
\end{equation}
When CLIP is confident about a sample, the strong branch dominates and pre-training knowledge is preserved; when CLIP is uncertain, the weak branch contributes more of its task-specific evidence. Unlike equal-weight averaging, Eq.~\eqref{eq:fusion} requires no extra parameters and adapts the fusion weights to every sample. The fused logits are normalized by softmax into the final distribution $\mathbf{p}^{\mathrm{fin}}$.

\subsection{Guidance Losses and Optimization}
\label{sec:loss}

\noindent\textbf{Objective.}
The total loss combines classification and two zero-shot guidance terms:
\begin{equation}
    \mathcal{L} = \mathcal{L}_{\mathrm{ce}} + \mathcal{L}_{\mathrm{zfg}} + \mathcal{L}_{\mathrm{zpg}}.
    \label{eq:total}
\end{equation}
$\mathcal{L}_{\mathrm{ce}}$ is the cross-entropy on the fused logits $\mathbf{l}^{\mathrm{fin}}$. Because $\mathbf{l}^{\mathrm{fin}}$ is the confidence-weighted combination of both branches, the classification gradient supervises them jointly.

\noindent\textbf{Zero-shot feature guidance.}
To prevent the prompts from drifting away from CLIP's representation space, we align learned features with the zero-shot features via $L_1$ distances:
\begin{equation}
    \mathcal{L}_{\mathrm{zfg}} = \lambda_{\mathrm{zvg}} \mathcal{L}_{\mathrm{zvg}} + \lambda_{\mathrm{sg}}^{\mathrm{str}} \mathcal{L}_{\mathrm{sg}}^{\mathrm{str}} + \lambda_{\mathrm{sg}}^{\mathrm{weak}} \mathcal{L}_{\mathrm{sg}}^{\mathrm{weak}},
    \label{eq:zfg}
\end{equation}
\begin{equation}
    \mathcal{L}_{\mathrm{zvg}} = \frac{1}{B} \sum_{j=1}^{B} \|\mathbf{w}_j^{\mathrm{vis}} - \mathbf{w}_j^{\mathrm{zsv}}\|_1, \quad
    \mathcal{L}_{\mathrm{sg}}^{\mathrm{b}} = \frac{1}{N_c} \sum_{i=1}^{N_c} \|\mathbf{w}_i^{\mathrm{b}} - \mathbf{w}_i^{\mathrm{zst}}\|_1,
\end{equation}
where $B$ is the batch size and $\mathrm{b}\in\{\mathrm{str},\mathrm{weak}\}$. $L_1$ constrains feature \emph{positions} dimension-wise rather than only directions. Mirroring the asymmetric design, the guidance weights satisfy $\lambda_{\mathrm{sg}}^{\mathrm{str}} > \lambda_{\mathrm{sg}}^{\mathrm{weak}}$ (with $\lambda_{\mathrm{zvg}}$ and $\lambda_{\mathrm{sg}}^{\mathrm{weak}}$ following the mature configuration of~\cite{yao2023visual,wang2025msgcoop}), so the strong branch is anchored tightly while the weak branch keeps freedom to explore.

\noindent\textbf{Zero-shot probability guidance.}
At the distribution level, the zero-shot probabilities $\mathbf{p}^{\mathrm{zs}}$ act as a soft teacher through a KL divergence~\cite{kullback1951information}:
\begin{equation}
    \mathcal{L}_{\mathrm{zpg}} = \frac{1}{B \cdot N_c} \sum_{j=1}^{B} \sum_{i=1}^{N_c} p_{j,i}^{\mathrm{zs}} \log\frac{p_{j,i}^{\mathrm{zs}}}{p_{j,i}^{\mathrm{fin}}}.
    \label{eq:zpg}
\end{equation}
Beyond one-hot supervision, $\mathbf{p}^{\mathrm{zs}}$ encodes CLIP's inter-class structure (semantically close classes receive similar probabilities), which regularizes few-shot training against overfitting and retains the ability to predict unseen classes.

\noindent\textbf{Epoch-weighted averaging.}
Prompt learners conventionally use the last-epoch parameters, which are sensitive to stochastic fluctuation. As illustrated in Fig.~\ref{fig:ewa}, we snapshot the learnable parameters $\xi_t$ at every epoch and fuse them with a Gaussian schedule after training:
\begin{equation}
    \xi_{\mathrm{final}} = \sum_{t=1}^{T} \epsilon_t \cdot \xi_t, \quad
    \epsilon_t \propto \exp\!\left(-\frac{(t - \mu)^2}{2\sigma^2}\right),
    \label{eq:ewa}
\end{equation}
where $T$ is the number of epochs and $(\mu,\sigma)$ place and shape the weighting window. For base-to-novel generalization, later snapshots are of higher quality, so the window centers on late epochs ($\mu{=}15$, $\sigma{=}1$); for cross-domain and cross-dataset transfer, early snapshots retain more generic pre-trained semantics, so the window shifts early and widens ($\mu{=}6$, $\sigma{=}10$). Unlike SWA~\cite{izmailov2018averaging}, which uniformly averages late-phase checkpoints and therefore hurts transfer when the model overfits the source dataset, EWA explicitly selects the favorable phase per task family. Inference uses $\xi_{\mathrm{final}}$ directly at no additional cost.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.92\linewidth]{fig/dzg_ewa.pdf}
    \caption{Epoch-weighted averaging (EWA). Snapshots along the training trajectory are fused with a Gaussian weight schedule whose center $\mu$ and width $\sigma$ adapt to the transfer regime.}
    \label{fig:ewa}
\end{figure}