207 lines
16 KiB
TeX
207 lines
16 KiB
TeX
\section{Experiments}
|
|
\label{sec:exp}
|
|
|
|
\subsection{Setup}
|
|
\label{sec:setup}
|
|
|
|
\noindent\textbf{Benchmarks and protocols.}
|
|
We evaluate three transfer protocols.
|
|
\emph{(i) Base-to-novel generalization}: following the standard splits, 11 datasets (ImageNet~\cite{deng2009imagenet}, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) are divided into base classes for few-shot training and novel classes for evaluation, reporting base accuracy (B), novel accuracy (N), and their harmonic mean (HM).
|
|
\emph{(ii) Cross-domain transfer}: the model trained on the 1000 classes of ImageNet is evaluated on ImageNetV2~\cite{recht2019imagenet}, ImageNet-Sketch~\cite{wang2019learning}, ImageNet-A~\cite{hendrycks2021natural}, and ImageNet-R~\cite{hendrycks2021many}.
|
|
\emph{(iii) Cross-dataset transfer}: the ImageNet-trained model is directly evaluated on the other 10 datasets.
|
|
|
|
\noindent\textbf{Implementation details.}
|
|
We use CLIP ViT-B/16 as the backbone. All models are trained in the 16-shot setting with batch size 4 using SGD (learning rate 0.0025) for 20 epochs with cosine annealing. Shallow visual and textual prompts have 4 tokens each. Deep prompt depths follow the recommended configuration of MaPLe~\cite{khattak2023maple}: 9 layers for base-to-novel and 3 layers for cross-domain and cross-dataset transfer. Guidance weights are $\lambda_{\mathrm{zvg}}{=}8$ and $\lambda_{\mathrm{sg}}^{\mathrm{weak}}{=}8$, inherited from~\cite{yao2023visual,wang2025msgcoop}, and $\lambda_{\mathrm{sg}}^{\mathrm{str}}{=}24$ (3$\times$ the weak weight) selected by grid search to realize the conservative-vs-exploratory asymmetry. EWA uses $(\mu,\sigma){=}(15,1)$ for base-to-novel and $(6,10)$ otherwise. All results are averaged over 3 random seeds.
|
|
|
|
\begin{table*}[t]
|
|
\centering
|
|
\caption{\textbf{Base-to-novel generalization on 11 benchmarks (16-shot).}
|
|
\textbf{Bold} marks the best result per column.
|
|
\textcolor{MidnightBlue}{Blue} (resp. \textcolor{Bittersweet}{red}) numbers show absolute gains (resp. drops) of DZGCoOp over MaPLe.}
|
|
\label{tab:b2n}
|
|
\renewcommand{\arraystretch}{1.15}
|
|
\addtolength{\tabcolsep}{-2pt}
|
|
\scalebox{0.72}{
|
|
\begin{tabular}{l ccc ccc ccc ccc ccc ccc ccc}
|
|
\toprule
|
|
\textbf{Method} & \multicolumn{3}{c}{Average} & \multicolumn{3}{c}{ImageNet} & \multicolumn{3}{c}{Caltech101} & \multicolumn{3}{c}{OxfordPets} & \multicolumn{3}{c}{StanfordCars} & \multicolumn{3}{c}{Flowers102} \\
|
|
\cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}\cmidrule(lr){14-16}\cmidrule(lr){17-19}
|
|
& B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM \\
|
|
\midrule
|
|
CLIP & 69.34 & 74.22 & 71.70 & 72.43 & 68.14 & 70.22 & 96.84 & 94.00 & 95.40 & 91.17 & 97.26 & 94.12 & 63.37 & 74.89 & 68.65 & 72.08 & \textbf{77.80} & 74.83 \\
|
|
CoOp~\cite{zhou2022learning} & 82.69 & 63.22 & 71.66 & 76.47 & 67.88 & 71.92 & 98.00 & 89.81 & 93.73 & 93.67 & 95.29 & 94.47 & 78.12 & 60.40 & 68.13 & 97.60 & 59.67 & 74.06 \\
|
|
CoCoOp~\cite{zhou2022conditional} & 80.47 & 71.69 & 75.83 & 75.98 & 70.43 & 73.10 & 97.96 & 93.81 & 95.84 & 95.20 & 97.69 & 96.43 & 70.49 & 73.59 & 72.01 & 94.87 & 71.75 & 81.71 \\
|
|
AAPL~\cite{kim2024aapl} & 80.27 & 72.17 & 76.01 & 76.53 & 70.57 & 73.43 & 97.87 & 95.10 & 96.46 & 95.63 & 97.40 & 96.51 & 70.33 & 73.50 & 71.88 & 95.10 & 70.63 & 81.06 \\
|
|
KgCoOp~\cite{yao2023visual} & 80.73 & 73.36 & 77.00 & 75.83 & 69.96 & 72.78 & 97.72 & 94.39 & 96.03 & 94.65 & 97.76 & 96.18 & 71.76 & \textbf{75.04} & 73.36 & 95.00 & 74.73 & 83.65 \\
|
|
MSGCoOp~\cite{wang2025msgcoop} & 81.40 & 75.05 & 78.10 & 76.46 & 70.45 & 73.31 & 98.08 & \textbf{95.81} & \textbf{96.91} & 95.62 & \textbf{97.89} & \textbf{96.70} & 71.22 & 74.56 & 72.85 & 96.20 & 75.91 & 84.84 \\
|
|
MaPLe~\cite{khattak2023maple} & 82.28 & 75.14 & 78.55 & 76.66 & 70.54 & 73.47 & 97.74 & 94.36 & 96.02 & 95.43 & 97.76 & 96.58 & 72.94 & 74.00 & 73.47 & 95.92 & 72.46 & 82.56 \\
|
|
\rowcolor{tabhighlight}
|
|
DZGCoOp & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} & \textbf{77.80} & \textbf{70.74} & \textbf{74.11} & \textbf{98.30} & 95.05 & 96.65 & \textbf{95.67} & 97.61 & 96.63 & \textbf{78.35} & 74.85 & \textbf{76.56} & \textbf{97.69} & 77.14 & \textbf{86.21} \\
|
|
& \textcolor{MidnightBlue}{{+2.24}} & \textcolor{MidnightBlue}{{+1.40}} & \textcolor{MidnightBlue}{{+1.78}} & \textcolor{MidnightBlue}{{+1.14}} & \textcolor{MidnightBlue}{{+0.20}} & \textcolor{MidnightBlue}{{+0.64}} & \textcolor{MidnightBlue}{{+0.56}} & \textcolor{MidnightBlue}{{+0.69}} & \textcolor{MidnightBlue}{{+0.63}} & \textcolor{MidnightBlue}{{+0.24}} & \textcolor{Bittersweet}{{-0.15}} & \textcolor{MidnightBlue}{{+0.05}} & \textcolor{MidnightBlue}{{+5.41}} & \textcolor{MidnightBlue}{{+0.85}} & \textcolor{MidnightBlue}{{+3.09}} & \textcolor{MidnightBlue}{{+1.77}} & \textcolor{MidnightBlue}{{+4.68}} & \textcolor{MidnightBlue}{{+3.65}} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
}
|
|
|
|
\bigskip
|
|
|
|
\scalebox{0.72}{
|
|
\begin{tabular}{l ccc ccc ccc ccc ccc ccc ccc}
|
|
\toprule
|
|
\textbf{Method} & \multicolumn{3}{c}{Food101} & \multicolumn{3}{c}{FGVCAircraft} & \multicolumn{3}{c}{SUN397} & \multicolumn{3}{c}{DTD} & \multicolumn{3}{c}{EuroSAT} & \multicolumn{3}{c}{UCF101} \\
|
|
\cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}\cmidrule(lr){14-16}\cmidrule(lr){17-19}
|
|
& B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM \\
|
|
\midrule
|
|
CLIP & 90.10 & 91.22 & 90.66 & 27.19 & 36.29 & 31.09 & 69.36 & 75.35 & 72.23 & 53.24 & 59.90 & 56.37 & 56.48 & 64.05 & 60.03 & 70.53 & 77.50 & 73.85 \\
|
|
CoOp~\cite{zhou2022learning} & 88.33 & 82.26 & 85.19 & 40.44 & 22.30 & 28.75 & 80.60 & 65.89 & 72.51 & 79.44 & 41.18 & 54.24 & 92.19 & 54.74 & 68.69 & 84.69 & 56.05 & 67.46 \\
|
|
CoCoOp~\cite{zhou2022conditional} & 90.70 & 91.29 & 90.99 & 33.41 & 23.71 & 27.74 & 79.74 & 76.86 & 78.27 & 77.01 & 56.00 & 64.85 & 87.49 & 60.04 & 71.21 & 82.33 & 73.45 & 77.64 \\
|
|
AAPL~\cite{kim2024aapl} & 90.70 & 91.60 & 91.15 & 34.07 & 24.17 & 28.28 & 79.65 & 76.90 & 78.25 & 73.90 & 53.43 & 62.02 & 87.00 & 66.30 & 75.25 & 82.20 & 74.27 & 78.03 \\
|
|
KgCoOp~\cite{yao2023visual} & 90.50 & 91.70 & 91.09 & 36.21 & 33.55 & 34.83 & 80.29 & 76.53 & 78.36 & 77.55 & 54.99 & 64.35 & 85.64 & 64.34 & 73.48 & 82.89 & 76.67 & 79.65 \\
|
|
MSGCoOp~\cite{wang2025msgcoop} & \textbf{90.72} & 91.62 & 91.18 & 36.73 & 34.57 & 35.62 & 81.38 & 76.20 & 78.71 & 79.32 & 56.64 & 66.09 & 86.29 & 74.97 & 80.23 & 83.52 & 77.05 & 80.16 \\
|
|
MaPLe~\cite{khattak2023maple} & 90.71 & \textbf{92.05} & \textbf{91.38} & 37.44 & 35.61 & 36.50 & 80.82 & 78.70 & 79.75 & 80.36 & 59.18 & 68.16 & \textbf{94.07} & 73.23 & 82.35 & 83.00 & 78.66 & 80.77 \\
|
|
\rowcolor{tabhighlight}
|
|
DZGCoOp & 90.54 & 91.52 & 91.03 & \textbf{43.48} & \textbf{37.07} & \textbf{40.02} & \textbf{82.62} & \textbf{79.01} & \textbf{80.77} & \textbf{83.72} & \textbf{60.02} & \textbf{69.92} & 94.05 & \textbf{78.35} & \textbf{85.49} & \textbf{87.49} & \textbf{80.51} & \textbf{83.85} \\
|
|
& \textcolor{Bittersweet}{{-0.17}} & \textcolor{Bittersweet}{{-0.53}} & \textcolor{Bittersweet}{{-0.35}} & \textcolor{MidnightBlue}{{+6.04}} & \textcolor{MidnightBlue}{{+1.46}} & \textcolor{MidnightBlue}{{+3.52}} & \textcolor{MidnightBlue}{{+1.80}} & \textcolor{MidnightBlue}{{+0.31}} & \textcolor{MidnightBlue}{{+1.02}} & \textcolor{MidnightBlue}{{+3.36}} & \textcolor{MidnightBlue}{{+0.84}} & \textcolor{MidnightBlue}{{+1.76}} & \textcolor{Bittersweet}{{-0.02}} & \textcolor{MidnightBlue}{{+5.12}} & \textcolor{MidnightBlue}{{+3.14}} & \textcolor{MidnightBlue}{{+4.49}} & \textcolor{MidnightBlue}{{+1.85}} & \textcolor{MidnightBlue}{{+3.08}} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
}
|
|
\end{table*}
|
|
|
|
\subsection{Base-to-Novel Generalization}
|
|
|
|
Table~\ref{tab:b2n} reports the 16-shot results on the 11 benchmarks. DZGCoOp achieves the best averages of 84.52\% (B), 76.54\% (N), and 80.33\% (HM), outperforming MaPLe by $+2.24$/$+1.40$/$+1.78$ and MSGCoOp by $+3.12$/$+1.49$/$+2.23$. It obtains the best HM on 8 of 11 datasets, with the largest gains on fine-grained and special-domain sets: FGVCAircraft ($+3.52$ HM over MaPLe), whose subtle inter-class differences benefit from the weak branch's exploratory features, Flowers102 ($+3.65$), StanfordCars ($+3.09$), EuroSAT ($+3.14$), and UCF101 ($+3.08$). Meanwhile, the strong branch keeps base accuracy high (best on 9 of 11 datasets), confirming that the asymmetric design improves novel-class generalization without sacrificing base-class fitness.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Cross-domain transfer with 16-shot setting. The model is trained on ImageNet and evaluated on four shifted target domains.}
|
|
\label{tab:xdo}
|
|
\small
|
|
\addtolength{\tabcolsep}{-2.pt}
|
|
\renewcommand{\arraystretch}{1.06}
|
|
\scalebox{0.90}{
|
|
\begin{tabular}{lcccccc}
|
|
\toprule
|
|
& \multicolumn{1}{c}{Source} & \multicolumn{5}{c}{Target} \\ \cline{2-7}
|
|
& ImageNet & V2 & Sketch & A & R & Avg. \\
|
|
\midrule
|
|
CLIP & 66.73 & 60.83 & 46.15 & 47.77 & 73.96 & 57.17 \\
|
|
CoOp & 71.51 & 64.20 & 47.99 & 49.71 & 75.21 & 59.28 \\
|
|
CoCoOp & 71.02 & 64.07 & 48.75 & 50.63 & 76.18 & 59.90 \\
|
|
AAPL & 71.37 & 64.20 & 48.80 & 50.60 & 76.87 & 60.12 \\
|
|
KgCoOp & 71.20 & 64.10 & 48.97 & 50.69 & 76.70 & 60.11 \\
|
|
MSGCoOp & 71.59 & 64.54 & 49.24 & 50.90 & 76.94 & 60.41 \\
|
|
MaPLe & 70.72 & 64.07 & 49.15 & 50.90 & 76.98 & 60.27 \\
|
|
\rowcolor{tabhighlight}
|
|
DZGCoOp & \textbf{71.66} & \textbf{64.77} & \textbf{49.43} & \textbf{51.37} & \textbf{77.08} & \textbf{60.66} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\subsection{Cross-Domain Transfer}
|
|
|
|
Table~\ref{tab:xdo} shows robustness to distribution shift. DZGCoOp ranks first on \emph{all four} target domains and on average (60.66\%, $+0.25$ over MSGCoOp, $+0.39$ over MaPLe). The gain is most pronounced on the adversarial ImageNet-A ($51.37\%$, $+0.47$), where the zero-shot guidance module supplies stable priors for samples far from the training distribution. We attribute the overall robustness to two designs: the frozen zero-shot branch remains a reliable reference under shift, and the EWA schedule configured for transfer ($\mu{=}6$, $\sigma{=}10$) keeps early, more generic snapshots in the fused parameters, mitigating overfitting to the source domain.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Cross-dataset transfer with 16-shot setting. The model is trained on ImageNet and directly evaluated on 10 downstream datasets.}
|
|
\label{tab:xda}
|
|
\tabstyle{4pt}
|
|
\scalebox{0.71}{
|
|
\begin{tabular}{l cccccccc}
|
|
\toprule
|
|
\textbf{Dataset} & \rotbox{CLIP} & \rotbox{CoOp} & \rotbox{CoCoOp} & \rotbox{AAPL} & \rotbox{KgCoOp} & \rotbox{MSGCoOp} & \rotbox{MaPLe} & \rotbox{DZGCoOp} \\
|
|
\midrule
|
|
ImageNet & 66.72 & 71.51 & 71.02 & 71.37 & 70.66 & 71.59 & 70.72 & \cellcolor{tabhighlight}\textbf{71.66} \\
|
|
Caltech101 & 92.94 & 93.70 & 94.43 & 94.17 & 93.92 & 93.29 & 93.53 & \cellcolor{tabhighlight}\textbf{94.69} \\
|
|
OxfordPets & 89.07 & 89.14 & 90.14 & \textbf{90.73} & 89.83 & 87.90 & 90.49 & \cellcolor{tabhighlight}90.20 \\
|
|
StanfordCars & 65.29 & 64.51 & 65.32 & 65.10 & 65.41 & 65.10 & \textbf{65.57} & \cellcolor{tabhighlight}65.51 \\
|
|
Flowers102 & 71.30 & 68.71 & 71.88 & 71.67 & 70.01 & 69.87 & \textbf{72.23} & \cellcolor{tabhighlight}71.00 \\
|
|
Food101 & 86.11 & 85.30 & 86.06 & 86.00 & \textbf{86.36} & 85.60 & 86.20 & \cellcolor{tabhighlight}\textbf{86.36} \\
|
|
Aircraft & \textbf{24.87} & 18.47 & 22.94 & 23.03 & 22.51 & 19.14 & 24.74 & \cellcolor{tabhighlight}22.89 \\
|
|
SUN397 & 62.62 & 64.15 & 67.36 & 66.80 & 66.16 & 65.14 & 67.01 & \cellcolor{tabhighlight}\textbf{68.02} \\
|
|
DTD & 44.56 & 41.92 & 45.73 & 44.80 & 46.35 & 42.59 & \textbf{46.49} & \cellcolor{tabhighlight}46.34 \\
|
|
EuroSAT & 47.69 & 46.39 & 45.37 & 41.83 & 46.04 & 45.59 & \textbf{48.06} & \cellcolor{tabhighlight}44.82 \\
|
|
UCF101 & 66.77 & 66.55 & 68.21 & \textbf{69.30} & 68.50 & 67.51 & 68.69 & \cellcolor{tabhighlight}\textbf{70.08} \\
|
|
\midrule
|
|
Average & 65.12 & 63.88 & 65.74 & 65.34 & 65.51 & 64.17 & \textbf{66.30} & \cellcolor{tabhighlight}65.99 \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\subsection{Cross-Dataset Transfer}
|
|
|
|
Table~\ref{tab:xda} evaluates direct transfer of the ImageNet-trained model. DZGCoOp reaches 65.99\% on average, $+1.82$ over MSGCoOp and above CLIP, CoOp, CoCoOp, AAPL, and KgCoOp, though slightly below MaPLe ($-0.31$). It is the best on Caltech101 (94.69\%), SUN397 (68.02\%, $+1.01$ over MaPLe), and UCF101 (70.08\%, $+1.39$). The confidence-guided fusion helps most on datasets whose categories are well covered by CLIP's pre-training (e.g., scenes and actions), while the residual gap to MaPLe on EuroSAT and Aircraft reflects the difficulty of satellite and fine-grained domains far from ImageNet semantics, where the source-trained weak branch provides limited additional evidence.
|
|
|
|
\subsection{Ablation Studies}
|
|
\label{sec:ablation}
|
|
|
|
All ablations use the 16-shot base-to-novel setting averaged over the 11 datasets unless stated otherwise.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Ablation of the zero-shot guidance module and losses.}
|
|
\label{tab:abl_zg}
|
|
\small
|
|
\renewcommand{\arraystretch}{1.15}
|
|
\scalebox{0.82}{
|
|
\begin{tabular}{l ccc}
|
|
\toprule
|
|
Method & B & N & HM \\
|
|
\midrule
|
|
IVLP & 84.21 & 71.79 & 77.51 \\
|
|
+ dual-branch + avg. fusion & 84.33 & 72.10 & 77.74 \\
|
|
+ ZG losses & 84.38 & 76.38 & 80.18 \\
|
|
\rowcolor{tabhighlight}
|
|
+ dynamic fusion (full model) & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\noindent\textbf{Zero-shot guidance.}
|
|
Table~\ref{tab:abl_zg} starts from IVLP (independent visual-language prompting, \emph{i.e.}, MaPLe's prompting backbone without its coupling)~\cite{khattak2023maple}. Adding the asymmetric dual-branch with plain average fusion yields only $+0.23$ HM; the zero-shot guidance (ZG) losses are the key driver, lifting novel accuracy by $+4.28$ and HM by $+2.44$. Replacing average fusion with the confidence-guided dynamic fusion contributes a further $+0.15$ HM on top.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Ablation of the dual-branch design.}
|
|
\label{tab:abl_dual}
|
|
\small
|
|
\renewcommand{\arraystretch}{1.15}
|
|
\scalebox{0.83}{
|
|
\begin{tabular}{l ccc}
|
|
\toprule
|
|
Method & B & N & HM \\
|
|
\midrule
|
|
Single branch (weak only) & 83.59 & 75.10 & 79.12 \\
|
|
Single branch (strong only) & 84.31 & 76.27 & 80.09 \\
|
|
Symmetric dual (two strong) + avg. & \textbf{84.65} & 75.50 & 79.81 \\
|
|
Asymmetric dual (strong + weak) + avg. & 84.38 & 76.38 & 80.18 \\
|
|
\rowcolor{tabhighlight}
|
|
Full (asym. dual + dynamic fusion) & 84.52 & \textbf{76.54} & \textbf{80.33} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\noindent\textbf{Asymmetric dual-branch.}
|
|
Table~\ref{tab:abl_dual} dissects the branch structure. A single strong branch (80.09\% HM) beats a single weak branch (79.12\%), confirming the value of template initialization. Two \emph{strong} branches with average fusion reach the highest base accuracy (84.65\%) but a lower HM (79.81\%) than the asymmetric pair (80.18\%)---their predictions are highly correlated, so the ensemble degenerates into redundancy. The asymmetric strong-weak pairing improves novel accuracy by $+0.88$, and dynamic fusion adds another $+0.15$ HM, validating that the value of a second branch comes from complementary roles and instance-adaptive weighting rather than from ensembling per se.
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Ablation of the parameter fusion strategy.}
|
|
\label{tab:abl_ewa}
|
|
\small
|
|
\renewcommand{\arraystretch}{1.15}
|
|
\scalebox{0.92}{
|
|
\begin{tabular}{l ccc}
|
|
\toprule
|
|
Method & B & N & HM \\
|
|
\midrule
|
|
Last-Epoch & 84.18 & 76.14 & 79.96 \\
|
|
SWA~\cite{izmailov2018averaging} & 84.35 & 76.28 & 80.11 \\
|
|
\rowcolor{tabhighlight}
|
|
EWA (ours) & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} \\
|
|
\bottomrule
|
|
\end{tabular}}
|
|
\end{table}
|
|
|
|
\noindent\textbf{Parameter fusion strategy.}
|
|
Table~\ref{tab:abl_ewa} compares the final-parameter selection. Uniform late averaging (SWA) gains $+0.15$ HM over the last epoch; EWA's Gaussian schedule centered at $\mu{=}15$ with $\sigma{=}1$ further reaches 80.33\% ($+0.37$ over last-epoch, $+0.22$ over SWA), showing that explicitly weighting the stable late phase smooths single-epoch fluctuation. In cross-domain and cross-dataset regimes, the schedule shifts early ($\mu{=}6$, $\sigma{=}10$), which is what preserves generic semantics in Tables~\ref{tab:xdo} and~\ref{tab:xda}.
|