Rain-Bus/DZGCoOp-Paper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DZGCoOp paper: CVPR author-kit scaffold and full draft
LaTeX Build / build (push) Canceled after 0s
Details

Browse Source 
Paper draft for the second work (thesis Ch.4 ZGDPE, renamed DZGCoOp):
abstract, intro, related work, method (zero-shot guidance module,
asymmetric dual-branch with deep prompts, confidence-guided dynamic
fusion, guidance losses + EWA), experiments (base-to-novel,
cross-domain, cross-dataset, ablations) and conclusion.

Compiles to 8 pages with tectonic (no errors, no overfull boxes).
This commit is contained in:
Rain&Bus
2026-10-14 08:01:40 +08:00
commit 11953eef67
18 changed files with 2724 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/latex-build.yml
+39
View File
@@ -0,0 +1,39 @@
name: LaTeX Build
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Compile LaTeX document
uses: xu-cheng/latex-action@v3
with:
root_file: main.tex
- name: Check for LaTeX errors and warnings
run: |
if grep -q "^!" main.log; then
echo "LaTeX errors found:"
grep "^!" main.log
exit 1
elif grep -q "Warning:" main.log; then
echo "LaTeX warnings found:"
grep "Warning:" main.log
echo "Build completed with warnings."
else
echo "LaTeX build completed successfully without errors or warnings."
fi
- name: Upload PDF
uses: actions/upload-artifact@v4
with:
name: main.pdf
path: main.pdf
.gitignore
+11
View File
@@ -0,0 +1,11 @@
# LaTeX build artifacts (regenerated by tectonic/latexmk)
*.aux
*.bbl
*.blg
*.brf
*.fdb_latexmk
*.fls
*.log
*.out
*.synctex.gz
*.xdv
README.md
+28
View File
@@ -0,0 +1,28 @@
# DZGCoOp
DZGCoOp: **D**ual-Branch **Z**ero-Shot **G**uided **Co**ntext **Op**timization for
Few-Shot Image Classification — the conference paper version of the second work
(Chapter 4 of the master's thesis, thesis method name: ZGDPE).
Based on the CVPR author kit (https://github.com/cvpr-org/author-kit), with the
original git history removed.
## Build
```bash
latexmk -pdf main.tex # or: pdflatex main && bibtex main && pdflatex main && pdflatex main
```
## Structure
- `main.tex` — document entry, title and authors
- `preamble.tex` — kit tweaks + table/color macros used by our tables
- `sec/` — abstract, intro, related work, method, experiments, conclusion
- `fig/` — architecture / fusion / EWA figures (copied from thesis `Figures/Work2`)
- `main.bib` — references (full-form entries)
## TODO before submission
- `main.tex`: fill `\paperID`, choose review vs. camera-ready `cvpr` option
- `main.bib`: update venue/status of `wang2025msgcoop` (our first work)
- Confirm the code release URL in the abstract
cvpr.sty
+508
View File
@@ -0,0 +1,508 @@
% ---------------------------------------------------------------
%
% No guarantee is given that the format corresponds perfectly to
% IEEE 8.5" x 11" Proceedings, but most features should be ok.
%
% ---------------------------------------------------------------
% with LaTeX2e:
% =============
%
% use as
% \documentclass[times,10pt,twocolumn]{article}
% \usepackage[options]{cvpr}
% \usepackage{times}
%
% "options" should be replaced by
% * "review" for submitting a paper for review,
% * "final" for the camera ready, and
% * "rebuttal" for the author rebuttal.
%
% specify references as
% {\small
% \bibliographystyle{ieee}
% \bibliography{...your files...}
% }
% ---------------------------------------------------------------
\NeedsTeXFormat{LaTeX2e}[1999/12/01]
\ProvidesPackage{cvpr}[2026 LaTeX class for IEEE CVPR]
\RequirePackage{times} % Integrate Times for here
\RequirePackage{xspace}
\RequirePackage[dvipsnames]{xcolor}
\RequirePackage{graphicx}
\RequirePackage{amsmath}
\RequirePackage{amssymb}
\RequirePackage{booktabs}
\RequirePackage[numbers,sort&compress]{natbib}
\setlength{\bibsep}{1pt plus 1pt minus 1pt}
\RequirePackage{silence} % Suppress unwanted warnings
\hbadness=10000 \vbadness=10000 \vfuzz=30pt \hfuzz=30pt
\WarningFilter{latexfont}{Font shape declaration}
\WarningFilter{latex}{Font shape}
\WarningFilter[rebuttal]{latex}{No \author given}
\RequirePackage{etoolbox}
% Use modern caption package to allow for sub-figures etc.
% Reproduces the original CVPR/ICCV style as closely as possible.
\RequirePackage[format=plain,labelformat=simple,labelsep=period,font=small,compatibility=false]{caption}
\RequirePackage[font=footnotesize,skip=3pt,subrefformat=parens]{subcaption}
\newtoggle{cvprfinal} % Camera-ready version
\newtoggle{cvprrebuttal} % Rebuttal
\newtoggle{cvprpagenumbers} % Force page numbers (in camera ready)
\toggletrue{cvprfinal}
\togglefalse{cvprrebuttal}
\togglefalse{cvprpagenumbers}
\DeclareOption{review}{\togglefalse{cvprfinal}\toggletrue{cvprpagenumbers}}
\DeclareOption{rebuttal}{\togglefalse{cvprfinal}\toggletrue{cvprrebuttal}}
\DeclareOption{pagenumbers}{\toggletrue{cvprpagenumbers}}
\DeclareOption*{\PackageWarning{cvpr}{Unkown option `\CurrentOption'}}
\ProcessOptions\relax
% Don't warn about missing author for rebuttal
\iftoggle{cvprrebuttal}{%
\ActivateWarningFilters[rebuttal]
}{}
% Breaking lines for URLs in the bib
\RequirePackage[hyphens]{url}
\Urlmuskip=0mu plus 1mu\relax
% ---------------------------------------------------------------
% Inlined version of the obsolete "everyshi-2001-05-15" package.
\newcommand{\@EveryShipout@Hook}{}
\newcommand{\@EveryShipout@AtNextHook}{}
\newcommand*{\EveryShipout}[1]
{\g@addto@macro\@EveryShipout@Hook{#1}}
\newcommand*{\AtNextShipout}[1]
{\g@addto@macro\@EveryShipout@AtNextHook{#1}}
\newcommand{\@EveryShipout@Shipout}{%
\afterassignment\@EveryShipout@Test
\global\setbox\@cclv= %
}
\newcommand{\@EveryShipout@Test}{%
\ifvoid\@cclv\relax
\aftergroup\@EveryShipout@Output
\else
\@EveryShipout@Output
\fi%
}
\newcommand{\@EveryShipout@Output}{%
\@EveryShipout@Hook%
\@EveryShipout@AtNextHook%
\gdef\@EveryShipout@AtNextHook{}%
\@EveryShipout@Org@Shipout\box\@cclv%
}
\newcommand{\@EveryShipout@Org@Shipout}{}
\newcommand*{\@EveryShipout@Init}{%
\message{ABD: EveryShipout initializing macros}%
\let\@EveryShipout@Org@Shipout\shipout
\let\shipout\@EveryShipout@Shipout
}
\AtBeginDocument{\@EveryShipout@Init}
% ---------------------------------------------------------------
% ---------------------------------------------------------------
% Inlined simplified version of the "eso-pic" package.
\newcommand\LenToUnit[1]{#1\@gobble}
\newcommand\AtPageUpperLeft[1]{%
\begingroup
\@tempdima=0pt\relax\@tempdimb=\ESO@yoffsetI\relax
\put(\LenToUnit{\@tempdima},\LenToUnit{\@tempdimb}){#1}%
\endgroup
}
\newcommand\AtPageLowerLeft[1]{\AtPageUpperLeft{%
\put(0,\LenToUnit{-\paperheight}){#1}}}
\newcommand\AtPageCenter[1]{\AtPageUpperLeft{%
\put(\LenToUnit{.5\paperwidth},\LenToUnit{-.5\paperheight}){#1}}%
}
\newcommand\AtTextUpperLeft[1]{%
\begingroup
\setlength\@tempdima{1in}%
\ifodd\c@page%
\advance\@tempdima\oddsidemargin%
\else%
\advance\@tempdima\evensidemargin%
\fi%
\@tempdimb=\ESO@yoffsetI\relax\advance\@tempdimb-1in\relax%
\advance\@tempdimb-\topmargin%
\advance\@tempdimb-\headheight\advance\@tempdimb-\headsep%
\put(\LenToUnit{\@tempdima},\LenToUnit{\@tempdimb}){#1}%
\endgroup
}
\newcommand\AtTextLowerLeft[1]{\AtTextUpperLeft{%
\put(0,\LenToUnit{-\textheight}){#1}}}
\newcommand\AtTextCenter[1]{\AtTextUpperLeft{%
\put(\LenToUnit{.5\textwidth},\LenToUnit{-.5\textheight}){#1}}}
\newcommand{\ESO@HookI}{} \newcommand{\ESO@HookII}{}
\newcommand{\ESO@HookIII}{}
\newcommand{\AddToShipoutPicture}{%
\@ifstar{\g@addto@macro\ESO@HookII}{\g@addto@macro\ESO@HookI}}
\newcommand{\ClearShipoutPicture}{\global\let\ESO@HookI\@empty}
\newcommand\ESO@isMEMOIR[1]{}
\@ifclassloaded{memoir}{\renewcommand\ESO@isMEMOIR[1]{#1}}{}
\newcommand{\@ShipoutPicture}{%
\bgroup
\@tempswafalse%
\ifx\ESO@HookI\@empty\else\@tempswatrue\fi%
\ifx\ESO@HookII\@empty\else\@tempswatrue\fi%
\ifx\ESO@HookIII\@empty\else\@tempswatrue\fi%
\if@tempswa%
\@tempdima=1in\@tempdimb=-\@tempdima%
\advance\@tempdimb\ESO@yoffsetI%
\ESO@isMEMOIR{%
\advance\@tempdima\trimedge%
\advance\@tempdima\paperwidth%
\advance\@tempdima-\stockwidth%
\if@twoside\ifodd\c@page\else%
\advance\@tempdima-2\trimedge%
\advance\@tempdima-\paperwidth%
\advance\@tempdima\stockwidth%
\fi\fi%
\advance\@tempdimb\trimtop}%
\unitlength=1pt%
\global\setbox\@cclv\vbox{%
\vbox{\let\protect\relax
\pictur@(0,0)(\strip@pt\@tempdima,\strip@pt\@tempdimb)%
\ESO@HookIII\ESO@HookI\ESO@HookII%
\global\let\ESO@HookII\@empty%
\endpicture}%
\nointerlineskip%
\box\@cclv}%
\fi
\egroup
}
\EveryShipout{\@ShipoutPicture}
\RequirePackage{keyval}
\newif\ifESO@dvips\ESO@dvipsfalse
\newif\ifESO@texcoord\ESO@texcoordfalse
\AtBeginDocument{%
\IfFileExists{color.sty}
{%
\RequirePackage{color}
\let\ESO@color=\color\let\ESO@colorbox=\colorbox
\let\ESO@fcolorbox=\fcolorbox
}{}
\@ifundefined{Gin@driver}{}%
{%
\ifx\Gin@driver\@empty\else%
\filename@parse{\Gin@driver}\def\reserved@a{dvips}%
\ifx\filename@base\reserved@a\ESO@dvipstrue\fi%
\fi
}%
\ifx\pdfoutput\undefined\else
\ifx\pdfoutput\relax\else
\ifcase\pdfoutput\else
\ESO@dvipsfalse%
\fi
\fi
\fi
}
\ifESO@texcoord
\def\ESO@yoffsetI{0pt}\def\ESO@yoffsetII{-\paperheight}
\else
\def\ESO@yoffsetI{\paperheight}\def\ESO@yoffsetII{0pt}
\fi
% ---------------------------------------------------------------
\typeout{CVPR 8.5 x 11-Inch Proceedings Style `cvpr.sty'.}
% ten point helvetica bold required for captions
% eleven point times bold required for second-order headings
% in some sites the name of the fonts may differ,
% change the name here:
\font\cvprtenhv = phvb at 8pt % *** IF THIS FAILS, SEE cvpr.sty ***
\font\elvbf = ptmb scaled 1100
\font\tenbf = ptmb scaled 1000
% If the above lines give an error message, try to comment them and
% uncomment these:
%\font\cvprtenhv = phvb7t at 8pt
%\font\elvbf = ptmb7t scaled 1100
%\font\tenbf = ptmb7t scaled 1000
% set dimensions of columns, gap between columns, and paragraph indent
\setlength{\textheight}{8.875in}
\setlength{\textwidth}{6.875in}
\setlength{\columnsep}{0.3125in}
\setlength{\topmargin}{0in}
\setlength{\headheight}{0in}
\setlength{\headsep}{0in}
\setlength{\parindent}{1pc}
\setlength{\oddsidemargin}{-0.1875in}
\setlength{\evensidemargin}{-0.1875in}
% Suppress page numbers when the appropriate option is given
\iftoggle{cvprpagenumbers}{}{%
\pagestyle{empty}
}
\AtBeginDocument{%
% Print an error if document class other than article is used
\@ifclassloaded{article}{}{%
\PackageError{cvpr}{Package only meant to be used with document class `article'}{Change document class to `article'.}
}
% Print a warning if incorrect options for article are specified
\@ifclasswith{article}{10pt}{}{%
\PackageWarningNoLine{cvpr}{Incorrect font size specified - CVPR requires 10-point fonts. Please load document class `article' with `10pt' option}
}
\@ifclasswith{article}{twocolumn}{}{%
\PackageWarningNoLine{cvpr}{Single column document - CVPR requires papers to have two-column layout. Please load document class `article' with `twocolumn' option}
}
\@ifclasswith{article}{letterpaper}{}{%
\PackageWarningNoLine{cvpr}{Incorrect paper size - CVPR uses paper size `letter'. Please load document class `article' with `letterpaper' option}
}
% Print a warning if hyperref is not loaded and/or if the pagebackref option is missing
\iftoggle{cvprfinal}{%
\@ifpackageloaded{hyperref}{}{%
\PackageWarningNoLine{cvpr}{Package `hyperref' is not loaded, but highly recommended for camera-ready version}
}
}{%
\@ifpackageloaded{hyperref}{
\@ifpackagewith{hyperref}{pagebackref}{}{
\PackageWarningNoLine{cvpr}{Package `hyperref' is not loaded with option `pagebackref', which is strongly recommended for review version}
}
}{%
\PackageWarningNoLine{cvpr}{Package `hyperref' is not loaded, but strongly recommended for review version}
}
}
}
\def\@maketitle{
\newpage
\null
\iftoggle{cvprrebuttal}{\vspace*{-.3in}}{\vskip .375in}
\begin{center}
% smaller title font only for rebuttal
\iftoggle{cvprrebuttal}{{\large \bf \@title \par}}{{\Large \bf \@title \par}}
% additional two empty lines at the end of the title
\iftoggle{cvprrebuttal}{\vspace*{-22pt}}{\vspace*{24pt}}{
\large
\lineskip .5em
\begin{tabular}[t]{c}
\iftoggle{cvprfinal}{
\@author
}{
\iftoggle{cvprrebuttal}{}{
Anonymous \confName~submission\\
\vspace*{1pt}\\
Paper ID \paperID
}
}
\end{tabular}
\par
}
% additional small space at the end of the author name
\vskip .5em
% additional empty line at the end of the title block
\vspace*{12pt}
\end{center}
}
\def\abstract{%
% Suppress page numbers when the appropriate option is given
\iftoggle{cvprpagenumbers}{}{%
\thispagestyle{empty}
}
\centerline{\large\bf Abstract}%
\vspace*{12pt}\noindent%
\it\ignorespaces%
}
\def\endabstract{%
% additional empty line at the end of the abstract
\vspace*{12pt}
}
\def\affiliation#1{\gdef\@affiliation{#1}} \gdef\@affiliation{}
% correct heading spacing and type
\def\cvprsection{\@startsection {section}{1}{\z@}
{-10pt plus -2pt minus -2pt}{7pt} {\large\bf}}
\def\cvprssect#1{\cvprsection*{#1}}
\def\cvprsect#1{\cvprsection{\texorpdfstring{\hskip -1em.~}{}#1}}
\def\section{\@ifstar\cvprssect\cvprsect}
\def\cvprsubsection{\@startsection {subsection}{2}{\z@}
{-8pt plus -2pt minus -2pt}{5pt} {\elvbf}}
\def\cvprssubsect#1{\cvprsubsection*{#1}}
\def\cvprsubsect#1{\cvprsubsection{\texorpdfstring{\hskip -1em.~}{}#1}}
\def\subsection{\@ifstar\cvprssubsect\cvprsubsect}
\def\cvprsubsubsection{\@startsection {subsubsection}{3}{\z@}
{-6pt plus -2pt minus -2pt}{3pt} {\tenbf}}
\def\cvprssubsubsect#1{\cvprsubsubsection*{#1}}
\def\cvprsubsubsect#1{\cvprsubsubsection{\texorpdfstring{\hskip -1em.~}{}#1}}
\def\subsubsection{\@ifstar\cvprssubsubsect\cvprsubsubsect}
%% --------- Page background marks: Ruler and confidentiality (only for review and rebuttal)
\iftoggle{cvprfinal}{
% In review and rebuttal mode, we use the "lineno" package for numbering lines.
% When switching to a different mode, the "\@LN" macro may remain in cached .aux files,
% leading to build errors (https://github.com/cvpr-org/author-kit/issues/49).
% Defining the macro as empty fixes that (https://tex.stackexchange.com/a/125779).
\makeatletter
\providecommand{\@LN}[2]{}
\makeatother
}{
% ----- define vruler
\makeatletter
\newbox\cvprrulerbox
\newcount\cvprrulercount
\newdimen\cvprruleroffset
\newdimen\cv@lineheight
\newdimen\cv@boxheight
\newbox\cv@tmpbox
\newcount\cv@refno
\newcount\cv@tot
% NUMBER with left flushed zeros \fillzeros[<WIDTH>]<NUMBER>
\newcount\cv@tmpc@ \newcount\cv@tmpc
\def\fillzeros[#1]#2{\cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi
\cv@tmpc=1 %
\loop\ifnum\cv@tmpc@<10 \else \divide\cv@tmpc@ by 10 \advance\cv@tmpc by 1 \fi
\ifnum\cv@tmpc@=10\relax\cv@tmpc@=11\relax\fi \ifnum\cv@tmpc@>10 \repeat
\ifnum#2<0\advance\cv@tmpc1\relax-\fi
\loop\ifnum\cv@tmpc<#1\relax0\advance\cv@tmpc1\relax\fi \ifnum\cv@tmpc<#1 \repeat
\cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi \relax\the\cv@tmpc@}%
\makeatother
% ----- end of vruler
%% Define linenumber setup
\RequirePackage[switch,mathlines]{lineno}
% Line numbers in CVPR blue using font from \cvprtenhv
\renewcommand\linenumberfont{\cvprtenhv\color[rgb]{.5,.5,1}}
\renewcommand\thelinenumber{\fillzeros[3]{\arabic{linenumber}}}
\setlength{\linenumbersep}{.75cm}
% Bug: An equation with $$ ... $$ isn't numbered, nor is the previous line.
% Patch amsmath commands so that the previous line and the equation itself
% are numbered. Bug: multiline has an extra line number.
% https://tex.stackexchange.com/questions/461186/how-to-use-lineno-with-amsmath-align
\RequirePackage{etoolbox} %% <- for \pretocmd, \apptocmd and \patchcmd
\newcommand*\linenomathpatch[1]{%
\expandafter\pretocmd\csname #1\endcsname {\linenomath}{}{}%
\expandafter\pretocmd\csname #1*\endcsname {\linenomath}{}{}%
\expandafter\apptocmd\csname end#1\endcsname {\endlinenomath}{}{}%
\expandafter\apptocmd\csname end#1*\endcsname {\endlinenomath}{}{}%
}
\newcommand*\linenomathpatchAMS[1]{%
\expandafter\pretocmd\csname #1\endcsname {\linenomathAMS}{}{}%
\expandafter\pretocmd\csname #1*\endcsname {\linenomathAMS}{}{}%
\expandafter\apptocmd\csname end#1\endcsname {\endlinenomath}{}{}%
\expandafter\apptocmd\csname end#1*\endcsname {\endlinenomath}{}{}%
}
%% Definition of \linenomathAMS depends on whether the mathlines option is provided
\expandafter\ifx\linenomath\linenomathWithnumbers
\let\linenomathAMS\linenomathWithnumbers
%% The following line gets rid of an extra line numbers at the bottom:
\patchcmd\linenomathAMS{\advance\postdisplaypenalty\linenopenalty}{}{}{}
\else
\let\linenomathAMS\linenomathNonumbers
\fi
% Add the numbers
\linenumbers
\AtBeginDocument{%
\linenomathpatch{equation}%
\linenomathpatchAMS{gather}%
\linenomathpatchAMS{multline}%
\linenomathpatchAMS{align}%
\linenomathpatchAMS{alignat}%
\linenomathpatchAMS{flalign}%
}
% \makevruler[<SCALE>][<INITIAL_COUNT>][<STEP>][<DIGITS>][<HEIGHT>]
\def\cvprruler#1{\makevruler[12pt][#1][1][3][0.993\textheight]\usebox{\cvprrulerbox}}
\AddToShipoutPicture{%
\color[rgb]{.5,.5,1}
\def\pid{\parbox{1in}{\begin{center}\bf\sf{\small \confName}\\\small \#\paperID\end{center}}}
\AtTextUpperLeft{%paperID in corners
\put(\LenToUnit{-65pt},\LenToUnit{45pt}){\pid}
\put(\LenToUnit{\textwidth-12pt},\LenToUnit{45pt}){\pid}
}
\AtTextUpperLeft{%confidential
\put(0,\LenToUnit{1cm}){\parbox{\textwidth}{\centering\cvprtenhv
\confName~\confYear~Submission \#\paperID. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.}}
}
}
} % end of not cvprfinal
%%% Make figure placement a little more predictable.
% We trust the user to move figures if this results
% in ugliness.
% Minimize bad page breaks at figures
\renewcommand{\textfraction}{0.01}
\renewcommand{\floatpagefraction}{0.99}
\renewcommand{\topfraction}{0.99}
\renewcommand{\bottomfraction}{0.99}
\renewcommand{\dblfloatpagefraction}{0.99}
\renewcommand{\dbltopfraction}{0.99}
\setcounter{totalnumber}{99}
\setcounter{topnumber}{99}
\setcounter{bottomnumber}{99}
% Add a period to the end of an abbreviation unless there's one
% already, then \xspace.
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\eg{\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot}
\def\ie{\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot}
\def\cf{\emph{cf}\onedot} \def\Cf{\emph{Cf}\onedot}
\def\etc{\emph{etc}\onedot} \def\vs{\emph{vs}\onedot}
\def\wrt{w.r.t\onedot} \def\dof{d.o.f\onedot}
\def\iid{i.i.d\onedot} \def\wolog{w.l.o.g\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
% ---------------------------------------------------------------
%% redefine the \title command so that a variable name is saved in \thetitle, and provides the \maketitlesupplementary command
\let\titleold\title
\renewcommand{\title}[1]{\titleold{#1}\newcommand{\thetitle}{#1}}
\def\maketitlesupplementary
{
\newpage
\twocolumn[
\centering
\Large
\textbf{\thetitle}\\
\vspace{0.5em}Supplementary Material \\
\vspace{1.0em}
] %< twocolumn
}
% ---------------------------------------------------------------
%% Support for easy cross-referencing (e.g. \cref{sec:intro}
% configured with \AtEndPreamble as it needs to be called after hyperref
\AtEndPreamble{
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
}
% ---------------------------------------------------------------
%% More compact compact itemize/enumeration (e.g. list contributions)
\RequirePackage[shortlabels,inline]{enumitem}
\setlist[itemize]{noitemsep,leftmargin=*,topsep=0em}
\setlist[enumerate]{noitemsep,leftmargin=*,topsep=0em}
fig/dzg_arch.pdf
BIN
View File
Binary file not shown.
fig/dzg_ewa.pdf
BIN
View File
Binary file not shown.
fig/dzg_fusion.pdf
BIN
View File
Binary file not shown.
ieeenat_fullname.bst
+1448
View File
File diff suppressed because it is too large Load Diff
main.bib
+158
View File
@@ -0,0 +1,158 @@
% References for the DZGCoOp paper.
% Full-form entries, no string abbreviations.
@inproceedings{radford2021learning,
title={Learning transferable visual models from natural language supervision},
author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and Krueger, Gretchen and Sutskever, Ilya},
booktitle={Proceedings of the International Conference on Machine Learning},
pages={8748--8763},
year={2021}
}
@inproceedings{jia2021scaling,
title={Scaling up visual and vision-language representation learning with noisy text supervision},
author={Jia, Chao and Yang, Yinfei and Xia, Ye and Chen, Yi-Ting and Parekh, Zarana and Pham, Hieu and Le, Quoc and Sung, Yun-Hsuan and Li, Zhen and Duerig, Tom},
booktitle={Proceedings of the International Conference on Machine Learning},
pages={4904--4916},
year={2021}
}
@article{zhou2022learning,
title={Learning to prompt for vision-language models},
author={Zhou, Kaiyang and Yang, Jingkang and Loy, Chen Change and Liu, Ziwei},
journal={International Journal of Computer Vision},
volume={130},
number={9},
pages={2337--2348},
year={2022}
}
@inproceedings{zhou2022conditional,
title={Conditional prompt learning for vision-language models},
author={Zhou, Kaiyang and Yang, Jingkang and Loy, Chen Change and Liu, Ziwei},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={16816--16825},
year={2022}
}
@inproceedings{yao2023visual,
title={Visual-language prompt tuning with knowledge-guided context optimization},
author={Yao, Hantao and Zhang, Rui and Xu, Changsheng},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={6757--6767},
year={2023}
}
@inproceedings{khattak2023maple,
title={MaPLe: Multi-modal prompt learning},
author={Khattak, Muhammad Uzair and Rasheed, Hanoona and Maaz, Muhammad and Khan, Salman and Khan, Fahad Shahbaz},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={19113--19122},
year={2023}
}
@inproceedings{kim2024aapl,
title={AAPL: Adding attributes to prompt learning for vision-language models},
author={Kim, Gahyeon and Kim, Sohee and Lee, Seokju},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={1572--1582},
year={2024}
}
@inproceedings{zhu2023prompt,
title={Prompt-aligned gradient for prompt tuning},
author={Zhu, Beier and Niu, Yulei and Han, Yucheng and Wu, Yue and Zhang, Hanwang},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={15659--15669},
year={2023}
}
@inproceedings{jia2022visual,
title={Visual prompt tuning},
author={Jia, Menglin and Tang, Luming and Chen, Bor-Chun and Cardie, Claire and Belongie, Serge and Hariharan, Bharath and Lim, Ser-Nam},
booktitle={European Conference on Computer Vision},
pages={709--727},
year={2022}
}
@inproceedings{izmailov2018averaging,
title={Averaging weights leads to wider optima and better generalization},
author={Izmailov, Pavel and Podoprikhin, Dmitrii and Garipov, Timur and Vetrov, Dmitry P. and Wilson, Andrew Gordon},
booktitle={Proceedings of the Conference on Uncertainty in Artificial Intelligence},
pages={876--885},
year={2018}
}
@article{kullback1951information,
title={On information and sufficiency},
author={Kullback, Solomon and Leibler, Richard A.},
journal={The Annals of Mathematical Statistics},
volume={22},
number={1},
pages={79--86},
year={1951}
}
@inproceedings{deng2009imagenet,
title={ImageNet: A large-scale hierarchical image database},
author={Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={248--255},
year={2009}
}
@inproceedings{recht2019imagenet,
title={Do ImageNet classifiers generalize to ImageNet?},
author={Recht, Benjamin and Roelofs, Rebecca and Schmidt, Ludwig and Shankar, Vaishaal},
booktitle={Proceedings of the International Conference on Machine Learning},
pages={5389--5400},
year={2019}
}
@inproceedings{wang2019learning,
title={Learning robust global representations by penalizing local predictive power},
author={Wang, Haohan and Ge, Songwei and Lipton, Zachary and Xing, Eric P.},
booktitle={Advances in Neural Information Processing Systems},
volume={32},
year={2019}
}
@inproceedings{hendrycks2021natural,
title={Natural adversarial examples},
author={Hendrycks, Dan and Zhao, Kevin and Basart, Steven and Steinhardt, Jacob and Song, Dawn},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={15262--15271},
year={2021}
}
@inproceedings{hendrycks2021many,
title={The many faces of robustness: A critical analysis of out-of-distribution generalization},
author={Hendrycks, Dan and Basart, Steven and Mu, Norman and Kadavath, Saurav and Wang, Frank and Dorundo, Evan and Desai, Rahul and Zhu, Tyler and Parajuli, Samyak and Guo, Mike and Song, Dawn and Steinhardt, Jacob and Gilmer, Justin},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={8340--8349},
year={2021}
}
@inproceedings{brown2020language,
title={Language models are few-shot learners},
author={Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D. and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and others},
booktitle={Advances in Neural Information Processing Systems},
volume={33},
pages={1877--1901},
year={2020}
}
@article{achiam2023gpt,
title={GPT-4 technical report},
author={Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Altman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Max and Altman, Sam and Anadkat, Shyamal and others},
journal={arXiv preprint arXiv:2303.08774},
year={2023}
}
% TODO: update the venue/status of our first work (MSGCoOp) before submission.
@article{wang2025msgcoop,
title={{MSGCoOp}: Multiple semantic-guided context optimization for few-shot learning},
author={Wang, Zhaolong and Sun, Tongfeng and Du, Mingzheng and Huang, Yachao and Xu, Xinzheng and Li, Zhongnian},
journal={arXiv preprint arXiv:XXXX.XXXXX},
year={2025}
}
main.pdf
BIN
View File
Binary file not shown.
main.tex
+80
View File
@@ -0,0 +1,80 @@
% DZGCoOp paper, based on the CVPR author-kit template
% (https://github.com/cvpr-org/author-kit)
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage{cvpr} % To produce the CAMERA-READY version
\usepackage[review]{cvpr} % To produce the REVIEW version
% \usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version
% Import additional packages in the preamble file, before hyperref
\input{preamble}
% It is strongly recommended to use hyperref, especially for the review version.
\definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
\usepackage[pagebackref,breaklinks,colorlinks,allcolors=cvprblue]{hyperref}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\paperID{*****} % *** Enter the Paper ID here
\def\confName{CVPR}
\def\confYear{2026}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{DZGCoOp: Dual-Branch Zero-Shot Guided Context Optimization for Few-Shot Image Classification}
%%%%%%%%% AUTHORS - PLEASE UPDATE
\author{Zhaolong Wang\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small zhaolongwang@cumt.edu.cn}
\and
Tongfeng Sun$^{*}$\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small suntf@cumt.edu.cn}
\and
Mingzheng Du\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small mingzhengdu@cumt.edu.cn}
\and
Yachao Huang\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small yachaohuang@cumt.edu.cn}
\and
Xinzheng Xu\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small xxzheng@cumt.edu.cn}
\and
Zhongnian Li\\
School of Computer Science and Technology\\
China University of Mining and Technology\\
Xuzhou, China\\
{\tt\small zhongnianli@cumt.edu.cn}
}
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_related}
\input{sec/3_method}
\input{sec/4_experiments}
\input{sec/5_conclusion}
{
\small
\bibliographystyle{ieeenat_fullname}
\bibliography{main}
}
\end{document}
preamble.tex
+85
View File
@@ -0,0 +1,85 @@
%% This file contains a number of tweaks that are typically applied to the main document.
%% They are not enabled by default, but can be enabled by uncommenting the relevant lines.
%%
%% Official support for 2-column teaser figure
%%
\usepackage{cuted} %< for strip environment
%%
%% For auto-referencing of figures (see fig/teaser.tex and \label{} therein)
%%
\usepackage{currfile} %< for \currfilebase macro
%%
%% caption and (global) spacing overrides
%%
\usepackage{caption} %< for "\captionof" commands (teaser)
% \setlength{\abovecaptionskip}{.5em} % space above caption
% \setlength{\belowcaptionskip}{1em} % space below caption
%%
%% Inline annotations; for predefined colors, refer to "dvipsnames" in the xcolor package:
%% https://tinyurl.com/overleaf-colors
%%
\newcommand{\red}[1]{{\color{red}#1}}
\newcommand{\todo}[1]{{\color{red}#1}}
\newcommand{\TODO}[1]{\textbf{\color{red}[TODO: #1]}}
%%
%% Disable for camera ready / submission by uncommenting these lines:
%%
% \renewcommand{\TODO}[1]{}
% \renewcommand{\todo}[1]{#1}
%%
%% Work harder in optimizing text layout. Typically shrinks text by 1/6 of page, enable
%% it at the very end of the writing process, when you are just above the page limit.
%%
% \usepackage{microtype}
%%
%% Fine-tune paragraph spacing.
%%
% \renewcommand{\paragraph}[1]{\vspace{.5em}\noindent\textbf{#1.}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Additions for DZGCoOp tables and figures
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{colortbl}
\definecolor{MidnightBlue}{RGB}{30,144,255}
\definecolor{Bittersweet}{RGB}{191,81,24}
\definecolor{tabhighlight}{HTML}{e5e5e5}
\newcommand{\rotbox}[1]{\rotatebox{55}{#1}}
\newcommand{\tabstyle}[1]{%
\setlength{\tabcolsep}{#1}\centering\small}
%%
%% Globally adjusts space between figure and caption.
%%
% \setlength{\abovecaptionskip}{.5em}
%%
%% Allows "the use of \paper to refer to the project name"
%% with automatic management of space at the end of the word.
%%
% \usepackage{xspace}
% \newcommand{\paper}{ProjectName\xspace}
%%
%% Commonly used math definitions.
%%
% \DeclareMathOperator*{\argmin}{arg\,min}
% \DeclareMathOperator*{\argmax}{arg\,max}
%%
%% Tighten underline.
%%
% \usepackage{soul}
% \setuldepth{foobar}
%%
%% Reconfigure paragraph (tweaks spacing)
%%
sec/0_abstract.tex
+3
View File
@@ -0,0 +1,3 @@
\begin{abstract}
Prompt learning has become the prevailing paradigm for adapting vision-language models such as CLIP to few-shot image classification. However, existing multi-branch prompt methods fuse branch predictions with fixed equal weights at the logit level, which is redundant for samples that the frozen CLIP already classifies with high confidence yet unable to reconcile the large disagreements that arise on low-confidence samples. Moreover, parallel branches typically share identical objectives, data, and losses, leaving the division of labor among branches implicit, while prompt injection restricted to the encoder input layer fails to exploit the hierarchical representations of deep transformers. We propose DZGCoOp, a Dual-Branch Zero-Shot Guided Context Optimization framework. DZGCoOp first distills frozen-CLIP priors---LLM-enriched semantic anchors, zero-shot visual features, a zero-shot probability distribution, and an instance-level confidence---then learns an asymmetric dual-branch multimodal prompt network in which a strongly guided, hand-crafted-initialized branch preserves pre-trained knowledge while a weakly guided, randomly-initialized branch explores task-specific semantics. The two branches, coupled through shared deep prompt layers on both encoders, are fused per-instance with the zero-shot confidence, and zero-shot guidance losses align learned features and output distributions with the frozen priors. An epoch-weighted averaging strategy further consolidates parameter snapshots along the training trajectory. On 11 benchmarks, DZGCoOp attains 80.33\% average harmonic mean in base-to-novel generalization (+1.78\% over MaPLe), and it also achieves consistent gains in cross-domain (60.66\%) and cross-dataset (65.99\%) transfer. Our code is available at: \href{https://github.com/Rain-Bus/DZGCoOp}{https://github.com/Rain-Bus/DZGCoOp}.
\end{abstract}
sec/1_intro.tex
+21
View File
@@ -0,0 +1,21 @@
\section{Introduction}
\label{sec:intro}
Vision-language models (VLMs) such as CLIP~\cite{radford2021learning} and ALIGN~\cite{jia2021scaling} learn aligned image-text embedding spaces from web-scale pairs and exhibit remarkable zero-shot classification: an image is categorized by comparing its embedding with class-name embeddings produced from hand-crafted templates such as ``a photo of a \{class\}''. Adapting such models to downstream tasks by full fine-tuning is expensive and prone to catastrophic forgetting, which has made \emph{prompt learning}~\cite{zhou2022learning} the dominant few-shot adaptation paradigm: only a small set of continuous context vectors is optimized while both encoders stay frozen.
Despite steady progress, two structural limitations persist in current prompt learners.
\textbf{(i) Sample-agnostic branch fusion.}
Multi-branch or ensemble-style methods---including our prior MSGCoOp~\cite{wang2025msgcoop}, CoCoOp~\cite{zhou2022conditional}, and AAPL~\cite{kim2024aapl}---aggregate branch logits with equal weights. For samples that frozen CLIP already predicts with high confidence, the branches largely agree and equal-weight averaging wastes both computation and representational capacity; for samples with high zero-shot uncertainty, the branches disagree substantially, and a plain mean cannot integrate their complementary evidence. The fusion weight is moreover identical for every instance, ignoring how reliably CLIP ``knows'' each sample.
\textbf{(ii) Implicit division of labor among branches.}
Parallel branches are typically trained with the same objective, data, and losses. Regularizers such as the diversity loss of MSGCoOp only implicitly discourage branch similarity, so branches may still converge to near-identical semantics, and none of them is explicitly responsible for preserving pre-trained knowledge versus exploring task-specific ones.
In addition, text-side prompt learning that inserts vectors only at the encoder input~\cite{zhou2022learning,yao2023visual} underuses the hierarchical representations of deep transformers, whereas VPT~\cite{jia2022visual} and MaPLe~\cite{khattak2023maple} show that injecting prompts across layers of both encoders yields finer-grained adaptation.
This paper proposes \textbf{DZGCoOp} (\textbf{D}ual-Branch \textbf{Z}ero-Shot \textbf{G}uided \textbf{Co}ntext \textbf{Op}timization), which turns the frozen CLIP itself into a per-sample routing and supervision signal. A zero-shot guidance module first distills four priors from the frozen encoders: LLM-enriched semantic anchors~\cite{wang2025msgcoop,brown2020language}, zero-shot visual features, a zero-shot probability distribution, and an instance-level confidence $\alpha$ given by the maximum zero-shot probability. On top of these priors, a dual-branch multimodal learning module builds an asymmetric architecture: a \emph{strong branch}, initialized from a hand-crafted template and strongly regularized toward the semantic anchors, is conservative and preserves pre-trained knowledge; a \emph{weak branch}, randomly initialized and weakly regularized, explores task-specific semantics. Both text branches share deep prompt layers coupled with deep visual prompts following MaPLe~\cite{khattak2023maple}. The final logits fuse the two branches \emph{per instance} as $\alpha\,\mathbf{l}^{\mathrm{str}}+(1-\alpha)\,\mathbf{l}^{\mathrm{weak}}$, so reliable CLIP knowledge dominates on confident samples while exploratory representations take over on uncertain ones. Two guidance losses align learned features and output distributions with the zero-shot priors, and an epoch-weighted averaging (EWA) strategy consolidates parameter snapshots along the training trajectory with a Gaussian schedule.
Our contributions are summarized as follows:
\begin{itemize}
\item We propose a zero-shot guidance module that exploits the frozen CLIP encoders to produce LLM-enriched semantic anchors, zero-shot visual features, zero-shot probabilities, and instance-level confidence, providing stable priors and an adaptive fusion signal for prompt learning.
\item We design an asymmetric dual-branch multimodal prompt network with shared deep prompt layers, in which strong and weak branches assume explicit complementary roles and are fused dynamically by the instance-level zero-shot confidence.
\item We introduce zero-shot feature and probability guidance losses together with an epoch-weighted averaging strategy, aligning optimization with pre-trained knowledge and smoothing training stochasticity.
\item Extensive experiments show that DZGCoOp achieves 80.33\% average harmonic mean on 11 base-to-novel benchmarks (+1.78\% over MaPLe), 60.66\% average accuracy in cross-domain transfer, and 65.99\% in cross-dataset transfer, with ablations validating each component.
\end{itemize}
sec/2_related.tex
+11
View File
@@ -0,0 +1,11 @@
\section{Related Work}
\label{sec:relw}
\noindent\textbf{Prompt learning for vision-language models.}
CoOp~\cite{zhou2022learning} first replaces hand-crafted templates with learnable continuous context vectors for CLIP. CoCoOp~\cite{zhou2022conditional} conditions the prompt on each input image to improve generalization to unseen classes. ProGrad~\cite{zhu2023prompt} restricts prompt updates to directions consistent with the zero-shot CLIP gradient, and KgCoOp~\cite{yao2023visual} regularizes learnable prompts toward the ``a photo of a \{class\}'' embeddings to mitigate forgetting. Beyond the text side, VPT~\cite{jia2022visual} inserts learnable tokens into vision transformers, and MaPLe~\cite{khattak2023maple} jointly prompts both encoders across depth by mapping deep text prompts to deep visual prompts. AAPL~\cite{kim2024aapl} disentangles attribute-related tokens from augmentation noise. Our prior MSGCoOp~\cite{wang2025msgcoop} ensembles $N$ parallel context vectors with LLM-generated semantic anchors and a diversity regularizer. These methods either fuse branch outputs with fixed equal weights or regularize branches only implicitly, and none exploits CLIP's zero-shot prediction itself to decide \emph{per sample} how much each branch should contribute. DZGCoOp differs in using the frozen encoders to produce an instance-level confidence that routes between an explicitly conservative branch and an explicitly exploratory one, while inheriting the deep cross-modal prompting of MaPLe.
\noindent\textbf{Knowledge preservation with zero-shot priors.}
Preserving pre-trained knowledge during few-shot adaptation is a central concern of prompt learning. KgCoOp~\cite{yao2023visual} and ProGrad~\cite{zhu2023prompt} anchor optimization to hand-crafted template embeddings; MSGCoOp~\cite{wang2025msgcoop} enriches such anchors with LLM-generated class descriptions~\cite{brown2020language,achiam2023gpt}. Distillation-style approaches additionally constrain output distributions. DZGCoOp extends this line along two axes: the zero-shot prior is used not only as a feature-space anchor but also as a distribution-level teacher via a KL guidance loss, and the zero-shot confidence actively participates in inference-time branch fusion rather than only in training-time regularization.
\noindent\textbf{Weight averaging.}
Stochastic Weight Averaging (SWA)~\cite{izmailov2018averaging} averages checkpoints sampled in a late phase of training and finds flatter minima. Prompt learning, however, exhibits strong epoch-dependent behavior: models trained longer overfit the source classes and transfer worse~\cite{wang2025msgcoop}. Our epoch-weighted averaging (EWA) generalizes uniform averaging with a Gaussian weighting schedule whose center and width select the favorable phase per transfer regime, subsuming ``early-checkpoint'' selection without extra inference cost.
sec/3_method.tex
+120
View File
@@ -0,0 +1,120 @@
\section{Method}
\label{sec:method}
We propose DZGCoOp, a prompt learning framework that exploits the frozen CLIP's zero-shot capability to guide an asymmetric dual-branch multimodal prompt network. Fig.~\ref{fig:framework} illustrates the overall pipeline: a zero-shot guidance stage builds priors from the frozen encoders, a dual-branch learning stage extracts deep-prompted visual and textual features and fuses them per instance, and a loss-and-optimization stage aligns the fused prediction with both labels and zero-shot priors. Throughout training, only prompt-related parameters are optimized.
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\textwidth]{fig/dzg_arch.pdf}
\caption{Architecture of DZGCoOp. \textbf{Blue}: the zero-shot guidance module distills LLM-enriched semantic anchors, zero-shot visual features, a zero-shot probability distribution, and an instance-level confidence $\alpha$ from the frozen CLIP encoders. \textbf{Green}: the dual-branch multimodal learning module couples a conservative strong branch and an exploratory weak branch through shared deep prompt layers on both encoders and fuses their logits with $\alpha$. \textbf{Purple \& orange}: guidance losses and the epoch-weighted averaging (EWA) strategy. Only prompt parameters are updated.}
\label{fig:framework}
\end{figure*}
\subsection{Preliminaries}
\label{sec:prelim}
CLIP consists of a vision encoder $\phi(\cdot)$ and a text encoder $\theta(\cdot)$ trained contrastively on web-scale image-text pairs. For a task with $N_c$ classes $\{\mathrm{cls}_1,\dots,\mathrm{cls}_{N_c}\}$, zero-shot classification encodes the template ``a photo of a [CLASS]'' into text features $\mathbf{w}_i^{\mathrm{txt}}$ and predicts via cosine similarity between $\mathbf{w}^{\mathrm{img}}=\phi(I)$ and each $\mathbf{w}_i^{\mathrm{txt}}$. Prompt learning replaces the fixed template with learnable context vectors while keeping $\phi$ and $\theta$ frozen. Following MaPLe~\cite{khattak2023maple}, we insert prompts at both the input layer (\emph{shallow}) and intermediate layers (\emph{deep}) of both encoders.
\subsection{Zero-Shot Guidance Module}
\label{sec:zsg}
The zero-shot guidance module distills four priors from the frozen CLIP, which are consumed later by the fusion rule and the guidance losses.
\noindent\textbf{Class-level priors.}
Following MSGCoOp~\cite{wang2025msgcoop}, a large language model~\cite{achiam2023gpt} generates multiple visually grounded descriptions of each class covering appearance, color, shape, and texture. The descriptions of class $\mathrm{cls}_i$ form a set $D_i$; encoding them with the frozen text encoder and averaging yields the \emph{semantic anchor}
\begin{equation}
\mathbf{w}^{\mathrm{zst}}_i = \frac{1}{|D_i|} \sum_{d \in D_i} \theta(d),
\label{eq:anchor}
\end{equation}
a consensus representation of the class in CLIP's semantic space. Since anchors are instance-independent, they are precomputed once before training.
\noindent\textbf{Instance-level priors.}
For an input image $I$, the frozen vision encoder yields the zero-shot visual feature $\mathbf{w}^{\mathrm{zsv}} = \phi(I)$. Cosine similarities against all anchors form the zero-shot logits $l^{\mathrm{zs}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{zsv}}, \mathbf{w}^{\mathrm{zst}}_i)$, which after softmax with temperature $\tau$ give the zero-shot probability distribution
\begin{equation}
p^{\mathrm{zs}}_i = \frac{\exp(l^{\mathrm{zs}}_i / \tau)}{\sum_{j=1}^{N_c} \exp(l^{\mathrm{zs}}_j / \tau)}.
\label{eq:pzs}
\end{equation}
The \emph{instance-level confidence} is the maximum zero-shot probability,
\begin{equation}
\alpha = \max_{i}\, p^{\mathrm{zs}}_i \in [1/N_c,\, 1],
\label{eq:alpha}
\end{equation}
which measures how deterministically the frozen CLIP ``knows'' this sample: $\alpha\!\to\!1$ indicates a confident zero-shot prediction, whereas $\alpha\!\to\!1/N_c$ indicates a near-uniform distribution. On large-scale benchmarks such as ImageNet, the average confidence on correct classes lies around $0.6$--$0.8$, making $\alpha$ a reliable per-sample signal. All computations in this module use frozen parameters and incur no gradient cost.
\subsection{Dual-Branch Multimodal Learning Module}
\label{sec:dual}
\begin{figure}[t]
\centering
\includegraphics[width=0.92\linewidth]{fig/dzg_fusion.pdf}
\caption{Zero-shot-confidence-guided dynamic fusion. The instance-level confidence $\alpha$ extracted from the frozen CLIP adaptively balances the conservative strong branch and the exploratory weak branch.}
\label{fig:fusion}
\end{figure}
The learnable part of DZGCoOp is an asymmetric dual-branch network with deep cross-modal prompting.
\noindent\textbf{Deep cross-modal prompting.}
On the vision side, shallow visual prompts $\mathbf{v}^{\mathrm{vis}}$ are prepended to the patch tokens, and deep visual prompts $\text{prompt}_{\text{layer}_i}^{\mathrm{vis}}$ are injected into intermediate transformer layers, yielding a prompted encoder $\tilde{\phi}(\cdot;\cdot)$ whose backbone stays frozen. On the text side, shallow contexts $\mathbf{v}^{\mathrm{str}}$ and $\mathbf{v}^{\mathrm{weak}}$ feed the strong and weak branches respectively, while the deep text prompts $\text{prompt}_{\text{layer}_i}^{\mathrm{text}}$ are \emph{shared} between the two branches. The visual pathway is single and shared: the visual feature $\mathbf{w}^{\mathrm{vis}} = \tilde{\phi}(I; V^{\mathrm{vis}})$ serves both branches.
\noindent\textbf{Asymmetric branch design.}
The strong branch concatenates $\mathbf{v}^{\mathrm{str}}$ (initialized from the hand-crafted template ``a photo of a'') with the class token and encodes it as $\mathbf{w}^{\mathrm{str}}_i = \tilde{\theta}(\mathrm{cls}_i; V^{\mathrm{str}})$; the weak branch uses randomly initialized $\mathbf{v}^{\mathrm{weak}}$ and yields $\mathbf{w}^{\mathrm{weak}}_i = \tilde{\theta}(\mathrm{cls}_i; V^{\mathrm{weak}})$. The strong branch thus starts aligned with pre-trained knowledge and is trained under a strong anchoring constraint, so it \emph{preserves} zero-shot generalization; the weak branch starts from a random point under a weak constraint, so it \emph{explores} task-specific semantics that the template cannot express. Compared with implicit diversity regularization in multi-prompt ensembles~\cite{wang2025msgcoop}, this initialization-plus-guidance asymmetry assigns the two branches explicit and complementary roles.
\noindent\textbf{Confidence-guided dynamic fusion.}
Each branch scores the image against its class features,
\begin{equation}
l^{\mathrm{str}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{vis}}, \mathbf{w}^{\mathrm{str}}_i), \quad
l^{\mathrm{weak}}_i = \mathrm{sim}(\mathbf{w}^{\mathrm{vis}}, \mathbf{w}^{\mathrm{weak}}_i),
\end{equation}
and the final logit vector is an instance-level convex combination (Fig.~\ref{fig:fusion}):
\begin{equation}
\mathbf{l}^{\mathrm{fin}} = \alpha \cdot \mathbf{l}^{\mathrm{str}} + (1 - \alpha) \cdot \mathbf{l}^{\mathrm{weak}}.
\label{eq:fusion}
\end{equation}
When CLIP is confident about a sample, the strong branch dominates and pre-training knowledge is preserved; when CLIP is uncertain, the weak branch contributes more of its task-specific evidence. Unlike equal-weight averaging, Eq.~\eqref{eq:fusion} requires no extra parameters and adapts the fusion weights to every sample. The fused logits are normalized by softmax into the final distribution $\mathbf{p}^{\mathrm{fin}}$.
\subsection{Guidance Losses and Optimization}
\label{sec:loss}
\noindent\textbf{Objective.}
The total loss combines classification and two zero-shot guidance terms:
\begin{equation}
\mathcal{L} = \mathcal{L}_{\mathrm{ce}} + \mathcal{L}_{\mathrm{zfg}} + \mathcal{L}_{\mathrm{zpg}}.
\label{eq:total}
\end{equation}
$\mathcal{L}_{\mathrm{ce}}$ is the cross-entropy on the fused logits $\mathbf{l}^{\mathrm{fin}}$. Because $\mathbf{l}^{\mathrm{fin}}$ is the confidence-weighted combination of both branches, the classification gradient supervises them jointly.
\noindent\textbf{Zero-shot feature guidance.}
To prevent the prompts from drifting away from CLIP's representation space, we align learned features with the zero-shot features via $L_1$ distances:
\begin{equation}
\mathcal{L}_{\mathrm{zfg}} = \lambda_{\mathrm{zvg}} \mathcal{L}_{\mathrm{zvg}} + \lambda_{\mathrm{sg}}^{\mathrm{str}} \mathcal{L}_{\mathrm{sg}}^{\mathrm{str}} + \lambda_{\mathrm{sg}}^{\mathrm{weak}} \mathcal{L}_{\mathrm{sg}}^{\mathrm{weak}},
\label{eq:zfg}
\end{equation}
\begin{equation}
\mathcal{L}_{\mathrm{zvg}} = \frac{1}{B} \sum_{j=1}^{B} \|\mathbf{w}_j^{\mathrm{vis}} - \mathbf{w}_j^{\mathrm{zsv}}\|_1, \quad
\mathcal{L}_{\mathrm{sg}}^{\mathrm{b}} = \frac{1}{N_c} \sum_{i=1}^{N_c} \|\mathbf{w}_i^{\mathrm{b}} - \mathbf{w}_i^{\mathrm{zst}}\|_1,
\end{equation}
where $B$ is the batch size and $\mathrm{b}\in\{\mathrm{str},\mathrm{weak}\}$. $L_1$ constrains feature \emph{positions} dimension-wise rather than only directions. Mirroring the asymmetric design, the guidance weights satisfy $\lambda_{\mathrm{sg}}^{\mathrm{str}} > \lambda_{\mathrm{sg}}^{\mathrm{weak}}$ (with $\lambda_{\mathrm{zvg}}$ and $\lambda_{\mathrm{sg}}^{\mathrm{weak}}$ following the mature configuration of~\cite{yao2023visual,wang2025msgcoop}), so the strong branch is anchored tightly while the weak branch keeps freedom to explore.
\noindent\textbf{Zero-shot probability guidance.}
At the distribution level, the zero-shot probabilities $\mathbf{p}^{\mathrm{zs}}$ act as a soft teacher through a KL divergence~\cite{kullback1951information}:
\begin{equation}
\mathcal{L}_{\mathrm{zpg}} = \frac{1}{B \cdot N_c} \sum_{j=1}^{B} \sum_{i=1}^{N_c} p_{j,i}^{\mathrm{zs}} \log\frac{p_{j,i}^{\mathrm{zs}}}{p_{j,i}^{\mathrm{fin}}}.
\label{eq:zpg}
\end{equation}
Beyond one-hot supervision, $\mathbf{p}^{\mathrm{zs}}$ encodes CLIP's inter-class structure (semantically close classes receive similar probabilities), which regularizes few-shot training against overfitting and retains the ability to predict unseen classes.
\noindent\textbf{Epoch-weighted averaging.}
Prompt learners conventionally use the last-epoch parameters, which are sensitive to stochastic fluctuation. As illustrated in Fig.~\ref{fig:ewa}, we snapshot the learnable parameters $\xi_t$ at every epoch and fuse them with a Gaussian schedule after training:
\begin{equation}
\xi_{\mathrm{final}} = \sum_{t=1}^{T} \epsilon_t \cdot \xi_t, \quad
\epsilon_t \propto \exp\!\left(-\frac{(t - \mu)^2}{2\sigma^2}\right),
\label{eq:ewa}
\end{equation}
where $T$ is the number of epochs and $(\mu,\sigma)$ place and shape the weighting window. For base-to-novel generalization, later snapshots are of higher quality, so the window centers on late epochs ($\mu{=}15$, $\sigma{=}1$); for cross-domain and cross-dataset transfer, early snapshots retain more generic pre-trained semantics, so the window shifts early and widens ($\mu{=}6$, $\sigma{=}10$). Unlike SWA~\cite{izmailov2018averaging}, which uniformly averages late-phase checkpoints and therefore hurts transfer when the model overfits the source dataset, EWA explicitly selects the favorable phase per task family. Inference uses $\xi_{\mathrm{final}}$ directly at no additional cost.
\begin{figure}[t]
\centering
\includegraphics[width=0.92\linewidth]{fig/dzg_ewa.pdf}
\caption{Epoch-weighted averaging (EWA). Snapshots along the training trajectory are fused with a Gaussian weight schedule whose center $\mu$ and width $\sigma$ adapt to the transfer regime.}
\label{fig:ewa}
\end{figure}
sec/4_experiments.tex
+206
View File
@@ -0,0 +1,206 @@
\section{Experiments}
\label{sec:exp}
\subsection{Setup}
\label{sec:setup}
\noindent\textbf{Benchmarks and protocols.}
We evaluate three transfer protocols.
\emph{(i) Base-to-novel generalization}: following the standard splits, 11 datasets (ImageNet~\cite{deng2009imagenet}, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) are divided into base classes for few-shot training and novel classes for evaluation, reporting base accuracy (B), novel accuracy (N), and their harmonic mean (HM).
\emph{(ii) Cross-domain transfer}: the model trained on the 1000 classes of ImageNet is evaluated on ImageNetV2~\cite{recht2019imagenet}, ImageNet-Sketch~\cite{wang2019learning}, ImageNet-A~\cite{hendrycks2021natural}, and ImageNet-R~\cite{hendrycks2021many}.
\emph{(iii) Cross-dataset transfer}: the ImageNet-trained model is directly evaluated on the other 10 datasets.
\noindent\textbf{Implementation details.}
We use CLIP ViT-B/16 as the backbone. All models are trained in the 16-shot setting with batch size 4 using SGD (learning rate 0.0025) for 20 epochs with cosine annealing. Shallow visual and textual prompts have 4 tokens each. Deep prompt depths follow the recommended configuration of MaPLe~\cite{khattak2023maple}: 9 layers for base-to-novel and 3 layers for cross-domain and cross-dataset transfer. Guidance weights are $\lambda_{\mathrm{zvg}}{=}8$ and $\lambda_{\mathrm{sg}}^{\mathrm{weak}}{=}8$, inherited from~\cite{yao2023visual,wang2025msgcoop}, and $\lambda_{\mathrm{sg}}^{\mathrm{str}}{=}24$ (3$\times$ the weak weight) selected by grid search to realize the conservative-vs-exploratory asymmetry. EWA uses $(\mu,\sigma){=}(15,1)$ for base-to-novel and $(6,10)$ otherwise. All results are averaged over 3 random seeds.
\begin{table*}[t]
\centering
\caption{\textbf{Base-to-novel generalization on 11 benchmarks (16-shot).}
\textbf{Bold} marks the best result per column.
\textcolor{MidnightBlue}{Blue} (resp. \textcolor{Bittersweet}{red}) numbers show absolute gains (resp. drops) of DZGCoOp over MaPLe.}
\label{tab:b2n}
\renewcommand{\arraystretch}{1.15}
\addtolength{\tabcolsep}{-2pt}
\scalebox{0.72}{
\begin{tabular}{l ccc ccc ccc ccc ccc ccc ccc}
\toprule
\textbf{Method} & \multicolumn{3}{c}{Average} & \multicolumn{3}{c}{ImageNet} & \multicolumn{3}{c}{Caltech101} & \multicolumn{3}{c}{OxfordPets} & \multicolumn{3}{c}{StanfordCars} & \multicolumn{3}{c}{Flowers102} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}\cmidrule(lr){14-16}\cmidrule(lr){17-19}
& B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM \\
\midrule
CLIP & 69.34 & 74.22 & 71.70 & 72.43 & 68.14 & 70.22 & 96.84 & 94.00 & 95.40 & 91.17 & 97.26 & 94.12 & 63.37 & 74.89 & 68.65 & 72.08 & \textbf{77.80} & 74.83 \\
CoOp~\cite{zhou2022learning} & 82.69 & 63.22 & 71.66 & 76.47 & 67.88 & 71.92 & 98.00 & 89.81 & 93.73 & 93.67 & 95.29 & 94.47 & 78.12 & 60.40 & 68.13 & 97.60 & 59.67 & 74.06 \\
CoCoOp~\cite{zhou2022conditional} & 80.47 & 71.69 & 75.83 & 75.98 & 70.43 & 73.10 & 97.96 & 93.81 & 95.84 & 95.20 & 97.69 & 96.43 & 70.49 & 73.59 & 72.01 & 94.87 & 71.75 & 81.71 \\
AAPL~\cite{kim2024aapl} & 80.27 & 72.17 & 76.01 & 76.53 & 70.57 & 73.43 & 97.87 & 95.10 & 96.46 & 95.63 & 97.40 & 96.51 & 70.33 & 73.50 & 71.88 & 95.10 & 70.63 & 81.06 \\
KgCoOp~\cite{yao2023visual} & 80.73 & 73.36 & 77.00 & 75.83 & 69.96 & 72.78 & 97.72 & 94.39 & 96.03 & 94.65 & 97.76 & 96.18 & 71.76 & \textbf{75.04} & 73.36 & 95.00 & 74.73 & 83.65 \\
MSGCoOp~\cite{wang2025msgcoop} & 81.40 & 75.05 & 78.10 & 76.46 & 70.45 & 73.31 & 98.08 & \textbf{95.81} & \textbf{96.91} & 95.62 & \textbf{97.89} & \textbf{96.70} & 71.22 & 74.56 & 72.85 & 96.20 & 75.91 & 84.84 \\
MaPLe~\cite{khattak2023maple} & 82.28 & 75.14 & 78.55 & 76.66 & 70.54 & 73.47 & 97.74 & 94.36 & 96.02 & 95.43 & 97.76 & 96.58 & 72.94 & 74.00 & 73.47 & 95.92 & 72.46 & 82.56 \\
\rowcolor{tabhighlight}
DZGCoOp & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} & \textbf{77.80} & \textbf{70.74} & \textbf{74.11} & \textbf{98.30} & 95.05 & 96.65 & \textbf{95.67} & 97.61 & 96.63 & \textbf{78.35} & 74.85 & \textbf{76.56} & \textbf{97.69} & 77.14 & \textbf{86.21} \\
& \textcolor{MidnightBlue}{{+2.24}} & \textcolor{MidnightBlue}{{+1.40}} & \textcolor{MidnightBlue}{{+1.78}} & \textcolor{MidnightBlue}{{+1.14}} & \textcolor{MidnightBlue}{{+0.20}} & \textcolor{MidnightBlue}{{+0.64}} & \textcolor{MidnightBlue}{{+0.56}} & \textcolor{MidnightBlue}{{+0.69}} & \textcolor{MidnightBlue}{{+0.63}} & \textcolor{MidnightBlue}{{+0.24}} & \textcolor{Bittersweet}{{-0.15}} & \textcolor{MidnightBlue}{{+0.05}} & \textcolor{MidnightBlue}{{+5.41}} & \textcolor{MidnightBlue}{{+0.85}} & \textcolor{MidnightBlue}{{+3.09}} & \textcolor{MidnightBlue}{{+1.77}} & \textcolor{MidnightBlue}{{+4.68}} & \textcolor{MidnightBlue}{{+3.65}} \\
\bottomrule
\end{tabular}
}
\bigskip
\scalebox{0.72}{
\begin{tabular}{l ccc ccc ccc ccc ccc ccc ccc}
\toprule
\textbf{Method} & \multicolumn{3}{c}{Food101} & \multicolumn{3}{c}{FGVCAircraft} & \multicolumn{3}{c}{SUN397} & \multicolumn{3}{c}{DTD} & \multicolumn{3}{c}{EuroSAT} & \multicolumn{3}{c}{UCF101} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}\cmidrule(lr){14-16}\cmidrule(lr){17-19}
& B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM & B & N & HM \\
\midrule
CLIP & 90.10 & 91.22 & 90.66 & 27.19 & 36.29 & 31.09 & 69.36 & 75.35 & 72.23 & 53.24 & 59.90 & 56.37 & 56.48 & 64.05 & 60.03 & 70.53 & 77.50 & 73.85 \\
CoOp~\cite{zhou2022learning} & 88.33 & 82.26 & 85.19 & 40.44 & 22.30 & 28.75 & 80.60 & 65.89 & 72.51 & 79.44 & 41.18 & 54.24 & 92.19 & 54.74 & 68.69 & 84.69 & 56.05 & 67.46 \\
CoCoOp~\cite{zhou2022conditional} & 90.70 & 91.29 & 90.99 & 33.41 & 23.71 & 27.74 & 79.74 & 76.86 & 78.27 & 77.01 & 56.00 & 64.85 & 87.49 & 60.04 & 71.21 & 82.33 & 73.45 & 77.64 \\
AAPL~\cite{kim2024aapl} & 90.70 & 91.60 & 91.15 & 34.07 & 24.17 & 28.28 & 79.65 & 76.90 & 78.25 & 73.90 & 53.43 & 62.02 & 87.00 & 66.30 & 75.25 & 82.20 & 74.27 & 78.03 \\
KgCoOp~\cite{yao2023visual} & 90.50 & 91.70 & 91.09 & 36.21 & 33.55 & 34.83 & 80.29 & 76.53 & 78.36 & 77.55 & 54.99 & 64.35 & 85.64 & 64.34 & 73.48 & 82.89 & 76.67 & 79.65 \\
MSGCoOp~\cite{wang2025msgcoop} & \textbf{90.72} & 91.62 & 91.18 & 36.73 & 34.57 & 35.62 & 81.38 & 76.20 & 78.71 & 79.32 & 56.64 & 66.09 & 86.29 & 74.97 & 80.23 & 83.52 & 77.05 & 80.16 \\
MaPLe~\cite{khattak2023maple} & 90.71 & \textbf{92.05} & \textbf{91.38} & 37.44 & 35.61 & 36.50 & 80.82 & 78.70 & 79.75 & 80.36 & 59.18 & 68.16 & \textbf{94.07} & 73.23 & 82.35 & 83.00 & 78.66 & 80.77 \\
\rowcolor{tabhighlight}
DZGCoOp & 90.54 & 91.52 & 91.03 & \textbf{43.48} & \textbf{37.07} & \textbf{40.02} & \textbf{82.62} & \textbf{79.01} & \textbf{80.77} & \textbf{83.72} & \textbf{60.02} & \textbf{69.92} & 94.05 & \textbf{78.35} & \textbf{85.49} & \textbf{87.49} & \textbf{80.51} & \textbf{83.85} \\
& \textcolor{Bittersweet}{{-0.17}} & \textcolor{Bittersweet}{{-0.53}} & \textcolor{Bittersweet}{{-0.35}} & \textcolor{MidnightBlue}{{+6.04}} & \textcolor{MidnightBlue}{{+1.46}} & \textcolor{MidnightBlue}{{+3.52}} & \textcolor{MidnightBlue}{{+1.80}} & \textcolor{MidnightBlue}{{+0.31}} & \textcolor{MidnightBlue}{{+1.02}} & \textcolor{MidnightBlue}{{+3.36}} & \textcolor{MidnightBlue}{{+0.84}} & \textcolor{MidnightBlue}{{+1.76}} & \textcolor{Bittersweet}{{-0.02}} & \textcolor{MidnightBlue}{{+5.12}} & \textcolor{MidnightBlue}{{+3.14}} & \textcolor{MidnightBlue}{{+4.49}} & \textcolor{MidnightBlue}{{+1.85}} & \textcolor{MidnightBlue}{{+3.08}} \\
\bottomrule
\end{tabular}
}
\end{table*}
\subsection{Base-to-Novel Generalization}
Table~\ref{tab:b2n} reports the 16-shot results on the 11 benchmarks. DZGCoOp achieves the best averages of 84.52\% (B), 76.54\% (N), and 80.33\% (HM), outperforming MaPLe by $+2.24$/$+1.40$/$+1.78$ and MSGCoOp by $+3.12$/$+1.49$/$+2.23$. It obtains the best HM on 8 of 11 datasets, with the largest gains on fine-grained and special-domain sets: FGVCAircraft ($+3.52$ HM over MaPLe), whose subtle inter-class differences benefit from the weak branch's exploratory features, Flowers102 ($+3.65$), StanfordCars ($+3.09$), EuroSAT ($+3.14$), and UCF101 ($+3.08$). Meanwhile, the strong branch keeps base accuracy high (best on 9 of 11 datasets), confirming that the asymmetric design improves novel-class generalization without sacrificing base-class fitness.
\begin{table}[t]
\centering
\caption{Cross-domain transfer with 16-shot setting. The model is trained on ImageNet and evaluated on four shifted target domains.}
\label{tab:xdo}
\small
\addtolength{\tabcolsep}{-2.pt}
\renewcommand{\arraystretch}{1.06}
\scalebox{0.90}{
\begin{tabular}{lcccccc}
\toprule
& \multicolumn{1}{c}{Source} & \multicolumn{5}{c}{Target} \\ \cline{2-7}
& ImageNet & V2 & Sketch & A & R & Avg. \\
\midrule
CLIP & 66.73 & 60.83 & 46.15 & 47.77 & 73.96 & 57.17 \\
CoOp & 71.51 & 64.20 & 47.99 & 49.71 & 75.21 & 59.28 \\
CoCoOp & 71.02 & 64.07 & 48.75 & 50.63 & 76.18 & 59.90 \\
AAPL & 71.37 & 64.20 & 48.80 & 50.60 & 76.87 & 60.12 \\
KgCoOp & 71.20 & 64.10 & 48.97 & 50.69 & 76.70 & 60.11 \\
MSGCoOp & 71.59 & 64.54 & 49.24 & 50.90 & 76.94 & 60.41 \\
MaPLe & 70.72 & 64.07 & 49.15 & 50.90 & 76.98 & 60.27 \\
\rowcolor{tabhighlight}
DZGCoOp & \textbf{71.66} & \textbf{64.77} & \textbf{49.43} & \textbf{51.37} & \textbf{77.08} & \textbf{60.66} \\
\bottomrule
\end{tabular}}
\end{table}
\subsection{Cross-Domain Transfer}
Table~\ref{tab:xdo} shows robustness to distribution shift. DZGCoOp ranks first on \emph{all four} target domains and on average (60.66\%, $+0.25$ over MSGCoOp, $+0.39$ over MaPLe). The gain is most pronounced on the adversarial ImageNet-A ($51.37\%$, $+0.47$), where the zero-shot guidance module supplies stable priors for samples far from the training distribution. We attribute the overall robustness to two designs: the frozen zero-shot branch remains a reliable reference under shift, and the EWA schedule configured for transfer ($\mu{=}6$, $\sigma{=}10$) keeps early, more generic snapshots in the fused parameters, mitigating overfitting to the source domain.
\begin{table}[t]
\centering
\caption{Cross-dataset transfer with 16-shot setting. The model is trained on ImageNet and directly evaluated on 10 downstream datasets.}
\label{tab:xda}
\tabstyle{4pt}
\scalebox{0.71}{
\begin{tabular}{l cccccccc}
\toprule
\textbf{Dataset} & \rotbox{CLIP} & \rotbox{CoOp} & \rotbox{CoCoOp} & \rotbox{AAPL} & \rotbox{KgCoOp} & \rotbox{MSGCoOp} & \rotbox{MaPLe} & \rotbox{DZGCoOp} \\
\midrule
ImageNet & 66.72 & 71.51 & 71.02 & 71.37 & 70.66 & 71.59 & 70.72 & \cellcolor{tabhighlight}\textbf{71.66} \\
Caltech101 & 92.94 & 93.70 & 94.43 & 94.17 & 93.92 & 93.29 & 93.53 & \cellcolor{tabhighlight}\textbf{94.69} \\
OxfordPets & 89.07 & 89.14 & 90.14 & \textbf{90.73} & 89.83 & 87.90 & 90.49 & \cellcolor{tabhighlight}90.20 \\
StanfordCars & 65.29 & 64.51 & 65.32 & 65.10 & 65.41 & 65.10 & \textbf{65.57} & \cellcolor{tabhighlight}65.51 \\
Flowers102 & 71.30 & 68.71 & 71.88 & 71.67 & 70.01 & 69.87 & \textbf{72.23} & \cellcolor{tabhighlight}71.00 \\
Food101 & 86.11 & 85.30 & 86.06 & 86.00 & \textbf{86.36} & 85.60 & 86.20 & \cellcolor{tabhighlight}\textbf{86.36} \\
Aircraft & \textbf{24.87} & 18.47 & 22.94 & 23.03 & 22.51 & 19.14 & 24.74 & \cellcolor{tabhighlight}22.89 \\
SUN397 & 62.62 & 64.15 & 67.36 & 66.80 & 66.16 & 65.14 & 67.01 & \cellcolor{tabhighlight}\textbf{68.02} \\
DTD & 44.56 & 41.92 & 45.73 & 44.80 & 46.35 & 42.59 & \textbf{46.49} & \cellcolor{tabhighlight}46.34 \\
EuroSAT & 47.69 & 46.39 & 45.37 & 41.83 & 46.04 & 45.59 & \textbf{48.06} & \cellcolor{tabhighlight}44.82 \\
UCF101 & 66.77 & 66.55 & 68.21 & \textbf{69.30} & 68.50 & 67.51 & 68.69 & \cellcolor{tabhighlight}\textbf{70.08} \\
\midrule
Average & 65.12 & 63.88 & 65.74 & 65.34 & 65.51 & 64.17 & \textbf{66.30} & \cellcolor{tabhighlight}65.99 \\
\bottomrule
\end{tabular}}
\end{table}
\subsection{Cross-Dataset Transfer}
Table~\ref{tab:xda} evaluates direct transfer of the ImageNet-trained model. DZGCoOp reaches 65.99\% on average, $+1.82$ over MSGCoOp and above CLIP, CoOp, CoCoOp, AAPL, and KgCoOp, though slightly below MaPLe ($-0.31$). It is the best on Caltech101 (94.69\%), SUN397 (68.02\%, $+1.01$ over MaPLe), and UCF101 (70.08\%, $+1.39$). The confidence-guided fusion helps most on datasets whose categories are well covered by CLIP's pre-training (e.g., scenes and actions), while the residual gap to MaPLe on EuroSAT and Aircraft reflects the difficulty of satellite and fine-grained domains far from ImageNet semantics, where the source-trained weak branch provides limited additional evidence.
\subsection{Ablation Studies}
\label{sec:ablation}
All ablations use the 16-shot base-to-novel setting averaged over the 11 datasets unless stated otherwise.
\begin{table}[t]
\centering
\caption{Ablation of the zero-shot guidance module and losses.}
\label{tab:abl_zg}
\small
\renewcommand{\arraystretch}{1.15}
\scalebox{0.82}{
\begin{tabular}{l ccc}
\toprule
Method & B & N & HM \\
\midrule
IVLP & 84.21 & 71.79 & 77.51 \\
+ dual-branch + avg. fusion & 84.33 & 72.10 & 77.74 \\
+ ZG losses & 84.38 & 76.38 & 80.18 \\
\rowcolor{tabhighlight}
+ dynamic fusion (full model) & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} \\
\bottomrule
\end{tabular}}
\end{table}
\noindent\textbf{Zero-shot guidance.}
Table~\ref{tab:abl_zg} starts from IVLP (independent visual-language prompting, \emph{i.e.}, MaPLe's prompting backbone without its coupling)~\cite{khattak2023maple}. Adding the asymmetric dual-branch with plain average fusion yields only $+0.23$ HM; the zero-shot guidance (ZG) losses are the key driver, lifting novel accuracy by $+4.28$ and HM by $+2.44$. Replacing average fusion with the confidence-guided dynamic fusion contributes a further $+0.15$ HM on top.
\begin{table}[t]
\centering
\caption{Ablation of the dual-branch design.}
\label{tab:abl_dual}
\small
\renewcommand{\arraystretch}{1.15}
\scalebox{0.83}{
\begin{tabular}{l ccc}
\toprule
Method & B & N & HM \\
\midrule
Single branch (weak only) & 83.59 & 75.10 & 79.12 \\
Single branch (strong only) & 84.31 & 76.27 & 80.09 \\
Symmetric dual (two strong) + avg. & \textbf{84.65} & 75.50 & 79.81 \\
Asymmetric dual (strong + weak) + avg. & 84.38 & 76.38 & 80.18 \\
\rowcolor{tabhighlight}
Full (asym. dual + dynamic fusion) & 84.52 & \textbf{76.54} & \textbf{80.33} \\
\bottomrule
\end{tabular}}
\end{table}
\noindent\textbf{Asymmetric dual-branch.}
Table~\ref{tab:abl_dual} dissects the branch structure. A single strong branch (80.09\% HM) beats a single weak branch (79.12\%), confirming the value of template initialization. Two \emph{strong} branches with average fusion reach the highest base accuracy (84.65\%) but a lower HM (79.81\%) than the asymmetric pair (80.18\%)---their predictions are highly correlated, so the ensemble degenerates into redundancy. The asymmetric strong-weak pairing improves novel accuracy by $+0.88$, and dynamic fusion adds another $+0.15$ HM, validating that the value of a second branch comes from complementary roles and instance-adaptive weighting rather than from ensembling per se.
\begin{table}[t]
\centering
\caption{Ablation of the parameter fusion strategy.}
\label{tab:abl_ewa}
\small
\renewcommand{\arraystretch}{1.15}
\scalebox{0.92}{
\begin{tabular}{l ccc}
\toprule
Method & B & N & HM \\
\midrule
Last-Epoch & 84.18 & 76.14 & 79.96 \\
SWA~\cite{izmailov2018averaging} & 84.35 & 76.28 & 80.11 \\
\rowcolor{tabhighlight}
EWA (ours) & \textbf{84.52} & \textbf{76.54} & \textbf{80.33} \\
\bottomrule
\end{tabular}}
\end{table}
\noindent\textbf{Parameter fusion strategy.}
Table~\ref{tab:abl_ewa} compares the final-parameter selection. Uniform late averaging (SWA) gains $+0.15$ HM over the last epoch; EWA's Gaussian schedule centered at $\mu{=}15$ with $\sigma{=}1$ further reaches 80.33\% ($+0.37$ over last-epoch, $+0.22$ over SWA), showing that explicitly weighting the stable late phase smooths single-epoch fluctuation. In cross-domain and cross-dataset regimes, the schedule shifts early ($\mu{=}6$, $\sigma{=}10$), which is what preserves generic semantics in Tables~\ref{tab:xdo} and~\ref{tab:xda}.
sec/5_conclusion.tex
+6
View File
@@ -0,0 +1,6 @@
\section{Conclusion}
\label{sec:conclusion}
We presented DZGCoOp, a zero-shot guided dual-branch prompt learning framework for few-shot image classification. DZGCoOp distills LLM-enriched semantic anchors, zero-shot visual features, probability distributions, and an instance-level confidence from the frozen CLIP; learns an asymmetric strong-weak dual-branch network coupled through shared deep prompt layers; fuses the branches per instance with the zero-shot confidence; and regularizes optimization with feature- and distribution-level guidance losses plus an epoch-weighted averaging strategy. Experiments on base-to-novel, cross-domain, and cross-dataset transfer demonstrate consistent improvements over strong prompt learning baselines, and ablations quantify the contribution of each component. We believe that exploiting a frozen VLM's own predictive confidence to route between conservative and exploratory adaptation paths is a general recipe for knowledge-preserving few-shot learning, and we leave its extension to denser vision-language tasks as future work.
\noindent\textbf{Acknowledgments.} This work was supported by the National Natural Science Foundation of China (No. 62306320) and the Natural Science Foundation of Jiangsu Province (No. BK20231063).