DAPT/docs/CoOp.md

# Learning to Prompt for Vision-Language Models (CoOp, IJCV'22)
[![paper](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)](https://arxiv.org/abs/2109.01134)

We provide the scripts in [scripts/coop](../scripts/coop) to reproduce CoOp results (IJCV'22).

Make sure to configure the dataset paths in environment variable `DATA` and run the commands from the main directory `MaPLe/`.

## Few-Shot Learning

All you need is `scripts/coop/main.sh`, which contains six input arguments.

`DATASET` takes as input a dataset name, like `imagenet` or `caltech101`. The valid names are the files' names in `configs/datasets/`.

`CFG` means which config file to use, such as `rn50`, `rn101` or `vit_b32` (see `configs/trainers/coop/`). Note that for ImageNet, we use `configs/trainers/coop/*_ep50.yaml` for all settings (please follow the implementation details shown in the paper).

Below we provide examples on how to run CoOp on Caltech101.

**CLIP + CoOp (M=16, end)**:
- 1 shot: `bash scripts/coop/main.sh caltech101 rn50_ep50 end 16 1 False`
- 2 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 end 16 2 False`
- 4 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 end 16 4 False`
- 8 shots: `bash scripts/coop/main.sh caltech101 rn50 end 16 8 False`
- 16 shots: `bash scripts/coop/main.sh caltech101 rn50 end 16 16 False`

**CLIP + CoOp (M=16, mid)**:
- 1 shot: `bash scripts/coop/main.sh caltech101 rn50_ep50 middle 16 1 False`
- 2 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 middle 16 2 False`
- 4 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 middle 16 4 False`
- 8 shots: `bash scripts/coop/main.sh caltech101 rn50 middle 16 8 False`
- 16 shots: `bash scripts/coop/main.sh caltech101 rn50 middle 16 16 False`

**CLIP + CoOp (M=16, end, CSC)**:
- 1 shot: `bash scripts/coop/main.sh caltech101 rn50_ep50 end 16 1 True`
- 2 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 end 16 2 True`
- 4 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 end 16 4 True`
- 8 shots: `bash scripts/coop/main.sh caltech101 rn50 end 16 8 True`
- 16 shots: `bash scripts/coop/main.sh caltech101 rn50 end 16 16 True`

**CLIP + CoOp (M=16, mid, CSC)**:
- 1 shot: `bash scripts/coop/main.sh caltech101 rn50_ep50 middle 16 1 True`
- 2 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 middle 16 2 True`
- 4 shots: `bash scripts/coop/main.sh caltech101 rn50_ep100 middle 16 4 True`
- 8 shots: `bash scripts/coop/main.sh caltech101 rn50 middle 16 8 True`
- 16 shots: `bash scripts/coop/main.sh caltech101 rn50 middle 16 16 True`

After the experiments are finished, you can use `parse_test_res.py` to calculate the average results instead of manually looking into the log files. Say the structure of `output/` is

```
output
|–– caltech101/
|   |–– CoOp/
|   |   |–– rn50_16shots/
|   |   |   |–– nctx16_cscFalse_ctpend/
|   |   |   |   |–– seed1/
|   |   |   |   |–– seed2/
|   |   |   |   |–– seed3/
|   |   |–– rn50_8shots/
|   |   |   |–– nctx16_cscFalse_ctpend/
|   |   |   |   |–– seed1/
|   |   |   |   |–– seed2/
|   |   |   |   |–– seed3/
```

To calculate the average results for the folder `rn50_16shots/nctx16_cscFalse_ctpend/`, you can run

```bash
python parse_test_res.py output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend
```

Then, you will see something like this in your terminal

```bash
Parsing files in output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend
file: output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend/seed1/log.txt. accuracy: 91.81%. error: 8.19%.
file: output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend/seed2/log.txt. accuracy: 92.01%. error: 7.99%.
file: output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend/seed3/log.txt. accuracy: 92.17%. error: 7.83%.
===
Summary of directory: output/caltech101/CoOp/rn50_16shots/nctx16_cscFalse_ctpend
* accuracy: 92.00% +- 0.15%
* error: 8.00% +- 0.15%
===
```

**How to initialize the context tokens with pre-trained word vectors?** Specify the words for the parameter `TRAINER.COOP.CTX_INIT` in your config file. In our paper, we use `configs/trainers/rn50_ctxv1.yaml` (give this file to `--config-file`, see `scripts/coop/main.sh`), which uses "a photo of a" as the initialization words.

**How to visualize nearest words for the learned context tokens?** All you need is `interpret_prompt.py`. Say the learned tokens are saved in `a/b/c/prompt_learner/model.pth.tar` and you would like to see the top-3 nearest words for each token. In this case, run `python interpret_prompt.py a/b/c/prompt_learner/model.pth.tar 3`

## Robustness to Distribution Shift
To reproduce the robustness experiments, you can simply load the models learned on ImageNet and evaluate them on the following datasets: `imagenetv2`, `imagenet-sketch`, `imagenet-a` and `imagenet-r`.

The command is provided in `scripts/coop/eval.sh`. The key arguments are `--model-dir`, `--load-epoch` and `--eval-only`. `--model-dir` indicates the directory where the models are saved (i.e. the entire folder containing `log.txt`, the tensorboard file and `prompt_learner/`). `--load-epoch` tells the code to load the model saved at a specific epoch, like `--load-epoch 50` for ImageNet (see the [source code](https://github.com/KaiyangZhou/Dassl.pytorch/blob/master/dassl/engine/trainer.py#L169) for more details).

For example, to evaluate `CLIP + CoOp (M=16, end)` on ImageNetV2, you can do

```bash
# Don't need to use rn5_ep50 here as no training is performed
bash scripts/coop/eval.sh imagenetv2 rn50
```

The default setting is `SHOTS=16`. Feel free to modify the script.

Again, you can use `parse_test_res.py` to automate the calculation of average performance. This time you should append `--test-log`, e.g., `python parse_test_res.py directory --test-log`.

## Zero-Shot CLIP

See `**scripts/zsclip/zeroshot.sh**`.

## Linear Probe CLIP

Please move to [lpclip/](lpclip/).