170 lines
7.1 KiB
Markdown
170 lines
7.1 KiB
Markdown
# PromptSRC Training
|
||
|
||
We provide bash scripts in [scripts/](../scripts) for training PromptSRC and independent V-L prompting baseline.
|
||
Make sure to update the `DATA` variable with dataset path in the script file and run the commands from the main directory `PromptSRC/`.
|
||
Below we provide training and testing instructions for PromptSRC. The same instructions are applicable for the baseline *independent V-L prompting* approach, MaPLe, CoOp and CoCoOp.
|
||
|
||
### Training time and compute
|
||
We train PromptSRC on each dataset with a batch size of 4 using a **single** NVIDIA A100 GPU.
|
||
Training PromptSRC on ImageNet for 20 epochs takes around 6 hours for a single seed. So results for 3 seeds takes around 18 hours. For all remaining 10 datasets, it combinedly takes around around 8 hours (for all 3 seeds) on a single A100 GPU.
|
||
|
||
## PromptSRC
|
||
|
||
#### (1) Base-to-Novel class generalization setting
|
||
The base-to-novel PromptSRC configuration is provided in config file at `configs/trainers/PromptSRC/vit_b16_c2_ep20_batch4_4+4ctx.yaml`. All hyper-parameters such as GPA STD, GPA Mean, SCL loss weights coefficients, prompt length and prompt depth etc., can be modified using this config file.
|
||
|
||
Run the commands below to train PromptSRC on ImageNet.
|
||
|
||
```bash
|
||
# Other possible dataset values includes [caltech101, food101, dtd, ucf101, oxford_flowers, oxford_pets, fgvc_aircraft, stanford_cars, sun397, eurosat]
|
||
|
||
# seed=1
|
||
# trains and evaluates on base classes
|
||
bash scripts/promptsrc/base2new_train.sh imagenet 1
|
||
# evaluates on novel classes
|
||
bash scripts/promptsrc/base2new_test.sh imagenet 1
|
||
|
||
# seed=2
|
||
# trains and evaluates on base classes
|
||
bash scripts/promptsrc/base2new_train.sh imagenet 2
|
||
# evaluates on novel classes
|
||
bash scripts/promptsrc/base2new_test.sh imagenet 2
|
||
|
||
# seed=3
|
||
# trains and evaluates on base classes
|
||
bash scripts/promptsrc/base2new_train.sh imagenet 3
|
||
# evaluates on novel classes
|
||
bash scripts/promptsrc/base2new_test.sh imagenet 3
|
||
```
|
||
|
||
#### Averaging results over 3 seeds:
|
||
Once the above trainings and evaluations are completed, the `output/` directory should have the following structure:
|
||
|
||
```
|
||
output
|
||
|–– base2new/
|
||
| |–– test_new/
|
||
| | |–– imagenet/
|
||
| | | |–– shots_16/
|
||
| | | | |–– PromptSRC/
|
||
| | | | | |–– vit_b16_c2_ep20_batch4_4+4ctx/
|
||
| | | | | | |–– seed1/
|
||
| | | | | | |–– seed2/
|
||
| | | | | | |–– seed3/
|
||
| |–– train_base/
|
||
| | |–– imagenet/
|
||
| | | |–– shots_16/
|
||
| | | | |–– PromptSRC/
|
||
| | | | | |–– vit_b16_c2_ep20_batch4_4+4ctx/
|
||
| | | | | | |–– seed1/
|
||
| | | | | | |–– seed2/
|
||
| | | | | | |–– seed3/
|
||
```
|
||
|
||
Now use the script `parse_test_res.py` and run the commands below to calculate the averaged results:
|
||
```bash
|
||
# prints averaged results for base classes
|
||
python output/base2new/train_base/imagenet/shots_16/PromptSRC/vit_b16_c2_ep20_batch4_4+4ctx --test-log
|
||
# averaged results for novel classes
|
||
python output/base2new/test_new/imagenet/shots_16/PromptSRC/vit_b16_c2_ep20_batch4_4+4ctx --test-log
|
||
```
|
||
|
||
The above steps can be repeated for other individual datasets.
|
||
|
||
#### (2) Cross-Dataset Transfer setting
|
||
We provide instructions to train PromptSRC on ImageNet using all 1000 classes with 16 shots and then evaluating it directly on new downstream datasets.
|
||
The corresponding cross-dataset config for PromptSRC is available at: `configs/trainers/PromptSRC/vit_b16_c2_ep20_batch4_4+4ctx_cross_datasets.yaml`. All PromptSRC hyper-parameters can be modified in this config file.
|
||
* Firstly, train PromptSRC on imagenet in few-shot manner (for all 3 seeds).
|
||
|
||
```bash
|
||
# seed=1
|
||
bash scripts/promptsrc/xd_train.sh imagenet 1
|
||
# seed=2
|
||
bash scripts/promptsrc/xd_train.sh imagenet 2
|
||
# seed=3
|
||
bash scripts/promptsrc/xd_train.sh imagenet 3
|
||
```
|
||
|
||
* Now directly evaluate the ImageNet trained model on downstream cross-datasets.
|
||
|
||
```bash
|
||
# Other possible dataset values includes [imagenet, food101, dtd, ucf101, oxford_flowers, fgvc_aircraft, sun397, eurosat]
|
||
|
||
for SEED in 1 2 3
|
||
do
|
||
bash scripts/promptsrc/xd_test.sh caltech101 ${SEED}
|
||
bash scripts/promptsrc/xd_test.sh oxford_pets ${SEED}
|
||
bash scripts/promptsrc/xd_test.sh stanford_cars ${SEED}
|
||
done
|
||
```
|
||
You can obtain averaged results by using the script `parse_test_res.py` and following the similar steps as provided in base-to-novel generalization experiments.
|
||
|
||
|
||
#### (3) Domain Generalization setting
|
||
We use the same ImageNet trained PromptSRC model for domain generalization experiments. The steps are similar to above cross-dataset experiments, however, the trained model is now evaluated on ImageNet variants.
|
||
The corresponding domain generalization config for PromptSRC is available at: `configs/trainers/PromptSRC/vit_b16_c2_ep20_batch4_4+4ctx_cross_datasets.yaml`.
|
||
* Evaluate ImageNet model on different variants of ImageNet (datasets with domain shifts).
|
||
|
||
```bash
|
||
for SEED in 1 2 3
|
||
do
|
||
bash scripts/promptsrc/xd_test.sh imagenetv2 ${SEED}
|
||
bash scripts/promptsrc/xd_test.sh imagenet_sketch ${SEED}
|
||
bash scripts/promptsrc/xd_test.sh imagenet_a ${SEED}
|
||
bash scripts/promptsrc/xd_test.sh imagenet_r ${SEED}
|
||
done
|
||
```
|
||
|
||
|
||
You can obtain averaged results by using the script `parse_test_res.py` and following the similar steps as provided in base-to-novel generalization experiments.
|
||
|
||
#### (4) Few-shot setting
|
||
In this setting, PromptSRC is trained on all classes individual datasets with different few-shot splits (K = 1, 2, 4, 8, 16). The corresponding few-shot setting config for PromptSRC is available at: `configs/trainers/PromptSRC/vit_b16_c2_ep50_batch4_4+4ctx_few_shot.yaml`.
|
||
|
||
Now use the training script `scripts/promptsrc/few_shot.sh` and run the commands below to calculate the results for imagenet dataset for all shots over 3 seeds:
|
||
|
||
```bash
|
||
# Other possible dataset values includes [caltech101, food101, dtd, ucf101, oxford_flowers, oxford_pets, fgvc_aircraft, stanford_cars, sun397, eurosat]
|
||
|
||
# train and test on given dataset for K=1 shot
|
||
bash scripts/promptsrc/few_shot.sh imagenet 1
|
||
# train and test on given dataset for K=2 shot
|
||
bash scripts/promptsrc/few_shot.sh imagenet 2
|
||
# train and test on given dataset for K=4 shot
|
||
bash scripts/promptsrc/few_shot.sh imagenet 4
|
||
# train and test on given dataset for K=8 shot
|
||
bash scripts/promptsrc/few_shot.sh imagenet 8
|
||
# train and test on given dataset for K=16 shot
|
||
bash scripts/promptsrc/few_shot.sh imagenet 16
|
||
```
|
||
|
||
|
||
You can obtain averaged results by using the script `parse_test_res.py` and following the similar steps as provided in base-to-novel generalization experiments.
|
||
<br>
|
||
|
||
|
||
#### Training and testing independent V-L prompting baseline approach
|
||
|
||
For training independent V-L prompting baseline approach, we provide their corresponding configs and scripts as follows.
|
||
|
||
```
|
||
configs
|
||
|–– datasets/
|
||
|–– trainers/
|
||
| |–– CoCoOp/
|
||
| |–– CoOp/
|
||
| |–– IVLP/
|
||
| |–– PromptSRC/
|
||
```
|
||
|
||
```
|
||
scripts
|
||
|–– cocoop/
|
||
|–– coop/
|
||
|–– promptsrc/
|
||
|–– independent-vlp/
|
||
```
|
||
|
||
Please use the corresponding config and script files and follow the same instructions as provided for PromptSRC for training and testing.
|
||
This repository also supports using official [MaPLe](MaPLe.md), [CoOp](CoOp.md) and [Co-CoOp](Co-CoOp.md) configs and models.
|