add swin_s and swin_b variants and improved swin_t

[jdsgomes]

Adding improved weights for swin_t and new weights for swin_s, and swin_b.

The new variants were trained with trivial augment, instead of rand augment because it is actually closer to the original implementation.

Trainning commands:

# swin_t
python -u run_with_submitit.py --timeout 3000 --ngpus 8 --nodes 1  --model swin_t --epochs 300 \
    --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  \
    --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr \
    --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 \
    --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 \
    --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler \
    --ra-reps 4  --val-resize-size 224 

# swin_s
python -u run_with_submitit.py --timeout 3000 --ngpus 8 --nodes 1  --model swin_s --epochs 300 \
    --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  \
    --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr \
    --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 \
    --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 \
    --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler \
    --ra-reps 4  --val-resize-size 224 

# swin_b
python -u run_with_submitit.py --timeout 3000 --ngpus 8 --nodes 1  --model swin_b --epochs 300 \
    --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0 \ 
    --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr \
    --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 \
    --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 \
    --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler \
    --ra-reps 4  --val-resize-size 224 

Test commands:

# swin_t
srun -p dev --cpus-per-task=96 -t 24:00:00 --gpus-per-node=1 torchrun --nproc_per_node=1 train.py \
    --model swin_t --test-only --resume $EXPERIMENTS_PATH/43599/model_299.pth --interpolation bicubic   \
    --val-resize-size 232
# Test:  Acc@1 81.474 Acc@5 95.776

# swin_s
srun -p dev --cpus-per-task=96 -t 24:00:00 --gpus-per-node=1 torchrun --nproc_per_node=1 train.py \
    --model swin_s --test-only --resume $EXPERIMENTS_PATH/43602/model_299.pth --interpolation bicubic   \
    --val-resize-size 246
# Test:  Acc@1 83.196 Acc@5 96.360

# swin_b
srun -p dev --cpus-per-task=96 -t 24:00:00 --gpus-per-node=1 torchrun --nproc_per_node=1 train.py \
    --model swin_b --test-only --resume $EXPERIMENTS_PATH/44985/model_299.pth --interpolation bicubic   \
    --val-resize-size 238
# Test:  Acc@1 83.582 Acc@5 96.640

cc @xiaohu2015 @datumbox

[jdsgomes]

original repo Acc@1 81.2 Acc@5 95.5

[jdsgomes]

original repo acc Acc@1 83.2 Acc@5 96.2

[jdsgomes]

original repo Acc@1 83.5 Acc@5 96.5

[datumbox]

@jdsgomes Can you provide proof of the validation stats for future reference?

Here is a recent PR for what I mean: #6019 (PR description)

[xiaohu2015]

@jdsgomes Good. It seems that you added some other tricks: ta_wide model-ema ra-sampler?

[datumbox]

@jdsgomes I've pushed a tiny change on doc strings to indicate its a "similar" recipe not the same. As @xiaohu2015 highlighted, we use a mix of the paper recipe with the standard recipe of TorchVision, so this will be more factual.

My intention was to merge this prior merging the PR that removes the old models.srt (to avoid conflicts) but it seems that your PR fails on a specific test (see this) which looks related. Could you please check it out?

[jdsgomes]

Thanks @datumbox for the change in the docstrings, and @xiaohu2015 you are right this is a slightly modified recipe, I should have highlighted that initially!

Looking in the error now

[datumbox]

LGTM, thanks @jdsgomes. Feel free to merge on Green-ish CI.

[xiaohu2015]

The SwinV2 offical code has released, maybe we can also add swin_v2

[datumbox]

@xiaohu2015 If you are up for it, by all means. I think it's possible to update this class to produce both Swin V1 and V2 models, similar to what we did with EfficientNets.