Shape Optimized ViT Models

[pjjajal]

How were the shape parameters derived for the shape-optimal models in So150M? Were the scaling exponents from the Getting ViT in Shape paper used? The analysis for this would be helpful, I am trying to derive other shape-optimized models.

[rwightman]

SO150M was covered in the paper, I modified it to run faster on GPU as both SO150M and S400M are poor shapes for GPU kernels. I didn't really have much luck working with the exponents in the paper. For the timm sbb vit series I altered shapes a bit but just iterated in the ranges around existing models (since they were all on the smaller side), and found deeper and/or a bit narrower to be easier to train to better results on modest sized datasets.

[pjjajal]

Okay, thank you! Just to make sure I understand correctly, essentially you used the shapes of existing models as starting points and then searched "around" those shapes? Was narrower and deeper a common trend for models that performed well?

Also, Do you remember what the problem was when you were trying to use the exponents?