Skip to content

EfficientNet B1 to B7 and InceptionV3 Pre-Trained Models Accuracy Discrepancies #5958

Closed
@yqihao

Description

@yqihao

🐛 Describe the bug

I am performing accuracy validations for several pre-trained models with all 50k images in the ImageNet LSVRC-2012 Validation dataset. And I find that in some results the validation accuracy has a significant difference from the accuracy claimed by the documentation. Here are the data I collect and the plots.

Pytorch TOP-1 Pytorch TOP-1 Validation Pytorch TOP-1 Difference Pytorch TOP-5 Pytorch TOP-5 Validation Pytorch TOP-5 Difference
ResNet 50 76.13% 76.06% 0.07% 92.86% 92.67% 0.19%
ResNet 101 77.37% 77.30% 0.07% 93.55% 93.36% 0.18%
ResNet 152 78.31% 78.25% 0.06% 94.05% 93.85% 0.19%
EfficientNet B0 77.69% 77.63% 0.06% 93.53% 93.40% 0.13%
EfficientNet B1 78.64% 77.55% 1.10% 94.19% 93.40% 0.78%
EfficientNet B2 80.61% 77.72% 2.89% 95.31% 93.55% 1.76%
EfficientNet B3 82.01% 78.48% 3.53% 96.05% 94.17% 1.88%
EfficientNet B4 83.38% 79.21% 4.18% 96.59% 94.33% 2.27%
EfficientNet B5 83.44% 73.07% 10.37% 96.63% 90.76% 5.87%
EfficientNet B6 84.01% 74.34% 9.66% 96.92% 91.67% 5.25%
EfficientNet B7 84.12% 73.86% 10.26% 96.91% 91.38% 5.53%
Inception V3 77.29% 69.47% 7.82% 93.45% 88.48% 4.97%
MobileNetV2 71.88% 71.80% 0.07% 90.29% 90.10% 0.18%

The result shows that the accuracy validations for ResNet 50, 101, 152, and MobileNetV2 are almost the same as the accuracy claimed on the documentation, but the rest of them have different levels of discrepancy. The accuracy validation scripts are available here. https://github.com/yqihao/PTMValidations All validations are performed on Google Colab with the same ImageNet dataset(https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar).
chart (1)
chart

Versions

PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.4
Libc version: glibc-2.26

Python version: 3.7.13 (default, Apr 24 2022, 01:04:09) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: Tesla K80
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu113
[pip3] torchaudio==0.11.0+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.12.0
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions