Skip to content

Incorrect preprocessing for ImageNet-C evaluation #59

@imrahulr

Description

@imrahulr

I see that the ImageNet-C evaluation uses the preprocessing: Resize(256)+CenterCrop(224)+ToTensor().

def load_imagenetc(
n_examples: Optional[int] = 5000,
severity: int = 5,
data_dir: str = './data',
shuffle: bool = False,
corruptions: Sequence[str] = CORRUPTIONS,
prepr: str = 'Res256Crop224'
) -> Tuple[torch.Tensor, torch.Tensor]:
transforms_test = PREPROCESSINGS[prepr]

This causes discrepancies with the scores reported in the original papers (DeepAugment, AugMix, Standard RN-50). The ImageNet-C dataset already contains 224x224 images and hence only ToTensor() should be used for consistency.

Fixing prepr='none' in load_imagenetc should solve the issue (assuming all the models are capable of handling 224x224 images as input).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions