**HOW** do I actually use CutMix or Mixup without re-implementing?

[pGit1]

According to this: https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/ and this: #3911

I am supposed to be able to use mixup and cutmix. However when do torchvision.ANYTHING the implementations are nowhere to be found. Searching for cutmix or mixup in the docs yields no results. https://pytorch.org/vision/stable/search.html?q=cutmix&check_keywords=yes&area=default

What is the recommend approach to using these updates in our code? Is the expectation to copy/paste stuff implemented in references? :( Sorry for dumb question.

[pGit1]

Dumb question. Simple answer is I can do whatever I want except torchvision.transforms implementation does not exist.

[NicolasHug]

It's not a dumb question @pGit1 , thank you for asking.
MixUp and CutMix are special transforms in the sense that they need to operate on batches. And the main problem is that this makes them incompatible with a typical (torchvision) training scenario, where transforms operate on single samples instead (due to the way the DataLoader and the datasets interact).
That + the fact that those transforms need to transforms samples on top of images is the reason we have not been able to release those transforms as built-ins so far.

We have 2 existing implementations of these transforms:

• in our training references https://github.com/pytorch/vision/blob/main/references/classification/transforms.py. To use those, you'll need to copy/paste the code into your own script
• in our prototype area https://github.com/pytorch/vision/blob/main/torchvision/prototype/transforms/_augment.py. To use those, you'll need to either build from source or use the nightly release (these are part of our "v2" transforms, most of them will be broadly available as Beta in a few weeks in normal releases, but MixUp and CutMix stay in prototype for now).

In both cases, you'll need to use these transforms as a special collate function, something similar to what we do in our references. The fact that they have to be used as a collate function and not as a regular transform is the reason we're not making those broadly available just yet, but we're planning on working towards that. HTH!

[pGit1]

Hi @NicolasHug! I actually didnt expect anyone to respond to this. Your reasons all make sense to me.

The way I am ultimately going to implement this is within a dataset class in the __getitem__ method, similar to this implementation, but slightly different. https://github.com/ildoonet/cutmix/blob/master/cutmix/cutmix.py

For example:

class CutMix(Dataset):
    def __init__(self, dataset, num_class, num_mix=1, beta=1., prob=1.0):
        self.dataset = dataset
        self.num_class = num_class
        self.num_mix = num_mix
        self.beta = beta
        self.prob = prob

    def __getitem__(self, index):
        img, lb = self.dataset[index]
        lb_onehot = onehot(self.num_class, lb)

        for _ in range(self.num_mix):
            r = np.random.rand(1)
            if self.beta <= 0 or r > self.prob:
                continue

            # generate mixed sample
            lam = np.random.beta(self.beta, self.beta)
            rand_index = random.choice(range(len(self)))

            img2, lb2 = self.dataset[rand_index]
            lb2_onehot = onehot(self.num_class, lb2)

            bbx1, bby1, bbx2, bby2 = rand_bbox(img.size(), lam)
            img[:, bbx1:bbx2, bby1:bby2] = img2[:, bbx1:bbx2, bby1:bby2]
            lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (img.size()[-1] * img.size()[-2]))
            lb_onehot = lb_onehot * lam + lb2_onehot * (1. - lam)

        return img, lb_onehot

    def __len__(self):
        return len(self.dataset)

This way all my downstream code can stay the same and I can still keep the flexibility provided by dataset class. Does this seem reasonable to you?