[Balancing strategy] How can I disable the balancing strategy for an AutoSklearnClassifier?

[johnantonn]

How can I disable the balancing strategy parameter for an AutoSklearnClassifier?

I am experimenting with CASH optimization using AutoSklearn. I use a AutoSklearnClassifier instance and I have disabled feature preprocessing and data preprocessing components following the provided examples, like so:

include={
'classifier': classifiers,
'feature_preprocessor': ['no_preprocessing'],
'data_preprocessor': ['NoPreprocessing']
}

However, I noticed in cv_results that there is an additional balancing strategy hyperparameter that results in double the evaluations without affecting the final metrics. More specifically, it takes two values:

• 'balancing:strategy': 'none'
• 'balancing:strategy': 'weighted'

The parameter is referenced in the AutoSklearnClassifier API (screenshot provided below), but I am not quite sure how to disable it. Any ideas?

[eddiebergman]

Hi @johnantonn,

So unfortunately I do not have a good solution for you. My only suggestion if you really need to disable it is to delete this line: https://github.com/automl/auto-sklearn/blob/master/autosklearn/pipeline/classification.py#L311. I tested it and it seems to work.

As for the reason why, you can see that around the line, there are 3 other steps DataPreprocessorChoice, etc.... These all have a get_available_components which process the include and exclude lines while Balancing step does not. I have no idea how to disable this and we will have to wait until the main developer is back (Mid April).

I apologies for the poor answer, if you find a better solution, please do let me know so I can document it.

Best,
Eddie

[johnantonn]

Hi @eddiebergman,

Thanks for the prompt response, much appreciated.

Indeed not including the Balancing instance inside classification.py, as you suggested, worked. I also tried providing the argument strategy='none' and keeping Balancing in the pipeline steps list, but the results showed that both 'none' and 'weighted' balancing were used anyway.

I guess the code is missing a handler at this point, as you already noticed. I will update the question if I find more on this.

Best,
Giannis

[mfeurer]

Indeed, the solution by @eddiebergman is the only way to disable balancing that is possible right now, and there is a handler missing to do this programatically. We'll fix this together with #379 when we get there.