Autosklearn results are not reproducible

[rabsr]

I have executed 2 autosklearn jobs with the same dataset and kept all the configuration settings same including seed, time and memory. However, I could see that the pipelines executed are different for both the jobs. What could be the possible issue?

Is Autosklearn deterministic? If not, is it possible to achieve so?

[mfeurer]

Auto-sklearn is not deterministic due to two reasons:

• there is a time limit applied to each function evaluation. In case something runs close to the time limit (and has a 50/50 chance of being over) or the computer usage differs (which can change turbo rates of the processor) this can lead to different results. Disk usage might also have an effect here.
• the SMAC optimization algorithms is not deterministic

The latter can be changed by following this snippet, which makes Auto-sklearn 99% deterministic.

[rabsr]

Thanks @mfeurer for reverting on it quickly.

Totally agreed for the first case. Difference on status for runs close to time limit is noticed. How can disk usage impact the results?

In my use case, we are using following,
ensemble_size=1,
max_models_on_disc=1
n_jobs=1
and given time limits and memory limits are sufficient for runs to be completed.

In auto-sklearn, deterministic param for smac_scenario_args is already set to true and restricted for any change. Isn't it sufficient to make SMAC deterministic.
From the snippet, I could see lot of configs are changed. Can you help me with mandatory config(s) changes to make SMAC algos deterministic?

[rabsr]

I have found few preprocessing techniques to be non-deterministic. I have raised separate issue for the same.
#1170

[rabsr]

Fixed via #1170