[Question] Integration with sklearn-evaluation

[idomic]

I think adding an integration/tutorial and the right documentation of the framework can go a long way.

Usually when using auto-sklearn you'd get the final model/estimator, and then you'll check performance and results.
This will allow an easier mechanism for the users to solve the second part of it.

[aron-bram]

Hi,

thank you for your suggestion. I will look into sklearn-evaluation and see how it could be combined with the results of our optimization runs. Hopefully small discrepancies between the implementations of some things like "cv_results_" in sklearn and auto-sklearn won't cause too much of a problem. Otherwise this could take a long time.
As a side note, we are currently in the process of updating our sklearn dependency to one of the more recent versions.

Then I could add a tutorial under the examples folder about how to find the optimal pipeline for some problem with auto-sklearn and analyze that pipeline using sklearn-evaluation. And for instance, one about analyzing the explored hyperparameter space using cv_results_.
Is this what you propose?

What do you think @eddiebergman?

[idomic]

Is this what you propose?

Yes exactly!
For instance, for hyperparameter analysis, we can take the grid approach and evaluate multiple parameters at once.
I'm also happy to help and collaborate on the PR. Once we have a solid example or two we can also blog about it.
Probably the first step is like you're suggesting, testing the compatibility of some parameters that are different between the two like cv_results_

[aron-bram]

Great! Thank you for the confirmation and for your willingness to collaborate.

I will let you know once I've completed the preliminary steps.