How to get the classifier with parameters and results of the best models?

[shabir1]

How to get the classifier with parameters and results of the best models?
I set ensemble_size=10 and run the auto sklearn for 1 hour, model.cv_results_ contains 243 rows, 240 shows success status and 3 shows timeout. but when I print model.show_models() it prints only 3 models. How would I know which are those 3 models in cv_results_. I tried to sort cv_results_ on mean_test_score key but the top 3 configs of cv_results__ are not matching with show_models() results. And If I set ensemble_size=10 why I got only 3 models from show_models().
Below are the top 5 configs when sort cv_results_ on mean_test_score.
adaboost is not present in show_models()

mean_test_score	mean_fit_time	param_classifier:choice	rank_test_scores	status	budgets	param_balancing:strategy
0.898388227049755	3.68728971481323	extra_trees	1	Success	0	weighting
0.896986685353889	6.91696286201477	adaboost	2	Success	0	none
0.894183601962158	3.38019490242004	extra_trees	3	Success	0	none
0.893482831114226	3.80625748634338	extra_trees	4	Success	0	weighting
0.893482831114226	3.25829768180847	extra_trees	4	Success	0	none

I have to parse show_models() result manually for getting selected classifier with their parameters or there is some way to parse it?

[mfeurer]

And If I set ensemble_size=10 why I got only 3 models from show_models()

We use ensemble selection with repetitions, that means a model can be added multiple times to the ensemble, giving it a higher weight. ensemble_size=10 means that there are 10 slots, and it turns out they are filled with 3 models in your case. Also, the ensemble does not contain the best configurations, but is constructed to optimize the given loss function on a holdout set, thus it can leave out the 3rd best model.

How would I know which are those 3 models in cv_results_?

Unfortunately, this is not possible yet in an automated fashion and you would have to compare the hyperparameters by hand. It would indeed be great to have this information available as well.

Please keep this open to remind us to add a way to automatically associate configurations in the ensemble with configurations in cv_results_.

[shabir1]

@mfeurer Thank you for your response,
How to get the classifier with parameters and results of the best models?
I have to parse show_models() or get_models_with_weight() result manually for getting selected classifier with their parameters or there is some way to parse it?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs for the next 7 days. Thank you for your contributions.

[mfeurer]

I have to parse show_models() or get_models_with_weight() result manually for getting selected classifier with their parameters or there is some way to parse it?

So far yes, but we are working on improving the situation here.

[eddiebergman]

Closing this issue as it has been grouped together in #1298