Modify Stopping Criterion

[vivekaxl]

Is it possible to modify the stopping criterion on Autosklearn from time_left_for_this_task to the number of evaluations?

[mfeurer]

Yes, this is possible. Unfortunately, we did not yet document this feature (please leave this issue open so we are reminded to do so). This an adaption of the holdout example which only runs 5 target algorithm runs:

import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

import autosklearn.classification


def main():
    X, y = sklearn.datasets.load_digits(return_X_y=True)
    X_train, X_test, y_train, y_test = \
        sklearn.model_selection.train_test_split(X, y, random_state=1)

    automl = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=120,
        per_run_time_limit=30,
        tmp_folder='/tmp/autosklearn_holdout_example_tmp',
        output_folder='/tmp/autosklearn_holdout_example_out',
        disable_evaluator_output=False,
        smac_scenario_args={'runcount_limit': 5}
    )
    automl.fit(X_train, y_train, dataset_name='digits')

    # Print the final ensemble constructed by auto-sklearn.
    print(automl.show_models())
    predictions = automl.predict(X_test)
    # Print statistics about the auto-sklearn run such as number of
    # iterations, number of models failed with a time out.
    print(automl.sprint_statistics())
    print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))


if __name__ == '__main__':
    main()

However, the ensemble script will not realize that the optimization finished and will continue until the given time is up. To avoid this, you can disable ensemble and build it after the optimization finished.

[vivekaxl]

Thank you, @mfeurer.

[smilesun]

@vivekaxl ,did you get it to work? I met the problem of refit error when settting ensemble_size =0, see #637, when using cross validation

[mfeurer]

This will be documented in an upcoming FAQ section of the manual.