Skip to content

Modify Stopping Criterion #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vivekaxl opened this issue Mar 26, 2018 · 4 comments
Closed

Modify Stopping Criterion #451

vivekaxl opened this issue Mar 26, 2018 · 4 comments
Labels
documentation Something to be documented

Comments

@vivekaxl
Copy link

Is it possible to modify the stopping criterion on Autosklearn from time_left_for_this_task to the number of evaluations?

@mfeurer
Copy link
Contributor

mfeurer commented Mar 27, 2018

Yes, this is possible. Unfortunately, we did not yet document this feature (please leave this issue open so we are reminded to do so). This an adaption of the holdout example which only runs 5 target algorithm runs:

import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

import autosklearn.classification


def main():
    X, y = sklearn.datasets.load_digits(return_X_y=True)
    X_train, X_test, y_train, y_test = \
        sklearn.model_selection.train_test_split(X, y, random_state=1)

    automl = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=120,
        per_run_time_limit=30,
        tmp_folder='/tmp/autosklearn_holdout_example_tmp',
        output_folder='/tmp/autosklearn_holdout_example_out',
        disable_evaluator_output=False,
        smac_scenario_args={'runcount_limit': 5}
    )
    automl.fit(X_train, y_train, dataset_name='digits')

    # Print the final ensemble constructed by auto-sklearn.
    print(automl.show_models())
    predictions = automl.predict(X_test)
    # Print statistics about the auto-sklearn run such as number of
    # iterations, number of models failed with a time out.
    print(automl.sprint_statistics())
    print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))


if __name__ == '__main__':
    main()

However, the ensemble script will not realize that the optimization finished and will continue until the given time is up. To avoid this, you can disable ensemble and build it after the optimization finished.

@mfeurer mfeurer added the documentation Something to be documented label Mar 27, 2018
@vivekaxl
Copy link
Author

Thank you, @mfeurer.

@smilesun
Copy link

smilesun commented Feb 19, 2019

@vivekaxl ,did you get it to work? I met the problem of refit error when settting ensemble_size =0, see #637, when using cross validation

@mfeurer mfeurer mentioned this issue May 29, 2019
@mfeurer
Copy link
Contributor

mfeurer commented Mar 26, 2021

This will be documented in an upcoming FAQ section of the manual.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Something to be documented
Projects
None yet
Development

No branches or pull requests

3 participants