On exception during fit script doesn't terminate cleanly

[choco]

Describe the bug

If an exception is triggered during the fit process of the auto-sklearn classifier the python interpreter process does NOT exit cleanly (needs a manual keyboard interrupt).

To Reproduce

Steps to reproduce the behavior:

1. Write simple script to trigger exception

import autosklearn.classification
import pandas as pd
automl = autosklearn.classification.AutoSklearnClassifier()
a = pd.DataFrame(columns=["id"])
b = pd.DataFrame(columns=["id", "y"])
a.set_index("id", inplace=True)
b.set_index("id", inplace=True)
combined_data = pd.merge(a, b, left_index=True, right_index=True)
x = combined_data.drop('y', axis=1)
y = combined_data['y']
automl.fit(x, y)

2. Execute script python script.py
3. Script shows this backtrace:

Traceback (most recent call last):
  File "script2.py", line 11, in <module>
    automl.fit(x, y)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/estimators.py", line 1045, in fit
    super().fit(
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/estimators.py", line 375, in fit
    self.automl_.fit(load_models=self.load_models, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/automl.py", line 2056, in fit
    return super().fit(
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/automl.py", line 639, in fit
    self.InputValidator.fit(X_train=X, y_train=y, X_test=X_test, y_test=y_test)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/data/validation.py", line 147, in fit
    self.target_validator.fit(y_train, y_test)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/data/target_validator.py", line 119, in fit
    self._fit(y_train, y_test)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/data/target_validator.py", line 182, in _fit
    self.encoder.fit(y_train.to_numpy().reshape(-1, 1))
  File "/usr/local/lib/python3.8/dist-packages/sklearn/preprocessing/_encoders.py", line 779, in fit
    self._fit(X)
  File "/usr/local/lib/python3.8/dist-packages/sklearn/preprocessing/_encoders.py", line 77, in _fit
    X_list, n_samples, n_features = self._check_X(
  File "/usr/local/lib/python3.8/dist-packages/sklearn/preprocessing/_encoders.py", line 44, in _check_X
    X_temp = check_array(X, dtype=None,
  File "/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py", line 726, in check_array
    raise ValueError("Found array with %d sample(s) (shape=%s) while a"
ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required.

Expected behavior

After showing the stack trace, script should terminate, I expect the python interpreter to quit.

Actual behavior, stacktrace or logfile

After showing the exception the script doesn't terminate, it's stuck processing some exit handlers.
If I send keyboard interrupt I get the following stack trace:

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
Process ForkProcess-1:
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/util/logging_.py", line 320, in start_log_server
    receiver.serve_until_stopped()
  File "/usr/local/lib/python3.8/dist-packages/autosklearn/util/logging_.py", line 350, in serve_until_stopped
    rd, wr, ex = select.select([self.socket.fileno()],
KeyboardInterrupt

Environment and installation:

Please give details about your installation:

• Ubuntu 20.04, virtual env, python 3.10
• Docker (https://hub.docker.com/r/mfeurer/auto-sklearn/ which is same Ubuntu version), python 3.8
• auto-sklearn version 0.14.7

[eddiebergman]

Hey thanks for the report, you're right that this does happen and sorry for that. I guess this easiest fix is a big try: except in the main fit method such that we can properly close the logging server when an error occurs.

It might take a bit before we can put this in due to some other deadlines we are reaching for right now but it seems like an easy fix (hopefully). The code snippet will help test this a lot easier :)

[eddiebergman]

Hi @choco,

Many thanks for providing the code for making it reproducible! There's a PR in #1503 for it now

Best,
Eddie

[choco]

Hey @eddiebergman, thanks for the quick fix :)