[MNT] Raise version bound for scikit-learn 1.6

[MatthewMiddlehurst]

What does this implement/fix? Explain your changes.

Raises the sklearn version bound and fixes broken tests.

This version introduces a new tags system used by sklearn. We have our own tag system, but I have integrated this somewhat so sklearn functions which use tags their should still work as intended.

The classification, regression and clustering base now extend the respective sklearn mixin. Just inheriting these is more maintainable than implementing our own methods for their tags and keeping up with changes to methods which use them IMO ( i.e. one of the breakages was is_classifier which now uses these tags instead of self._estimator_type).

Any other comments?

Currently, there are some expected results failures due to a bug with trees, see scikit-learn/scikit-learn#30557. I probably won't merge this until 1.6.1 so we dont have to swap this again unless someone really needs it.

[aeon-actions-bot]

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ $\color{#EC843A}{\textsf{maintenance}}$ ].
I have added the following labels to this PR based on the changes made: [ $\color{#BCAE15}{\textsf{classification}}$, $\color{#4011F3}{\textsf{clustering}}$, $\color{#7E0206}{\textsf{regression}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

• Run pre-commit checks for all files
• Run mypy typecheck tests
• Run all pytest tests and configurations
• Run all notebook example tests
• Run numba-disabled codecov tests
• Stop automatic pre-commit fixes (always disabled for drafts)
• Disable numba cache loading
• Push an empty commit to re-run CI checks

[MatthewMiddlehurst]

@baraline I think other than expected results the only thing is the shapelet visualiser i.e. https://github.com/aeon-toolkit/aeon/actions/runs/12699462793/job/35400155079?pr=2486#step:8:925, not sure what would cause this. Can take a look if you dont have time but could just be a small thing.

[baraline]

Will check during lunch time, doesnt look to bad

[baraline]

@MatthewMiddlehurst I don't seem to be able to reproduce the bug locally somehow? I don't see how the changes could cause it. I'm using

    python: 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\antoine\anaconda3\envs\aeon_dev\python.exe
   machine: Windows-10-10.0.22631-SP0
Python dependencies:
         aeon: 1.0.0
          pip: 23.3.1
   setuptools: 68.0.0
 scikit-learn: 1.2.2
        numpy: 1.24.4
        numba: 0.58.1
        scipy: 1.11.4
       pandas: 2.0.3

[MatthewMiddlehurst]

@baraline need scikit-learn 1.6.0

[baraline]

@MatthewMiddlehurst The issue is that the coef_ argument of RidgeClassifierCV is of shape (n_features) instead of shape (n_targets,n_features) as it should be (it's (1, n_features) according to docs in binary case) after fitting any shapelet classifier using it as classifier, so this could be an sklearn bug ? A simple fix on our side would be to check for at least 2D array on line 716 of _shapelets.py visualization file

[TonyBagnall]

lgtm

[baraline]

Fixed the shapelet visualisation part (nvm some notebook failure still happens), but it seems there is other failures with expected results for multiple estimators, is 1.6 supposed to change any behaviour ? I don't see any change on highlights or docs between 1.5 and 1.6.

Edit : for some linear models, the coef_ argument changed in 1.6, see scikit-learn/scikit-learn@2f8f9f3

[MatthewMiddlehurst]

Thanks @baraline. Could you try reproduce the current failures/replace the results locally if you find the time? I cant seem to do so.

[baraline]

Sure, will take à look

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[baraline]

Will changes the ones identified by this script, as they are also the one failing in CI.

import pandas as pd
import numpy as np
import warnings

warnings.filterwarnings("ignore")

from aeon.testing.expected_results.expected_regressor_outputs import (
    cardano_sentiment_preds,
    covid_3month_preds,
)
from aeon.testing.expected_results.expected_classifier_outputs import (
    basic_motions_proba,
    unit_test_proba,
)
from aeon.testing.expected_results.results_reproduction.classifier_results_reproduction import (
    _reproduce_classification_unit_test,
    _reproduce_classification_basic_motions,
)
from aeon.testing.expected_results.results_reproduction.regressor_results_reproduction import (
    _reproduce_regression_cardano_sentiment,
    _reproduce_regression_covid_3month,
)
from sklearn.utils._testing import set_random_state
from aeon.utils.discovery import all_estimators

from aeon.utils.show_versions import show_versions

show_versions()

regression_datasets = ["CardanoSentiment", "Covid3Month"]
classification_datasets = ["BasicMotions", "UnitTest"]
local_regression = {
    "Covid3Month": _reproduce_regression_covid_3month,
    "CardanoSentiment": _reproduce_regression_cardano_sentiment,
}
local_classification = {
    "BasicMotions": _reproduce_classification_basic_motions,
    "UnitTest": _reproduce_classification_unit_test,
}
expected_regression = {
    "Covid3Month": covid_3month_preds,
    "CardanoSentiment": cardano_sentiment_preds,
}
expected_classification = {
    "BasicMotions": basic_motions_proba,
    "UnitTest": unit_test_proba,
}
classifiers = all_estimators(type_filter="classifier")
regressors = all_estimators(type_filter="regressor")

for classifier_name, classifier in classifiers:
    for dataset_name in classification_datasets:
        if classifier_name in expected_classification[dataset_name]:
            try:
                clf = classifier._create_test_instance(
                    parameter_set="results_comparison"
                )
                set_random_state(clf, 0)
                preds = local_classification[dataset_name](clf)
                expected = expected_classification[dataset_name][classifier_name]
                n_classes = preds.shape[1]
                df = pd.DataFrame(index=np.arange(len(preds)))
                for i_cls in range(n_classes):
                    df[f"local cls_{i_cls}"] = preds[:, i_cls]
                    df[f"expected cls_{i_cls}"] = expected[:, i_cls]
                    df[f"diff cls_{i_cls}"] = np.abs(
                        preds[:, i_cls] - expected[:, i_cls]
                    )
                for i_cls in range(n_classes):
                    if (df[f"diff cls_{i_cls}"] > 0.01).any():
                        print(f"{classifier_name} - {dataset_name}")
                        # print(df.to_markdown())
                        break
            except Exception as e:
                print(f"{classifier_name} failed due to {e}")


for regressor_name, regressor in regressors:
    for dataset_name in regression_datasets:
        if regressor_name in expected_regression[dataset_name]:
            try:
                clf = regressor._create_test_instance(
                    parameter_set="results_comparison"
                )
                set_random_state(clf, 0)
                preds = local_regression[dataset_name](clf)
                expected = expected_regression[dataset_name][regressor_name]

                df = pd.DataFrame(index=np.arange(len(preds)))

                df[f"local"] = preds
                df[f"expected"] = expected
                df[f"diff"] = np.abs(preds - expected)
                if (df[f"diff"] > 0.01).any():
                    print(f"{regressor_name} - {dataset_name}")
                    # print(df.to_markdown())

            except Exception as e:
                print(f"{regressor_name} failed due to {e}")

Output :

System:
    python: 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\antoine\anaconda3\envs\aeon_dev\python.exe
   machine: Windows-10-10.0.26100-SP0
Python dependencies:
         aeon: 1.0.0
          pip: 23.3.1
   setuptools: 68.0.0
 scikit-learn: 1.6.0
        numpy: 1.24.4
        numba: 0.58.1
        scipy: 1.14.1
       pandas: 2.0.3
FreshPRINCEClassifier - BasicMotions
REDCOMETS failed due to tuple indices must be integers or slices, not tuple
REDCOMETS failed due to tuple indices must be integers or slices, not tuple
FreshPRINCERegressor - CardanoSentiment
RISTRegressor - CardanoSentiment

[baraline]

I can't seem to find a way to bring them in line with what I have locally, on python 3.11 I now get another fail with DrCIFRegressor - Covid3Month which doesn't appear in the CI fails, but nothing on 3.9.
So I'm a bit clueless here ...

[MatthewMiddlehurst]

I am just going to run these on Linux only for now. I am not sure what exactly is going on but this seems to only impact estimators which use the scikit-learn decision trees. I'm just going to trust they have not completely broken it and some of the internals have just changed 🙂.

[baraline]

LGTM. I think it's the correct way to go, these tests are to guarantee that we don't change a kind of default behaviour of the estimators, so only testing it on one platform should be enough in this regard.