[MNT] Add expected results for classifiers and regressors

[MatthewMiddlehurst]

trafficstars

Describe the issue

Typically we like to compare the results of our classifiers and regressors against a saved set. Some algorithms in these packages do not do this currently. This does not have to be done in a single PR.

This will most likely involve adding a "results comparison" parameter set for the estimator (i.e. this. The parameters chosen should be large enough that the results change when significant changes to the estimator are made, but small enough that the test does not clog up our CI runtime.

Suggest a potential alternative/fix

See the contents of testing/expected_results/. Run the missing estimators (not those which are non-deterministic, i.e. deep learners), using the utilities within, and add them to the expected results file.

Additional context

No response

[AnonymousCodes911]

Greetings! @MatthewMiddlehurst , From what i understand, We have to modify each script from testing/expected_results and make sure that the classifiers and regressors return output in the same set? further, i tried running the script, but it returned ModuleNotFoundError: No module named 'sklearn' (i have built Aeon using the guides given) , kindly guide me through this error. Thank you for your time and consideration.

[MatthewMiddlehurst]

Yes, so for estimators which are currently missing from the relevant expected results file, add it to the results reproduction file and run it.

It seems like you do not have any dependencies installed from that error? Try reinstalling using the developer install guide. You will want to create a fork and clone from it instead if you are going to make a PR.

[AnonymousCodes911]

Thank you so much!, I will look into it.

[harshithasudhakar]

Hi, I ran the test classifier_results_reproduction.py there's seems to be mismatch between the expected output, and the actual output for HIVECOTEV2.

Actual Output:
HIVECOTEV2 - UnitTest
[
[0.2802, 0.7198],
[0.5851, 0.4149],
[0.1088, 0.8912],
[1.0, 0.0],
[0.9769, 0.0231],
[1.0, 0.0],
[0.7753, 0.2247],
[0.2571, 0.7429],
[0.7984, 0.2016],
[0.929, 0.071],
]


Expected Output:
[
[0.2469, 0.7531],
[0.6344, 0.3656],
[0.0959, 0.9041],
[1.0, 0.0],
[0.9796, 0.0204],
[1.0, 0.0],
[0.802, 0.198],
[0.2265, 0.7735],
[0.8224, 0.1776],
[0.9374, 0.0626],
]

It seems to be the same case for HIVECOTEV2 - BasicMotions

[harshithasudhakar]

Would you assign me this issue? I'd like to work on it :)

[MatthewMiddlehurst]

If there are results present and there is a mismatch, there would be a test failure.

Are you using https://github.com/aeon-toolkit/aeon/blob/main/aeon/testing/expected_results/classifier_results_reproduction.py to generate the expected results?

[harshithasudhakar]

If there are results present and there is a mismatch, there would be a test failure.

Are you using https://github.com/aeon-toolkit/aeon/blob/main/aeon/testing/expected_results/classifier_results_reproduction.py to generate the expected results?

Yes, I'm using classifier_results_reproduction.py, but there seems to be no test failure

[MatthewMiddlehurst]

I would expect https://github.com/aeon-toolkit/aeon/blob/main/aeon/classification/tests/test_all_classifiers.py#L70 to fail if so. Gave it a quick run,

if __name__ == "__main__":
    # change as required when adding new classifiers, datasets or updating results
    _print_results_for_classifier("HIVECOTEV2", "UnitTest")

gives me the expected results.

[harshithasudhakar]

Hi, I ran the test classifier_results_reproduction.py there's seems to be mismatch between the expected output, and the actual output for HIVECOTEV2.

Actual Output:
HIVECOTEV2 - UnitTest
[
[0.2802, 0.7198],
[0.5851, 0.4149],
[0.1088, 0.8912],
[1.0, 0.0],
[0.9769, 0.0231],
[1.0, 0.0],
[0.7753, 0.2247],
[0.2571, 0.7429],
[0.7984, 0.2016],
[0.929, 0.071],
]


Expected Output:
[
[0.2469, 0.7531],
[0.6344, 0.3656],
[0.0959, 0.9041],
[1.0, 0.0],
[0.9796, 0.0204],
[1.0, 0.0],
[0.802, 0.198],
[0.2265, 0.7735],
[0.8224, 0.1776],
[0.9374, 0.0626],
]

It seems to be the same case for HIVECOTEV2 - BasicMotions

It seems that I've been using an older version of Aeon, I upgraded it and it works fine. There's no discrepancy in the results.

[MatthewMiddlehurst]

Classification has been updated in #1379