cobra icon indicating copy to clipboard operation
cobra copied to clipboard

Implementation for #126: supporting evaluation & forward feature selection for custom metrics (e.g. lift)

Open sandervh14 opened this issue 2 years ago • 4 comments

====> Do not merge yet! See below for explanation. <=====

Story Title

See issue #126: Tuning classification model on different metrics (lift, recall, precision,...)

Changes made

  • Adapted forward feature selection class:
    • it can now accept metric, metric_args and metric_kwargs and higher_is_better as argument.
    • Extensive documentation of what to fill in in those parameters.
    • compute_model_performances: passing metric, metric_args and metric_kwargs to model evaluate()
    • _find_next_best_model(): idem + adapted logic for selecting new model as next best, according to higher_is_better logic.
  • Adapted both LogisticRegressionModel and LinearRegressionModel to support a custom metric, metric_args and metric_kwargs in evaluate().

I tried out my changes in a notebook on a model, looks good. :-) But do not merge yet! I'm stuck on the 25 failing unit tests, mainly due to the cryptical error "mocker not found":

test setup failed
file C:\Users\sander.vandenhautte\PycharmProjects\cobra\tests\model_building\test_forward_selection.py, line 55
      @pytest.mark.parametrize("model_type", ["classification", "regression"])
      def test_compute_model_performances(self, mocker, model_type):
E       fixture 'mocker' not found
>       available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

=> Are we using fixtures? With a find, I can't locate where mocker is instantiated.

Anyway, next to this question, of course fixing the unit tests remain, before we can merge this into develop. But meanwhile Hendrik can use the issue branch, and I can discuss with you Sam how to fix the above unit test shizzle. :-) I'll plan in that work for next week.

How does the solution address the problem

This PR lets any data scientist the freedom to pick a different evaluation metric for forward feature selection or just for simple evaluation of a model.

Linked issues

Resolves #126

sandervh14 avatar Apr 08 '22 08:04 sandervh14