Anastassia Loukina

For classifiers that do not support expected probabilities, we currently rely on SKLL to raise a warning and proceed generating integer scores. The final report still says "Predictions analyzed in...

Look into parallelization

1

[JIRA] Perhaps, we can use multiprocessing or multithreading to speed up model training, report generation etc. This might be relevant: http://ipyparallel.readthedocs.io/en/latest/intro.html

Consistent approach to warnings

5

In some parts of the code we use `logger.warning`while in other parts we used `warnings`. Is there any reason not use `warnings` everywhere?

QWK generates runtime warning

1

There are edge cases where QWK function generates runtime warning: ```/opt/python/rsmtool/lib/python3.6/site-packages/rsmtool/utils/metrics.py:471: RuntimeWarning: invalid value encountered in double_scalars kappa = numerator / denominator /opt/python/rsmtool/lib/python3.6/site-packages/rsmtool/utils/metrics.py:471: RuntimeWarning: invalid value encountered in double_scalars kappa...

Add suppress_warnings argument to various metrics

1

The functions for computing individual metrics raise warning in edge cases (for example, when standard deviation is zero). In some cases this can result in multiple warnings which all have...

Currently config files/dictionary can only contain a path to training/test file. It would be nice to allow the user to pass a pandas Dataframe if they are using API with...

Resource warning when running tests

8

`...../nose/tools/nontrivial.py:62: ResourceWarning: unclosed file pass ResourceWarning: Enable tracemalloc to get the object allocation traceback`

Currently in `correlation_helper` we let `pearsonr` return np.nan if the input has zero variance which leads to RunTime warnings. We should check for these and return np.nan ourselves with a...

Kappa computation when predicted scores are on a different scale

10

In a situation where predicted scores are on a completely different scale from the observed scores, kappa computation fails because the range of possible scores is too large. We saw...