[air] Integration with sklearn-evaluation

[idomic]

Description

I think adding an integration/tutorial and the right documentation of the framework can go a long way.

You'd be able to evaluate your classification, regression and clustering models, generate reports from it and share across the org.

Use case

Usually, when using Ray you'd get the final/best model, and then you'll go and check your model's performance. This will allow an easier mechanism for the users to get to the second part of it.

[richardliaw]

@idomic could you share a bit about what your ideal workflow/code looks like? I'm having a bit of trouble understanding how to draw up an integration here.

[idomic]

Yes, sorry for the delay. So the tutorial I was thinking about was around training with Ray, and evaluating with sklearn-evaluation. That way you can train multiple models in parallel and then compare them at once without a lot of effort.

[richardliaw]

Maybe something simple like:

from sklearn.ensemble import RandomForestClassifier
from tune_sklearn import TuneGridSearchCV  # uses Tune here
from sklearn import datasets
from sklearn_evaluation import plot

data = datasets.make_classification(
    n_samples=200, n_features=10, n_informative=4, class_sep=0.5
)

X = data[0]
y = data[1]

hyperparameters = {
    "n_estimators": [1, 10, 50, 100],
    "criterion": ["gini", "entropy"],
    "max_features": ["sqrt", "log2"],
}

est = RandomForestClassifier(n_estimators=5)
clf = TuneGridSearchCV(est, hyperparameters, cv=3)
clf.fit(X, y)

grid_scores = clf.cv_results_

plot.grid_search(clf.cv_results_, change="n_estimators", kind="bar")

?

[stale[bot]]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale[bot]]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!