Best Practices: Fitting thousands of models in a performant and vectorized manner ?

[kylemcmearty]

I saw the updates in 1.1.1 and 1.1.2 rely on using vectorized methods, among other things, to achieve a great speed increase.

If I have thousands of items to predict for, say 20 - 60k items, is there a way to speed up fitting and predicting to take advantage of the vectorized methods?

One way to get speed out of pandas is to run a vectorized method over a dataframe. Is there a way to run a fit and predict method in a vectorized manner for many thousands of items?

Currently I'm running fit and predict on each individual item via multiprocessing and then gluing the resultant dataframes from each item together to then post process all my information in one shot.

I had an idea to fit all items in one shot, store the models in a list and then run through the list calling predict on each, in an effort to try to utilize the vectorized methods. I know that approach isnt really vectorized, but this is where I had the thought about my question posted above ^^.

Any other suggestions or best practices to speed up fitting and predicting tens of thousands of items would be appreciated!

[donggu-kang]

Hi. Like you, I am waiting a lot of time to fit hundreds of thousands of values.

I too am very interested in this issue. Right now, we are using appropriate sampling methods or adjusting the number of samples to predict as needed to reduce the number of values.

But this is still not a fundamental solution.