Running more than one instance of thundersvm on the same GPU increases time linearly

[julianofoleiss]

Hi everyone!

I need to optimize parameters C and gamma. I wondered if it was possible to run more than one instance of thundersvm in the same GPU at the same time. I realized that it was taking much longer than usual.

To try to figure out what was happening, i did the following.

I) I trained a svm with thundersvm with a dataset that has 36k examples, each with 156 features. The dataset has 10 classes. It took about 5:30 minutes. During this test, only this instance of thundersvm was running on the system, and no other processes were running.

then,

II) I trained two svms at the same time on the same GPU (a Titan X). The setup was the same as in (I). It took about 11 minutes for both SVMs to complete training! (2x the time to train a single model. That's why said it increases model training time linearly)

To make sure the problem was related to using the same GPU to train both models at the same time I also trained each model in a separate gpu (on the same machine, at the same time). The first model finished training in 5:30 minutes in a Titan X 1080, while the second model finished in 5:05 minutes in a GeForce GTX 1070.

Unfortunately I don't have much time to look into this. However, I'll be glad to to this test again in the future if this gets fixed.

Thanks!

[zeyiwen]

Thanks for pointing that out!

You can try nvidia-cuda-mps-control -d to start MPS which aims to improve efficiency of multitasking. You may refer to Section 4.1 of this document for more detailed information. Please do let us know the testing results using MPS.

By default, CUDA runs multiple tasks using time sharing, and essentially running multiple tasks sequentially (cf. this post and this).

It is also possible through designing the code of ThunderSVM to run multiple GPU kernels for different hyper-parameters. But this is non-trivial. We will also consider this solution in the future upgrade.