distfit icon indicating copy to clipboard operation
distfit copied to clipboard

Incorporate numba with scipy.special and numba-stats

Open deanm0000 opened this issue 1 year ago • 3 comments

There may be some potentially significant speed improvements by running code that is compiled. At first glance, it seems that there doesn't exist a fit method in numba-stats so that might be a significant hurdle and not all the distributions are available that way so this is kind of a place holder or an aspirational enhancement.

deanm0000 avatar Oct 18 '23 18:10 deanm0000

Interesting. I have been looking into it but the number of available distributions is limited. In this line, I again tried to use numba but it keeps giving the same error since forever: https://github.com/numba/numba/issues/3250

Then I looked into numba-scipy but I could not find the documentation about distributions. Maybe it is not supported.

erdogant avatar Oct 19 '23 19:10 erdogant

numba-scipy opens up the scipy.special namespace so that numba is aware of them.

deanm0000 avatar Oct 19 '23 19:10 deanm0000

I just created a new release where I added parallelization to the bootstrap function which significantly speeds up the results. To enable it, simply set n_jobs=-1 during initialization. However, when using multiple cores, it sometimes randomly results in a RuntimeWarning: invalid value encountered. I am not sure why this happens but if you set it back to n_jobs=1, it does not show any RuntimeWarnings.

Regarding, scipy-special, I am not sure how these functions match with scipy.stats?

erdogant avatar Oct 19 '23 20:10 erdogant

This issue is related to #49 which is recently solved!

erdogant avatar May 17 '24 11:05 erdogant