scava icon indicating copy to clipboard operation
scava copied to clipboard

Delta creation getting out of hand on large projects

Open tdegueul opened this issue 5 years ago • 8 comments

I recently pushed a commit (d3f855e) that details how much time it takes to compute the deltas for VCS / BTS / communication channels. Here's the result for https://github.com/elastic/elasticsearch (one of the project used in Bitergia's use case):

INFO  [ProjectDelta (elasticsearch,20190101)] (14:47:12): Created Delta (vcs:4766ms, communications:1ms, bugs:1796820ms)

This is ~30min for the BTS. The delta creation seems to iterate on a lot of issues:

AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/36265/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/36263/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/29963/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/16654/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/36251/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/29970/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/27312/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/36258/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/36256/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/30957/comments?per_page=100&page=1 )
AbstractInterceptor.intercept( https://api.github.com/repos/elastic/elasticsearch/issues/17997/comments?per_page=100&page=1 )

I understand that the poor performance is probably due to the large amount of data on this project + rate limitation on GitHub's APIs; but would there be any way to improve this?

Also, is there a way to not print these AbstractInterceptor.intercept() debug messages to the console? Thanks!

tdegueul avatar Aug 23 '19 12:08 tdegueul