[Bug][Plugins][pipeline] Each stage in sonarQube pipeline taking hours to complete.

[suchetanvg]

Search before asking

• [x] I had searched in the issues and found no similar issues.

What happened

I am using v1.0.3 beta6. Create a devlake project with sonarqube connection with more than 1000 data scope and trigger the pipeline to collect the data. Each stage is successfully completing but taking hours to complete.

What do you expect to happen

Complete each stage quickly

How to reproduce

Create a devlake project with sonarqube connection with more than 1000 data scope and trigger the pipeline to collect the data. Each stage is successfully completing but taking hours to complete.

Anything else

No response

Version

v1.0.3 beta6

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[dosubot[bot]]

Hi @suchetanvg! I'm Dosu and I’m helping the incubator-devlake team.

Slow SonarQube pipeline stages in DevLake with large numbers of data scopes are a known challenge, especially at enterprise scale. The main bottleneck is often the single runner processing projects sequentially, which limits parallelism and causes long runtimes for each stage. Increasing container memory (at least 6GB), optimizing database indexes, and enabling shallow cloning (where applicable) have helped others speed up pipelines with large data sources. For very large scopes, reducing the scope of data collection (like limiting date ranges) or lowering page size (if configurable) can also help mitigate performance issues. Collecting backend logs during slow stages can help pinpoint bottlenecks. These strategies have been discussed in detail by other users running DevLake at scale here, here, and here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[suchetanvg]

Issue is still persists

[dosubot[bot]]

This is a known bottleneck with the SonarQube plugin in DevLake when handling large data scopes. The batching and splitting logic is hardcoded, so there aren’t any user-facing options to tune performance—large scopes will always result in many sequential API requests, which slows down the pipeline. The only immediate workaround is to reduce the scope per run, for example by limiting the number of projects or issues collected at once, or narrowing the date range.

If you haven’t already, try increasing your container memory to at least 6GB and make sure your database indexes are optimized, as these have helped others with similar issues. Some users have also seen improvements by upgrading to newer versions (like v1.0.4-alpha), which resolved related pipeline problems for them (source).

If the issue persists even after these steps, collecting backend logs and resource usage metrics during slow pipeline execution can help pinpoint the bottleneck (source). Let me know if you can share those details, or if you’ve already tried the latest version.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}