Queue buffer size exhausted

[ceprateek]

I keep seeing this in the logs for the exporter: {"level":"warning","msg":"queue buffer size exhausted, skipping scheduling of task..","task_type":"PullRefMetrics","task_unique_id":"1760622161","time":"2023-05-23T20:29:32Z"}

Can you please let me know what is the impact. I see in the code that the size is set to 1000 and there is no option to change it without forking the repo

[Routhinator]

I guess the workaround is multiple exporters and targetting smaller subgroups of projects, but seeing this as well.

[Routhinator]

Well I found the implication @ceprateek - the agent is effectively dead and produces zero metrics. The only way out of this seems to be to manually restart the process....

This needs some fixing it would seem...

[Routhinator]

@mvisonneau I am guessing the only workaround for now that is stable is to deploy multiple exporters scoped to smaller project groups and it removes the ability to use the wildcard "all projects" setting...

Do you have any explanation on why the 1000 setting was hardcoded? Could this be changed without harming the application?

[R-Studio]

We have the same issue with a maximum_jobs_queue_size of 5000. Any news or tips how to scale the exporter for big GitLab instances?

@ceprateek I think you can change the setting: https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/blob/0687efd1075d33283c3762a135aa24de94ffe21c/docs/configuration_syntax.md?plain=1#L106C21-L106C21

[ehonda]

There's also an explanation what can be configured to mitigate this here