QPS / burst options for k8s go-client

[mrkwtz]

Is your feature request related to a problem? Please describe. Our use-case looks like this: We get upload events from S3 sent to SQS, then the SQS EventSource fetches these messages and a Sensor triggers a Argo Workflow (with the k8s trigger) to kick off a conversion of these files. The conversion takes on average ~12s and we're going to have thousands of S3 events per second. Also we're going to start reconversions of maybe tens of thousands of files simutanouesly.

What we observed is, that the rate at which the ArgoWorkflow resources are created is very low (5/s). With 10,000 files/events it would take around 30 minutes until all workflow objects are created. That just slows down the whole pipeline.

The 5/s is the default QPS defined in the k8s go-client used in the k8s trigger.

Describe the solution you'd like I would like to be able to set the qps and burst options for the k8s client. I already built a sensor image with fixed qps and burst of 100 to verify that this is the bottleneck, it was. I'm just not sure how to implemented this. My first shot would be environment variables or arguments for the sensor.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.