Limit parallel AMQP link creation requests
When developing the fix for #2660, load-tests have been done with many concurrent device requests for different tenants. The first such request for a tenant causes tenant-specific AMQP links from the protocol adapter to the device registry and command router to be created.
The tests have revealed that a high number of such parallel link creation requests can cause a considerable amount of timeout failures like
link establishment [peer: hono-service-command-router.hono] timed out after 1000ms.
causing 503 errors returned to the device:
failed to create new client for [cmd_router-loadtesttenant46]
org.eclipse.hono.client.ServerErrorException: Error Code: 503
at org.eclipse.hono.client.amqp.connection.impl.HonoConnectionImpl.onLinkEstablishmentTimeout(HonoConnectionImpl.java:800)
at org.eclipse.hono.client.amqp.connection.impl.HonoConnectionImpl.lambda$createSender$11(HonoConnectionImpl.java:669)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:893)
Although, to be fair, it is not fully clear whether the parallel link creation requests are the main factor here or rather the fact that in this phase of there not being any cached tenant/device data entries in the adapter yet, the general amount of load being generated on the device registry and command router components is causing this.
But in any case, I think limiting the number of parallel link creation requests per client/connection could be a good idea. The number of AMQP messages being sent at a time is already limited via the credit mechanism, but there is nothing equivalent concerning link creation requests.