hono icon indicating copy to clipboard operation
hono copied to clipboard

Limit parallel AMQP link creation requests

Open calohmn opened this issue 3 years ago • 0 comments

When developing the fix for #2660, load-tests have been done with many concurrent device requests for different tenants. The first such request for a tenant causes tenant-specific AMQP links from the protocol adapter to the device registry and command router to be created. The tests have revealed that a high number of such parallel link creation requests can cause a considerable amount of timeout failures like link establishment [peer: hono-service-command-router.hono] timed out after 1000ms. causing 503 errors returned to the device:

failed to create new client for [cmd_router-loadtesttenant46]
org.eclipse.hono.client.ServerErrorException: Error Code: 503
	at org.eclipse.hono.client.amqp.connection.impl.HonoConnectionImpl.onLinkEstablishmentTimeout(HonoConnectionImpl.java:800)
	at org.eclipse.hono.client.amqp.connection.impl.HonoConnectionImpl.lambda$createSender$11(HonoConnectionImpl.java:669)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:893)

Although, to be fair, it is not fully clear whether the parallel link creation requests are the main factor here or rather the fact that in this phase of there not being any cached tenant/device data entries in the adapter yet, the general amount of load being generated on the device registry and command router components is causing this.

But in any case, I think limiting the number of parallel link creation requests per client/connection could be a good idea. The number of AMQP messages being sent at a time is already limited via the credit mechanism, but there is nothing equivalent concerning link creation requests.

calohmn avatar Jul 29 '22 09:07 calohmn