open-balena-api icon indicating copy to clipboard operation
open-balena-api copied to clipboard

Setting a release to successful can take a long time with big fleets

Open CameronDiver opened this issue 5 years ago • 0 comments

Whenever we set a release state to success, we also ensure every device has a service install for the services in the release. This can take an extremely long time, causing the request to time out. The code which does this is here: https://github.com/balena-io/open-balena-api/blob/master/src/hooks/resources/release.ts#L83

We should move to a more on-demand creation of service install resources to avoid a big bulk being done in the same transaction as the request. If the request times out, the transaction is rolled back, and without the caller increasing their timeout, they'll constantly hit the same problem.

See: https://jel.ly.fish/pattern-changing-tracked-release-large-fleet-fails-timeout-when-new-release-services-previous-one-a0bef96

CameronDiver avatar Sep 06 '19 17:09 CameronDiver