open-balena-api
open-balena-api copied to clipboard
Setting a release to successful can take a long time with big fleets
Whenever we set a release state to success, we also ensure every device has a service install for the services in the release. This can take an extremely long time, causing the request to time out. The code which does this is here: https://github.com/balena-io/open-balena-api/blob/master/src/hooks/resources/release.ts#L83
We should move to a more on-demand creation of service install resources to avoid a big bulk being done in the same transaction as the request. If the request times out, the transaction is rolled back, and without the caller increasing their timeout, they'll constantly hit the same problem.
See: https://jel.ly.fish/pattern-changing-tracked-release-large-fleet-fails-timeout-when-new-release-services-previous-one-a0bef96