resolve
resolve copied to clipboard
Creating new deployment and reset sagas leads to high load and accessibility problems with independent deployments
Describe the bug We have a task to automate our release. It does the following steps:
- Create new deployment (resolve-cloud deploy)
- Clone old event store to backup it (resolve-cloud clone)
- Link new deployment old event store (resolve-cloud link)
- Reset read models and sagas for new deployment (resolve-cloud read-models reset all & sagas reset-all)
- Manually create about 5 projects, 1 team, remove 3 projects (about 20 events).
Yesterday we run this script twice in the afternoon. After that our independent main production deployment had the following issues:
- 20-40 sec delays for any api requests
- a lot of 503 errors
In new deployments event store adds about 5000 events in hour without any activity. In a 5 hours the event store had 25000 events
After two hours the situation became better and we tried to reproduce the issue and run the steps above one more time, exactly after that the issue with production deployment returned. After that we decided to delete all three new deployments and in a minute the load became zero and any issues were gone.
- Three new deployment ids: tthtfa cyixfh menhr6, event store id l5pcpg
- ReSolve Version [e.g. 0.32.0]