litmus icon indicating copy to clipboard operation
litmus copied to clipboard

Improve error handling in upgrade-agent

Open stoehdoi opened this issue 3 years ago • 2 comments

What happened: Upgrading the litmuschaos helm release fails if litmusportal version remains unchanged due to error in upgrade agent which runs a a post upgrade hook.

What you expected to happen: Helm upgrade command to succeed even if app version does not change.

Where can this issue be corrected? (optional) Using custom type errors would allow proper error handling at a later stage. In the above use-case, the upgrade agent should should do noop and just return without error.

Returning a custom error: https://github.com/litmuschaos/litmus/blob/master/litmus-portal/upgrade-agents/control-plane/versions/upgrade-manager.go#L92 Handling the error: https://github.com/litmuschaos/litmus/blob/master/litmus-portal/upgrade-agents/control-plane/main.go#L59

More info: https://go.dev/blog/go1.13-errors

How to reproduce it (as minimally and precisely as possible): Run the following commands:

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm upgrade --install litmus litmuschaos/litmus -n litmus # the first run will succeed
helm upgrade --install litmus litmuschaos/litmus -n litmus # this will fail due to the post-hook which errors

Anything else we need to know?: n/a

stoehdoi avatar Mar 15 '22 14:03 stoehdoi

cc: @rajdas98 @Jonsy13

uditgaurav avatar Mar 18 '22 17:03 uditgaurav

Still a problem in 2.8.1

❯ helm upgrade --install litmus litmus --repo https://litmuschaos.github.io/litmus-helm/ --version 2.8.1 --namespace litmus --create-namespace -f dev.yaml
Release "litmus" does not exist. Installing it now.
NAME: litmus
LAST DEPLOYED: Wed Apr 27 13:11:50 2022
NAMESPACE: litmus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing litmus 😀

Your release is named litmus and it's installed to namespace: litmus.

Visit https://docs.litmuschaos.io to find more info.
❯ helm upgrade --install litmus litmus --repo https://litmuschaos.github.io/litmus-helm/ --version 2.8.1 --namespace litmus --create-namespace -f dev.yaml
Error: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded

brnck avatar Apr 27 '22 10:04 brnck

HI @stoehdoi @brnck This issue was fixed in 2.9.0 - https://github.com/litmuschaos/litmus/releases/tag/2.9.0 Closing this issue. Feel free to reopen it if the issue still persists.

Jonsy13 avatar Oct 18 '22 04:10 Jonsy13