fix: issue where canary overwrites default backend

[Spazzy757]

What this PR does / why we need it:

We have a bug in the case of canary deployments where if the canary deployment returns an error code it will ignore the custom-error codes and custom default backend setup.

This is due to how we overwrite the balancer in the function route_to_alternative_balancer as it will take precedence over any balancer that is set

NOTE: Majority of the code added here is around E2E testing for this specific scenario. This code will be simplified with the work in https://github.com/kubernetes/ingress-nginx/issues/9948

Types of changes

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] CVE Report (Scanner found CVE and adding report)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Documentation only

Which issue/s this PR fixes

fixes: https://github.com/kubernetes/ingress-nginx/issues/9944

How Has This Been Tested?

Checklist:

• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I've read the CONTRIBUTION guide
• [ ] I have added unit and/or e2e tests to cover my changes.
• [ ] All new and existing tests passed.

/kind bug

[Spazzy757]

/kind bug

[Spazzy757]

/triage accepted

[Spazzy757]

/assign @tao12345666333

[tao12345666333]

I am reading the full context of this PR and issue, it will take some time. Thanks

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Spazzy757 Once this PR has been reviewed and has the lgtm label, please ask for approval from tao12345666333. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[netlify[bot]]

✅ Deploy Preview for kubernetes-ingress-nginx canceled.

Name	Link
🔨 Latest commit	64229d2c08988fbf98dcab90f453fb322ea0da01
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-ingress-nginx/deploys/65e18ef7f76bdb0008e3356c

[Spazzy757]

@tao12345666333 any update on this PR?

[strongjz]

/priority backlog /ok-to-test

[strongjz]

Possible candidate for 1.9.0

[dfl-aeb]

@tao12345666333 @Spazzy757 @strongjz any news on this PR?

[dfl-aeb]

@tao12345666333 @Spazzy757 Sorry to bother you again, this PR looks pretty ready to me. Can we help push it forward?

[Spazzy757]

I will rebase the PR, but I am waiting on @tao12345666333 or @strongjz to approve this. As I believe the concern is that it's a breaking change

[tao12345666333]

Thanks 🙏 let me do a review today

[dfl-aeb]

@tao12345666333 Friendly reminder 😊👌

[Spazzy757]

I'll try rebase again tomorrow @tao12345666333 @strongjz

[dfl-aeb]

@Spazzy757 @tao12345666333 are we good to go?

[Spazzy757]

@Spazzy757 @tao12345666333 are we good to go?

I assume so but not sure when it will be merged