During v76 -> v77 upgrade canary deployment, UAA delete user endpoint returns false error

[peterhaochen47]

Description

During the upgrade to UAA v77 from UAA v76 or below with canary deployment (where briefly both the new and the old UAA servers could be running), UAA delete user endpoint might respond with an error even though the user deletion is successful.

Cause

During a canary deployment, briefly both the new and the old UAA servers could be running, resulting in the following:

• The new server (v77): Since the MFA feature is removed in V77.0.0, a DB migration will be run to drop the DB tables related to the MFA feature.
• The old server (v76): A user deletion would trigger explicit delete operations on the associated entries in other DB tables. In this case, a user deletion will trigger a delete operation on the MFA-related tables (even in the case where the MFA feature is not enabled). And since these tables have been dropped already by the new server, the operation fails. See relevant code below:
  • SystemDeletable
  • JdbcMfaProviderProvisioning

What version of UAA are you running?

V76 -> V77

How are you deploying the UAA?

Any upgrade scenarios with canary deployment.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/187263539

The labels on this github issue will be updated when the story is started.

[peterhaochen47]

So far, we have abandoned the proposed fix to this issue (https://github.com/cloudfoundry/uaa/pull/2790) as there are no active reports that users are encountering this issue. This issue may be closed after a few months once most users have gone through the upgrade to v77 (at which point, this issue will no longer occur).

[strehle]

@peterhaochen47 do you want to have this open ? Because now we are on 77.10.0 and discussions about Regression are not somehow obsolete after this time. our production runs on 77.9.0, our Canary on 77.10.0,

[peterhaochen47]

@strehle, Hi, let's keep this open until we have v78 (when the upgrade scenario of v76->v77 becomes rare). This issue is more of a documentation of a known issue, rather than actually asking for a fix.