Che should not update storage-type for existing workspaces

[amisevsk]

Is your enhancement related to a problem? Please describe

When setting a storage-type using the checluster .spec.devEnvironments.storage.pvcStrategy field, Che attempts to apply the value to all workspaces when they are started. However, there are issues with this approach:

• If a workspace has been previously started/used, changing the storage type will result in a loss of data as a different PVC will be used for the next workspace start. Any changes made in the workspace before the storage type was changed will no longer be accessible.
• If switching from the common/per-user storage type to another, any data stored by the workspace will be orphaned on the cluster. For example, if the workspace clones 500MB of data in a repo, that 500MB will be left on the common PVC even after the workspace is deleted. It will also prevent automatic deletion of the common PVC, as that only happens when the last common/per-user DevWorkspace is deleted.
• Setting the storage type on every start prevents using a non-default storage class; if the CheCluster specifies per-user, then per-user is the only supported storage type for DevWorkspaces started by Che, and it's not possible to e.g. start an ephemeral workspace through the dashboard UI.

Normally, the DevWorkspace Operator has webhooks to prevent switching the storage-type when one of the issues above could occur. However, for compatibility, we may need to temporarily disable those webhooks to support current dashboard functionality.

Describe the solution you'd like

The dashboard should only apply the default storage class to new workspaces as they are created. This would allow using non-default storage classes for workspaces and ensure that existing workspaces remain unchanged after updating the CheCluster storage class property. Once a workspace has been created, the storage-type defined in the Devfile should be used.

In general, since switching storage type results in losing data stored in the previous storage type anyways, the general advice should be to re-create the workspace with the new storage type from the get-go; updating existing workspaces has the same user-facing effect but also causes complications internally.

Describe alternatives you've considered

The dashboard could attempt to follow similar heuristics to the DevWorkspace Operator to decide if it is safe to update the storage type, though these are more complicated:

• Switching from ephemeral to another storage type is permitted, as data is lost when the workspace stops anyways. However, this may not be what the user wants if they've explicitly started an ephemeral workspace.
• Switching between "" (empty/not defined), common and per-user is permitted as these storage types use the same PVC
• Switching between storage types in general is permitted as long as the storage.controller.devfile.io finalizer is not present. This finalizer is added when a workspace begins using storage in the common/per-user and async cases and signifies that storage cleanup needs to be performed.

The full check is defined here.

If additional information would help the dashboard use this heuristic, it can be added to the DevWorkspace Operator.

Additional context

Related issue: https://github.com/eclipse/che/issues/21405