revisit persistent storage model

[joel-bluedata]

In K8s land, persistent shared storage is needed not only for migration, but also if we want to support a pause/restart kind of behavior that doesn't lose disk state.

I think we have some leftover "old thinking" about persistent storage that needs to be revisited. Questions like:

• [x] Should we really have default directories or always explicitly list them all? (answer: defaults are good)
• [x] App CR should probably should be allowed to specify the directories per-role.
• [x] Can/should we support directories that are not top-level dirs?
• [ ] How "optional" is persistent storage really; what is the correct interaction between site default, app requirements, and deploy-time choices?

May require issue #5 (KD site settings).

[swamibluedata]

all default directories /etc, /opt, /var and /usr are required for bdvcli and appconfig to work after migration. we can move all the guestconfig related files to a single folder.

[swamibluedata]

How "optional" is persistent storage really; what is the correct interaction between site default, app requirements, and deploy-time choices?

• Site Default must specify one or more storaClass names that is setup (we don't have this today)
• App requirements should indicate which directories are required for a successful migration
• deployment time should purely indicate which storageClass and size to use.

[joel-bluedata]

BTW @riteshja has signed up to look at the KD administrative configuration stuff (issue #5).

[joel-bluedata]

Moving to 0.3 for the remaining design issues.