datajoint-python icon indicating copy to clipboard operation
datajoint-python copied to clipboard

Enforce to separate external storage based on external types, and provide a validator of each type

Open shenshan opened this issue 4 years ago • 1 comments

Blob, attach and filapath should be stored in separated dj.config['stores'] fields and they should be validated when configured. For example, cache should be required (or only allowed) for blob and attach store, and stage should be required for filepath.

In this case, we can separately control the default behavior when cleaning up the external contents. For example, for external blob, we should not allow delete_external_files=False.

shenshan avatar Oct 09 '20 15:10 shenshan

Several notes:

  • Transaction of external table deletion is users’ responsibility, but we would like to provide a lock on the external table while files are being deleted.
  • Separation of stores. Each store is for one external type, but we don’t have to force them to use different directories - but recommended (to be discussed more).

For now, we instruct the users to have separate directories for different external types (especially for blob/attach vs filepath).

shenshan avatar Oct 09 '20 17:10 shenshan