nerd icon indicating copy to clipboard operation
nerd copied to clipboard

Keep metadata in dataset custom resource also from the flex volume

Open advdv opened this issue 7 years ago • 2 comments

During initial implementation it became clear that we need to keep metadata for datasets. At the very least we will need to keep the filesize of a dataset and the jobs they are being used in. We also encountered the probable case that we might need to keep recent errors in there and maybe the state of the dataset (uploading, downloading?)

Updating this metadata needs to happen on both the client side as well as the flex volume side. This makes it attractive to use the dataset custom resource as the storage for this metadata.

Keep:

  • [x] Size
  • [ ] (Last) Flex volume error(s)

advdv avatar Jan 31 '18 14:01 advdv

Alternatively, maybe the flex volume error is available as part of the pod status that we're already fetching for the list overview. Using that would be lot less work but I'm not sure it is available there, maybe @Overv knows this?

advdv avatar Feb 12 '18 08:02 advdv

Having talked with @Overv this is more difficult than anticipated for several reasons: 1) The Pod information we're already fetching does not contain any flex volume errors. 2) The Kubernetes events do have the necessary information but cannot be easily filtered ONLY for pods/jobs managed by nerd. 3) Keeping our own errors in the dataset is error prone and doesn't capture all errors (such as wrongly configured flex volumes) or other mount types users would like to use.

Actual solution is to either gather events for a specific pod through nerd job logs or create a nerd job describe in the next milestone. This would also make room for many other events we wanted to capture in the first place.

advdv avatar Feb 12 '18 16:02 advdv