datamon icon indicating copy to clipboard operation
datamon copied to clipboard

Provide example use-case for datamon as a volume mount

Open jakedsouza opened this issue 5 years ago • 1 comments

Couple of useful examples could be

  • Mount as a shared k8s volume
  • Shared volume in a simple argo workflow

jakedsouza avatar Jun 11 '19 07:06 jakedsouza

want to make clear that the main use case of datamon as a volume mount is not the appearance of datamon as a FUSE filesystem mounted via a k8s Shared Volume or whatever.

the volume mount is useful because it enables streaming downloads.

with the volume mount, i can open() and seek() files before read()ing (all pseudo syscalls here) and only download they exact bytes (give or take 1-2k) my data-science simulation or whatever requires.

we do not have a similar story for incremental uploads like the volume mount provides for streaming downloads. #255 is part of the way there, imo.


also, a "dirty secret" of the streaming downloads that i only noticed within the last several months is that google cloud charges for operations on buckets not just storage space of the buckets themselves. so if you want a disadvantage of the feature (i.e. streaming downloads – the feature is not volume mount. that's the reason i've disabled all the FUSE stuff in my fork), the disadvantage is cost, and the cost is due to the billing structure of the data operations. i haven't done the math.

i think streaming is super useful. in fact, i think we need the "accelerator pods" setup in #255 to have something like streaming, except for uploads. i just don't think streaming is going to be useful for any personal use-case that i foresee.

ransomw1c avatar Sep 08 '20 00:09 ransomw1c