Volumes seem unusable via Kale

[prashanthb-ai]

2 things that completely took me by surprise:

1. Changing the volume size is not allowed. I get "Message: VolumeConfig: Field 'size' is expected of type 'int' but type 'str' was found." anytime I try to modify that field.
2. When I hand provision a volume and plug that into kale (via "Use Existing Volume"), kale assumes it can use it as the marshal volume. This seems extremely risky with parallel runs of the same pipeline using a RWX volume.

@StefanoFioravanzo wondering if you can you either:

• recommend an image that doesn't face these problems
• recommend a workaround
• point me at the code so I can fix it

I'm running in a cluster without Rok. I'm using gcr.io/arrikto/jupyter-kale:1.1-pre-328-g657cbfb749 (though I also tried 6f87224e-9f74f267).

$ gcloud container images list-tags gcr.io/arrikto/jupyter-kale
DIGEST        TAGS                     TIMESTAMP
a103ad9a9c6a  6f87224e-9f74f267        2021-03-05T21:57:56
1ed23b755237  1.1-pre-328-g657cbfb749  2021-02-15T15:51:38
1cd93dcf8eda  v1.1-pre-207-g6c6282e72  2021-01-20T22:30:37
910b49bfa674  4a5d7e63-9f74f267        2020-11-12T22:10:18
f9edb63efd94  a1dbe0cb-03892c7f        2020-09-28T15:25:00
068024fde59c  v0.5.0-47-g2427cc9       2020-08-11T20:49:02
0e675e9dda96  v0.5.0-40-g978b535       2020-08-11T15:58:22
d9705e9fc7be  v0.5.0-18-g5e6642f       2020-07-15T19:49:09
3c834c6172d9  v0.5.0-13-gf5308d4       2020-07-02T23:26:51
83a1a2b7cda3                           2020-07-02T14:39:24
128f1a239d10  1fa5c29,v0.5.0           2020-06-17T18:08:10

[sensor2sensor]

I am also facing the same issue, is there any workaround apart from updating the PVC size from k8?