Update k8s-api to v1.21 (to support ephemeral volumes)

[karpoftea]

It would be great if spark application could use generic ephemeral volumes to store shuffle/cached data (because of transient nature of this data). This option is more storage efficient comparing to an emptyDir (another type of ephemeral volume) because you can adjust storage size on a per application basis. It is more reliable then classic persistentVolumeClaim in case of disaster recovery (e.g. AZ goes down) because failed job will allocate volume in another AZ without problems (classic persistentVolumeClaim will complain because it will be bound to PV that reside in dead AZ, and pod will not schedule). Unfortunately operator currently supports k8s.io/api v0.19.6, while generic ephemeral volumes were introduces in v.0.21.x. To support this feature we should consider update k8s api version.

[ArshiAAkhavan]

any update on this issue?

we really need this feature in our cluster since 50MB is not enough for shuffling