Add runPrivileged/runAsUser options, for running on more restricted/s…

[maartenvandenbogaard]

…ecured K8s clusters

These options make it possible to run the operator and es-clusters on a Kubernetes cluster, that has a Pod Security Policy in place, that:

• disallows running containers as root
• (and/or) disallows running containers in privileged mode

Note: the default elasticsearch image (upmcenterprises/docker-elasticsearch-kubernetes:6.1.3_0 as of writing) will not work if you don't run it as root (uid 0), as its wrapper script tries a 'ulimit -l unlimited', and eventually su-execs to elasticsearch user with uid 1000; both actions will fail. Setting ulimit should not be neccessary with IPC_LOCK/SYS_RESOURCE capabilities, however they get wiped when running a container as non-root. So running this image will require some modifications, e.g. chowning folders, setcap cap_ipc_lock=+ep on java binary + dependencies.

Most (recent) information on this topic that allowed me to solve the puzzle: https://medium.com/@thejasongerard/resource-limits-mlock-and-containers-oh-my-cca1e5d1f259

Change-Id: I600e9dd4a49cab15a289fc50cc2a605c83ac3aa9

[jjgraham]

we also want this change!

[abh]

The image also has to not chown and su-exec (as mentioned in the original comment).