How use Fairing on Kubeflow running on local cluster?

[tarun0]

/kind bug

What steps did you take and what happened: I'm trying to setup Fairing for onPrem cluster using Microk8s. Fairing documentation states that Fairing uses docker to build/push images. However, the custom Jupyter notebook doesn't seem to run docker daemon showing error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? whenever docker commands are used.

What did you expect to happen: Fairing should work on local cluster's custom Jupyter notebook too.

Anything else you would like to add: Is it possible to make a custom Jupyter image which can be used with Kubeflow? I managed to run Docker commands on the custom image running locally by bind mounting the volume to provide access to var/run/docker.sock from host system; i.e. using docker run -v /var/run/docker.sock:/var/run/docker.sock ...

How can such a thing be accomplished while running the custom Jupyter image from Kubeflow instead of running from host using docker run commands? Or is that incorrect approach/infeasible?

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"): 0.7.0
• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): v1beta1
• Minikube version:Running on Microk8s version 1.14
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:41:00Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:32:32Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"

NOTE: If you are using fair from master, please provide us the git commit hash.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[DdMad]

I got the exactly the same issue after installing docker. It seems that there is some issue when installing docker inside a docker container, but don't know how to fix it. docker run -v /var/run/docker.sock:/var/run/docker.sock might be a workaround, but for my case is not ideal :(.

[marcindulak]

What about using append builder. https://github.com/kubeflow/fairing/blob/7e4c23c47a6b0544a722d04e1f9910c2f283a4f6/kubeflow/fairing/builders/append/append.py#L20-L22

Does the small example included in https://github.com/kubeflow/fairing/issues/460 work for you?

[mochiliu3000]

Found the same error, maybe can downgrade microk8s to 1.13, see this for detail: https://github.com/ubuntu/microk8s/issues/382

[mochiliu3000]

In case you want to do quick tests with fairing on microk8s, here is a dirty way to fix it @tarun0, @DdMad

1. Please downgrade your microk8s to 1.13 to get microk8s.docker (see https://github.com/ubuntu/microk8s/issues/382)
2. Fairing has hard coded docker-py to read local '/var/run/docker.sock' for docker client, hence need to create soft links for microk8s (see https://github.com/kubeflow/kubeflow/issues/2347) ln -s /var/snap/microk8s/common/var/lib/docker /var/lib/docker ln -s /var/snap/microk8s/current/docker.sock /var/run/docker.sock
3. Fairing uses local '~/.kube/config' to deploy jobs, need to create that file for microk8s by runing mkdir -p ~/.kube microk8s.kubectl config view > ~/.kube/config

And then I can build and deploy job with 'builder=docker':

[mochiliu3000]

What about using append builder. https://github.com/kubeflow/fairing/blob/7e4c23c47a6b0544a722d04e1f9910c2f283a4f6/kubeflow/fairing/builders/append/append.py#L20-L22

Does the small example included in #460 work for you?

@marcindulak Append builder is also working for microk8s v1.13, but not working for 1.14 and higher.