Running on Gitlab on Kubernetes

[niklaskorz]

Gitlab runners support using Kaniko in their pipelines: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html

Is there any way to use Makisu like this (i.e., without having to pass the Docker socket)?

[yiranwang52]

Makisu doesn't need docker socket, it's in the example script just so image could be load into docker daemon for easier local testing.

We have instructions about how to run Makisu as a kubernetes job: https://github.com/uber/makisu/tree/master/examples/k8s

[niklaskorz]

Ah indeed, my bad. I'll report back if I have a working gitlab ci config :)

[yiranwang52]

That would be great :)

[niklaskorz]

Seems like it won't work with the provided FROM scratch image, but making a similar one based on alpine should work. "The kaniko debug image is recommended (gcr.io/kaniko-project/executor:debug) because it has a shell, and a shell is required for an image to be used with GitLab CI/CD" (cited from the above link).

[yiranwang52]

hmm that's a weird requirement. I will create such a image later.

[niklaskorz]

Makusi itself appears to be working, but it fails at pushing the image to the Gitlab docker registry. I'm creating the registry config as part of the pipeline in my gitlab ci config, but it fails with a 401 unauthorized error (job log).

[evelynl94]

@niklaskorz "{\"$CI_REGISTRY\":{\".*\":{\"basic\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}}" doesnt look like the correct config...could you try adding security?

[niklaskorz]

@evelynl94 Ah indeed, sorry. It's working now, thanks!

[yiranwang52]

thanks for validating this for us - it's something we always wanted to do but never got the chance to.

[hatemosphere]

sorry for breaking in, we're also trying to make makisu work in Gitlab, but our case is a bit different: we are running private registry. and when i pass this yaml file as a --registry-config:

gitlab-service.example.internal:8443:
  .*:
    security:
      basic:
        username: valid-login
        password: valid-password

makisu is giving me the following output:

{"level":"error","ts":1545040666.3546736,"msg":"Failed to push cache: push layer sha256:72e5d10f50a5ec1f32a051deff49d4e2670899ec686ed70d835748eefeab320f: check layer exists: gitlab-service.example.internal:8443/devops/docker-vault (sha256:72e5d10f50a5ec1f32a051deff49d4e2670899ec686ed70d835748eefeab320f): get security opt: basic auth: ping v2 registry: network error: Get http://gitlab-service.example.internal:8443/v2/: EOF; push layer sha256:0c35a791643b0b8df3363ebce713f9237f471cd2ea47fb9368c2c7837a4e61ab: check layer exists: gitlab-service.example.internal:8443/devops/docker-vault (sha256:0c35a791643b0b8df3363ebce713f9237f471cd2ea47fb9368c2c7837a4e61ab): get security opt: basic auth: ping v2 registry: network error: Get http://gitlab-service.example.internal:8443/v2/: EOF; push layer sha256:a63202e22a8bb0ea259849750829be956caee7596a976fad065b62c74d25a9bf: check layer exists: gitlab-service.example.internal:8443/devops/docker-vault (sha256:a63202e22a8bb0ea259849750829be956caee7596a976fad065b62c74d25a9bf): get security opt: basic auth: ping v2 registry: network error: Get http://gitlab-service.example.internal:8443/v2/: EOF"}

i wonder if it breaks because of non-standard https port 8443 and how to force it to use https scheme instead of http. @yiranwang52

[evelynl94]

@hatemosphere which version of makisu are you using? 0.1.1 and later should use https by default.

[hatemosphere]

@evelynl94 thanks for the response! i was using 0.1.6. tried latest master branch, still no luck:

{"level":"info","ts":1545122036.3965364,"msg":"* Started pushing image gitlab-service.example.internal:8443/devops/docker-vault:c9df2b111014663751e95ea927de51b1fb840a23"}
2018/12/18 08:33:56 Command failure: failed to push image: failed to push image: check manifest exists for image gitlab-service.example.internal:8443/devops/docker-vault:c9df2b111014663751e95ea927de51b1fb840a23: get security opt: basic auth: ping v2 registry: network error: Get http://gitlab-service.example.internal:8443/v2/: EOF

is it trying to execute this code? https://github.com/uber/makisu/blob/master/lib/registry/security/basicauth.go#L60 as i understand, http scheme is hardcoded here https://github.com/uber/makisu/blob/master/lib/registry/security/basicauth.go#L30

[evelynl94]

@hatemosphere ah you are right! Dockerhub returns unauthorized with a different https location for handling its credential magic on the ping request. I think maybe your registry enforces https on the first ping...Let me see if I can change the ping to https so it works for both cases.

Just curious, how did you set up the registry? Do you use your own client and ca certs + basic auth?

[hatemosphere]

@evelynl94 yeah, Gitlab has it's own magic with registry hosting and nginx on top of it. from the client's perspective there is basic auth and trusted CA for internal domain. TLS is being terminated by LB, on top of nginx (so there is no TLS client auth). and yes, we have only https port opened, so 'http://%s' in code won't work for us (ping, image push/pull etc.). well, somehow vanilla Docker and Kaniko worked for us, but i still wonder what is "best practice" here. should we maybe expose registry on http and force https redirection?

[evelynl94]

@hatemosphere https probably makes more sense as most basic auth implies you use TLS.

[hatemosphere]

@evelynl94 thanks, looks like ping works! but these https://github.com/uber/makisu/blob/master/lib/registry/client.go#L221 etc. are still http :)

{"level":"info","ts":1545170725.7352955,"msg":"* Started pushing image gitlab-service.example.internal:8443/devops/docker-vault:af7823f630ce8f9dde3ea8b81968b41e860a6b61"}
2018/12/18 22:05:29 Command failure: failed to push image: failed to push image: push layer sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde: check layer exists: gitlab-service.example.internal:8443/devops/docker-vault (sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde): check manifest exists: network error: Head http://gitlab-service.example.internal:8443/v2/devops/docker-vault/blobs/sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde: EOF;

[evelynl94]

@hatemosphere It modifies the scheme with https. This probably happens when the https failed and fell back to http. Could you verify if there was a https request to the registry?

[hatemosphere]

@evelynl94 umm there is nothing in logs (that was the first line of error log). can i increase verbosity somehow? plan b would be to capture traffic 🔨

[evelynl94]

@hatemosphere I can add some more debug logs in our client too (I removed it because it a bit noisy during http to https migration for our registry). I was wondering if you can see nginx access or error log on your side.

[hatemosphere]

@evelynl94 uhh, nevermind, that was pebcak - i forgot to add our private CA certs to the CA bundle. just tested it and it works flawlessly, thanks a lot for your help!

[evelynl94]

@hatemosphere oh thats great!! no problem :)

[desaintmartin]

It would be nice if you share a how-to about this, I think this is a great use case of makisu.

[hatemosphere]

@desaintmartin i will contribute some kind of tutorial soon ;)

[niklaskorz]

I've also updated my example repository to use the new official alpine image: https://gitlab.com/niklaskorz/nkchat/blob/master/.gitlab-ci.yml