Under PopOS, docker run works, while gitlab-ci-local doesn't

[jg-cantaa]

TL;DR: This Issue happens under PopOS. The cause could not be pinpointed despite intensive debugging, so this Issue will stay unfixed. Currently the only workaround is to run kaniko with the --force flag. Using gitlab-ci-local --privileged will not work.

Minimal Project Structure Archive: Issue.zip

Expected behavior Like run.sh: the container builds successfully

Host information Linux: Pop!_OS 20.04 LTS gitlab-ci-local 4.27.1

Note This issue requires the same setup like #341, however the error is a new one.

[firecow]

@jg-cantaa https://github.com/firecow/gitlab-ci-local/releases/tag/4.28.1 Can you try out this version please, and reopen if the bug still persists. Works on my machine, with 4.28.1

[jg-cantaa]

Sadly the Issue persists, even with 4.28.1.

I have tried removing the dockerimages to redownload them. However I don't know what else I could try in order to get it working. Please let me know if you want me to test other workarounds/fixes.

[firecow]

@jg-cantaa Hmm.. Why the hell can't I reproduce this... What docker version are you running ?

[jg-cantaa]

Docker version 20.10.7, build 20.10.7-0ubuntu5.1

[bcouetil]

Hello 😊

I may try to reproduce to help the cause, if I find time.

Does it work with the --force flag ?

[firecow]

@jg-cantaa

---
test:
  needs: []
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - |
      cat <<EOF >> ${CI_PROJECT_DIR}/Dockerfile
      FROM ubuntu
      EOF
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --no-push

Can you try this out, real quick...

[jg-cantaa]

Yes it does, but that kind of defeats the purpose of using gitlab-ci-local. It should just work. It should not require --force, since a normal docker container also does not require --force. Now, It could also be the case, that it's not worth it to fix this, simply because Kaniko requires some obscure thing to indicate it running in a container, which might not be that easy to reproduce.

[firecow]

Yes it does, but that kind of defeats the purpose of using gitlab-ci-local. It should just work. It should not require --force, since a normal docker container also does not require --force. Now, It could also be the case, that it's not worth it to fix this, simply because Kaniko requires some obscure thing to indicate it running in a container, which might not be that easy to reproduce.

This must be fixed, lots of people use Kaniko.

Can you try my example real quick.

[jg-cantaa]

Still...

I used this gitlab-ci-local test with this ci yml:

stages:
  - build_dockerfile
  - test
Construct_Docker_Image:
  stage: build_dockerfile
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --no-push

test:
  stage: test
  needs: []
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - |
      cat <<EOF >> ${CI_PROJECT_DIR}/Dockerfile
      FROM ubuntu
      EOF
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --no-push

[firecow]

This is mine @jg-cantaa

@bcouetil Can you try running the test: job from above as well

[jg-cantaa]

Are there any other programs with specific versions or packages that could influence this kind of behavior?

[firecow]

@jg-cantaa That's what I'm wondering

[firecow]

Do you have .gitlab-ci-local-env or .gitlab-ci-local.yml located in CWD @jg-cantaa ?

[jg-cantaa]

Nope, using the plain directory as mentioned in:

Minimal Project Structure Archive: Issue.zip

(the plain node thing was just autocomplete)

Edit: and this:

$ ts-node -v
v10.3.0

[firecow]

@jg-cantaa Ok, make a new dir, and only put a .gitlab-ci.yml with the test: inside and call gitlab-ci-local test Just so we are 2000% sure nothing file/folder specific is fucking us.

[firecow]

Why are we talking about ts-node -v now ? @jg-cantaa

[firecow]

You need to have gitlab-ci-local installed in /usr/local/bin/gitlab-ci-local don't run it from a checkout or anything.

[jg-cantaa]

Why are we talking about ts-node -v now ? @jg-cantaa

I just thought it might be useful. I installed as usual via npm install -g gitlab-ci-local.

[firecow]

@jg-cantaa Yes, it could become useful later on.

Right now, we just need to see output similar to this. When moving to a new non-git folder

[jg-cantaa]

[firecow]

mjn@mjn-laptop:~$ cat /etc/docker/daemon.json 
{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true,
  "registry-mirrors": ["https://mirror.gcr.io"]
}

Here is mine, what's yours :laughing:

[jg-cantaa]

this file does not exist on my machine.

[bcouetil]

It might be a PopOS issue...

[firecow]

Hmm.... I'm pretty sure I have a dev that uses PopOS, I'll try his machine in the morrow 😁

Its Ubuntu/Debian based, so I find it unlikely, but lets eliminate that.

[firecow]

It really looks like something todo with PopOS now :smile:

My PopOS co-worker is getting it.

[firecow]

@jg-cantaa Can you print your bash version please

bash --version

[jg-cantaa]

~$ bash --version
GNU bash, Version 5.1.8(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2020 Free Software Foundation, Inc.
Lizenz GPLv3+: GNU GPL Version 3 oder jünger <http://gnu.org/licenses/gpl.html>

Dies ist freie Software.  Sie darf verändert und verteilt werden.
Es wird keine Garantie gewährt, soweit das Gesetz es zulässt.

[firecow]

@jg-cantaa Would you by any chance have selinux installed ?

[jg-cantaa]

No.

~$ selinux
selinux: Befehl nicht gefunden.

~$ apt list selinux
Auflistung… Fertig

Edit: Maybe?

~$ apt list *selinux
Auflistung… Fertig
android-libselinux/impish 10.0.0+r36-1 amd64
python3-selinux/impish 3.1-3build2 amd64
python3-selinux/impish 3.1-3build2 i386
ruby-selinux/impish 3.1-3build2 amd64
ruby-selinux/impish 3.1-3build2 i386

[firecow]

Try apt list *selinux*

Also does gitlab-ci-local --priviledged get rid of the warning?