make dev-env on Intel Mac broken because of --load flag

[longwuyuan]

• There have been some discussions about this on community-meeting and slack
• make dev-env is broken on Intel macOS Monterey with docker context working fine and even kind create cluster working
• The error reported is unknown flag: --load
• Contributors with Intel Mac depend on this
• Not sure if this is a problem on M1 Mac or Linux

k8s.io/apiserver/pkg/server/healthz
k8s.io/ingress-nginx/pkg/metrics
k8s.io/ingress-nginx/cmd/nginx
+ echo 'Building k8s.io/ingress-nginx/cmd/dbg'
Building k8s.io/ingress-nginx/cmd/dbg
+ go build -v -trimpath '-ldflags=-buildid= -w -s     -X k8s.io/ingress-nginx/version.RELEASE=1.0.0-dev     -X k8s.io/ingress-nginx/version.COMMIT=git-f0490cbfb     -X k8s.io/ingress-nginx/[email protected]:longwuyuan/ingress-nginx.git' -o rootfs/bin/amd64/dbg k8s.io/ingress-nginx/cmd/dbg
go: downloading github.com/spf13/cobra v1.5.0
github.com/spf13/cobra
k8s.io/ingress-nginx/cmd/dbg
+ echo 'Building k8s.io/ingress-nginx/cmd/waitshutdown'
Building k8s.io/ingress-nginx/cmd/waitshutdown
+ go build -v -trimpath '-ldflags=-buildid= -w -s     -X k8s.io/ingress-nginx/version.RELEASE=1.0.0-dev     -X k8s.io/ingress-nginx/version.COMMIT=git-f0490cbfb     -X k8s.io/ingress-nginx/[email protected]:longwuyuan/ingress-nginx.git' -o rootfs/bin/amd64/wait-shutdown k8s.io/ingress-nginx/cmd/waitshutdown
k8s.io/ingress-nginx/cmd/waitshutdown
removing old image gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev
Error: No such image: gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev
Building docker image (amd64)...
unknown flag: --load
See 'docker build --help'.
make[1]: *** [image] Error 125
make: *** [dev-env] Error 2
m@mypc [~/Documents/github/longwuyuan/ingress-nginx] make-broken
% make clean && make dev-env

• docker and kind are working

% docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

% docker images
REPOSITORY                                      TAG       IMAGE ID       CREATED       SIZE
gcr.io/k8s-staging-ingress-nginx/nginx          <none>    8bea7fc9758e   4 days ago    161MB
registry.k8s.io/ingress-nginx/e2e-test-runner   <none>    8b1253c38468   4 weeks ago   1.17GB
kindest/node                                    v1.24.2   d8d5552a11d9   4 weeks ago   912MB

% kind get clusters  
No kind clusters found.

% kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

% kind get clusters
kind
% k cluster-info
Kubernetes control plane is running at https://127.0.0.1:63495
CoreDNS is running at https://127.0.0.1:63495/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

% k get ns
NAME                 STATUS   AGE
default              Active   26s
kube-node-lease      Active   28s
kube-public          Active   28s
kube-system          Active   28s
local-path-storage   Active   21s
% 

/triage accepted /area stablization /priority important-soon /assign @strongjz @tao12345666333

[k8s-ci-robot]

@longwuyuan: The label(s) area/stablization cannot be applied, because the repository doesn't have them.

In response to this:

• There have been some discussions about this on community-meeting and slack
• make dev-env is broken on Intel macOS Monterey with docker context working fine and even kind create cluster working
• The error reported is unknown flag: --load
• Contributors with Intel Mac depend on this
• Not sure if this is a problem on M1 Mac or Linux

k8s.io/apiserver/pkg/server/healthz
k8s.io/ingress-nginx/pkg/metrics
k8s.io/ingress-nginx/cmd/nginx
+ echo 'Building k8s.io/ingress-nginx/cmd/dbg'
Building k8s.io/ingress-nginx/cmd/dbg
+ go build -v -trimpath '-ldflags=-buildid= -w -s     -X k8s.io/ingress-nginx/version.RELEASE=1.0.0-dev     -X k8s.io/ingress-nginx/version.COMMIT=git-f0490cbfb     -X k8s.io/ingress-nginx/[email protected]:longwuyuan/ingress-nginx.git' -o rootfs/bin/amd64/dbg k8s.io/ingress-nginx/cmd/dbg
go: downloading github.com/spf13/cobra v1.5.0
github.com/spf13/cobra
k8s.io/ingress-nginx/cmd/dbg
+ echo 'Building k8s.io/ingress-nginx/cmd/waitshutdown'
Building k8s.io/ingress-nginx/cmd/waitshutdown
+ go build -v -trimpath '-ldflags=-buildid= -w -s     -X k8s.io/ingress-nginx/version.RELEASE=1.0.0-dev     -X k8s.io/ingress-nginx/version.COMMIT=git-f0490cbfb     -X k8s.io/ingress-nginx/[email protected]:longwuyuan/ingress-nginx.git' -o rootfs/bin/amd64/wait-shutdown k8s.io/ingress-nginx/cmd/waitshutdown
k8s.io/ingress-nginx/cmd/waitshutdown
removing old image gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev
Error: No such image: gcr.io/k8s-staging-ingress-nginx/controller:1.0.0-dev
Building docker image (amd64)...
unknown flag: --load
See 'docker build --help'.
make[1]: *** [image] Error 125
make: *** [dev-env] Error 2
m@mypc [~/Documents/github/longwuyuan/ingress-nginx] make-broken
% make clean && make dev-env

• docker and kind are working

% docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

% docker images
REPOSITORY                                      TAG       IMAGE ID       CREATED       SIZE
gcr.io/k8s-staging-ingress-nginx/nginx          <none>    8bea7fc9758e   4 days ago    161MB
registry.k8s.io/ingress-nginx/e2e-test-runner   <none>    8b1253c38468   4 weeks ago   1.17GB
kindest/node                                    v1.24.2   d8d5552a11d9   4 weeks ago   912MB

% kind get clusters  
No kind clusters found.

% kind create cluster
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.24.0) 🖼 
✓ Preparing nodes 📦  
✓ Writing configuration 📜 
✓ Starting control-plane 🕹️ 
✓ Installing CNI 🔌 
✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

% kind get clusters
kind
% k cluster-info
Kubernetes control plane is running at https://127.0.0.1:63495
CoreDNS is running at https://127.0.0.1:63495/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

% k get ns
NAME                 STATUS   AGE
default              Active   26s
kube-node-lease      Active   28s
kube-public          Active   28s
kube-system          Active   28s
local-path-storage   Active   21s
% 

/triage accepted /area stablization /priority important-soon /assign @strongjz @tao12345666333

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/area stabilization

[LuckyDogg]

This problem also occurs on the M1 MAC, OS system is macOS Monterey

[longwuyuan]

@strongjz cc

[jaehnri]

Hi! I'm still facing this issue. However, when I simply remove the "--load" value from https://github.com/kubernetes/ingress-nginx/blob/main/Makefile#L59, it does work.

Does it make sense to open a PR removing it?

[strongjz]

It's works for me on my m1. Others it would load into the docker context.

I wonder if it's versions of docker as well?

[jaehnri]

Well, I'm not using M1...

$ System:
2,3 GHz Intel Core i9 8-Core
macOS Monterey 12.6

$ uname -m
x86_64

$ docker --version
Docker version 20.10.17, build 100c701

$ kind version
kind v0.15.0 go1.19 darwin/amd64

Maybe we should add the "--load" option just for M1s?

[jaehnri]

It seems like "--load" is a flag valid only for docker buildx build. I'm not sure why we are using it only for macOS builds though...

/assign

[strongjz]

Interesting, it looks like my docker build is always doing a docker buildx build

docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:04:45 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.12.0 (85629)
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:01 2022
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Jamess-MBP-2:distroless-build strongjz$ docker build 
ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build
Jamess-MBP-2:distroless-build strongjz$ 

[longwuyuan]

I think docker build from a prompt is completely different use-case.

At least for e2e test locally (or e2e test with limited scope set with $FOCUS=), make dev-env or make build image and all other make targets are calling run-in-docker.sh and make is what most contributors and users run, because it is integrated.

[longwuyuan]

i am not using Mac anymore