`docker build` fails when using BuildKit due to missing API

[samdoran]

Issue Description

When using DOCKER_BUILDKIT=1 docker build, the image build fails due to a missing API endpoint in podman-system-service.

The missing API endpoint is /v1.41/build/cancel.

> curl -X POST --unix-socket /Users/sdoran/.local/share/containers/podman/machine/podman-machine-default/podman.sock 'http://v1.41/build/cancel'
Not Found

When running docker buildx build -t test-image Dockerfile . the image builds successfully, but it should also work correctly when Docker is configured to build using buildkit using an environment variable without having to use the buildx build command.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine init --now
2. export DOCKER_HOST="unix:///$HOME/.local/share/containers/podman/machine/podman-machine-default/podman.sock"
3. DOCKER_BUILDKIT=1 docker build -t test-image -f Dockerfile .

Describe the results you received

The image fails to build with the following output:

[+] Building 0.0s (0/0)
failed to dial gRPC: unable to upgrade to h2c, received 404

The logs in the Linux VM show a 404:

Mar 17 12:28:05 localhost.localdomain systemd[1535]: Starting podman.service - Podman API Service...
Mar 17 12:28:05 localhost.localdomain systemd[1535]: Started podman.service - Podman API Service.
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="/usr/bin/podman filtering at log level info"
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="Setting parallel job count to 13"
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="Using systemd socket activation to determine API endpoint"
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="API service listening on \"/run/user/501/podman/podman.sock\". URI: \"/run/user/501/podman/podman.sock\""
Mar 17 12:28:05 localhost.localdomain podman[11260]: @ - - [17/Mar/2023:12:28:05 -0400] "HEAD /_ping HTTP/1.1" 200 0 "" "Docker-Client/20.10.23 (darwin)"
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="Failed Request: (404:Not Found) for POST:'/session'"
Mar 17 12:28:05 localhost.localdomain podman[11260]: 2023-03-17 12:28:05.329165813 -0400 EDT m=+0.027635009 image build
Mar 17 12:28:05 localhost.localdomain podman[11260]: @ - - [17/Mar/2023:12:28:05 -0400] "POST /v1.41/build?buildargs=%7B%7D&buildid=9ef18e5036fe8e1af85554ca6774c2d065ce6c5e4b092aaf424ed3b8fb568399&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile.minio&labels=%7B%7D&memory=0&memswap=0&networkmode=default&remote=client-session&rm=1&session=ranl46bckdvvtomv7lkn9p4zy&shmsize=0&t=local-minio&target=&ulimits=null&version=2 HTTP/1.1" 200 222 "" "Docker-Client/20.10.23 (darwin)"
Mar 17 12:28:05 localhost.localdomain podman[11260]: time="2023-03-17T12:28:05-04:00" level=info msg="Failed Request: (404:Not Found) for POST:'/v1.41/build/cancel?id=9ef18e5036fe8e1af85554ca6774c2d065ce6c5e4b092aaf424ed3b8fb568399'"

Describe the results you expected

The image should build successfully.

podman info output

host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-3.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: '
  cpuUtilization:
    idlePercent: 97.95
    systemPercent: 0.88
    userPercent: 1.17
  cpus: 4
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.1.14-200.fc37.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 423530496
  memTotal: 8201252864
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.fc37.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 2h 2m 34.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 7
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 7511818240
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 22
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676629538
  BuiltTime: Fri Feb 17 05:25:38 2023
  GitCommit: ""
  GoVersion: go1.19.5
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

> sw_vers
ProductName:		macOS
ProductVersion:		13.2.1
BuildVersion:		22D68

> docker version
Client:
 Cloud integration: v1.0.31
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:35:19 2023
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: linux/arm64/fedora-37
 Podman Engine:
  Version:          4.4.1
  APIVersion:       4.4.1
  Arch:             arm64
  BuildTime:        2023-02-17T05:25:38-05:00
  Experimental:     false
  GitCommit:
  GoVersion:        go1.19.5
  KernelVersion:    6.1.14-200.fc37.aarch64
  MinAPIVersion:    4.0.0
  Os:               linux
 Conmon:
  Version:          conmon version 2.1.6, commit:
  Package:          conmon-2.1.6-3.fc37.aarch64
 OCI Runtime (crun):
  Version:          crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/501/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  Package:          crun-1.8.1-1.fc37.aarch64
 Engine:
  Version:          4.4.1
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.19.5
  Git commit:
  Built:            Fri Feb 17 05:25:38 2023
  OS/Arch:          linux/arm64
  Experimental:     false

> podman version
Client:       Podman Engine
Version:      4.4.2
API Version:  4.4.2
Go Version:   go1.20.1
Git Commit:   74afe26887f814d1c39925a1624851ef3590e79c
Built:        Thu Feb 23 10:59:21 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.5
Built:        Fri Feb 17 05:25:38 2023
OS/Arch:      linux/arm64

[Luap99]

I would label this as feature, buildkit would be new complex API that we need to support. @mheon I know you looked at this before, WDYT?

[mheon]

There are two issues:

1. Missing APIs for Buildkit. In addition to this "cancel" endpoint, Buildkit also requires that the regular Build endpoint support upgrading, and passing Buildkit instructions directly.
2. Missing support in Buildah for Buildkit features. The Buildkit API mentioned in point 1 is sufficiently complex that Buildah presently does not support everything it can do. This is the main holdup at the moment - we can't really implement large portions of the Buildkit API until we have backend support in Buildah.

@flouthoc was looking into the missing features from point 2 and was going to create cards - did those ever get made?

[samdoran]

It's interesting that docker buildx build works but docker build with BuildKit enabled by other means does not. Evidently different API endpoints are being used by the docker CLI even though from a user perspective they seem functionally equivalent to "build this image using buildkit".

The image built using docker buildx build is correct. That seems to indicate enough of the backend functionality exists for a successful build.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

buildah can support most of the functionality required to match buildkit, but we have not implemented the API endpoint.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[didlawowo]

any news ? i have the same using pulumi

[flouthoc]

This is still not implemented but this needs to be implemented and prioritized. Thanks for bumping.

[sshuklao]

we are also facing the same issue, when can we expect this delivery?

[gebinic]

i run into the same issue

[lschierer]

I hit this today as well.