FreeBSD port

[akhramov]

trafficstars

Buildkit code is mostly generic enough to support FreeBSD, however there are some quirks / infrastructural pieces that need to be addressed for full support, to name some

• contenthash.NewFromStat attempts to set Devmajor / Devminor for regular files, assuming that RDev is zero for regular files. Unlike on Linux, it's not the case for FreeBSD.

• containerdexecutor.Run uses bind mounts for rootfs. Bind mounts are not supported in FreeBSD and we should use nullfs instead

• There is no CI job to run tests on FreeBSD

• Some dependencies weren't ported

This change

• ports buildkit to FreeBSD
• bumps tonistiigi/fsutil and docker/docker versions
• introduces a CI job to run unit tests on FreeBSD.

Signed-off-by: Artem Khramov [email protected]

[akhramov]

A question: is there a way to tell containerd to use a specific runtime?

[tonistiigi]

A question: is there a way to tell containerd to use a specific runtime?

https://pkg.go.dev/github.com/containerd/containerd#WithDefaultRuntime

[akhramov]

Thanks for the review!

To outline things left to be done:

• [x] For FreeBSD default the runtime to runj ("wtf.sbk.runj.v1")
• [x] Investigate and fix CI failures

[tonistiigi]

To outline things left to be done:

Heads up that if you also want to include buildkitd in binaries then (at least most of the) integration tests need to also run in CI first(skipped atm). For only buildctl this is not needed. So you may want to do this in 2 parts: fix unit tests in current PR and we will merge the client support and then work on buildkitd, runtime and enabling integration tests.

Full matrix is not needed for freebsd integration tests. Single worker or random worker is fine. For the workers, oci worker needs to be supported, containerd is optional.

[akhramov]

For the workers, oci worker needs to be supported, containerd is optional.

Honestly, I tested just containerd support locally. I am going to test OCI workers & enable both in CI.

---

As for unit tests, it depends on https://github.com/moby/moby/pull/42903

[tonistiigi]

Feel free to add linux build tag to https://github.com/moby/buildkit/blob/master/util/testutil/integration/dockerd.go unless you actually want to run tests through docker as well.

[akhramov]

Rebased the branch. Moving to draft, since I didn't have adequate amount of resources lately to finish this.

[AkihiroSuda]

Needs rebase

[akhramov]

@crazy-max, I'm sorry if you are a wrong person to ping, but can you please take a look?

[akhramov]

I will be looking into the CI failures later this week. Can't reproduce this locally.

[akhramov]

CI failed due to vagrant rate-limiting.

CI is green in the fork, though: https://github.com/akhramov/buildkit/runs/6893833060?check_suite_focus=true

[crazy-max]

Looks like coverage file is missing: https://github.com/akhramov/buildkit/runs/6893833060?check_suite_focus=true#step:7:17

Warning: No files were found with the provided path: ./coverage. No artifacts will be uploaded.

Not an hard blocker though, we can see how to mount back coverage folder to the client in a follow-up.

[akhramov]

Looks like coverage file is missing

Should be resolved, see https://github.com/akhramov/buildkit/runs/6925326234?check_suite_focus=true

[akhramov]

Re: the failure

mount_nullfs: /run/containerd/io.containerd.runtime.v2.task/buildkit/nbxh1elkrwu64xuir6pmmjrtw/rootfs: Resource deadlock avoided\n

Looks like it is intermittent, but I don't want to introduce flaky CI pipeline, so I will find a solution.

[thaJeztah]

looks like the second commit is now missing a DCO sign-off; probably because you clicked the "apply suggestion" button; could you squash the two commits?

[akhramov]

could you squash the two commits?

Oh, of course, thanks!

[akhramov]

Re: the failure

mount_nullfs: /run/containerd/io.containerd.runtime.v2.task/buildkit/nbxh1elkrwu64xuir6pmmjrtw/rootfs: Resource deadlock avoided\n

Looks like it is intermittent, but I don't want to introduce flaky CI pipeline, so I will find a solution.

That was https://github.com/samuelkarp/runj/issues/30 and now is resolved. I think we can proceed with the PR.

[akhramov]

@AkihiroSuda can you take a look, please?

[akhramov]

hey @tonistiigi. Is there anything preventing us from moving forward with this one? I'm sorry for bugging you. :)

[akhramov]

Hey, @crazy-max can you kindly trigger the CI? I rebased the branch :)

[koobs]

❤️ @akhramov

[akhramov]

Alright, the CI should be green now, I hope :see_no_evil:

[arrowd]

Just curious if there any movement on this?

[crazy-max]

After rebasing on my side, I see some errors when building buildkit: https://github.com/crazy-max/buildkit/actions/runs/5193670842/jobs/9364508523#step:4:348

    fbsd_13_2: # github.com/docker/docker/pkg/chrootarchive
    fbsd_13_2: vendor/github.com/docker/docker/pkg/chrootarchive/archive_unix.go:31:8: undefined: goInChroot
    fbsd_13_2: vendor/github.com/docker/docker/pkg/chrootarchive/archive_unix.go:53:8: undefined: goInChroot
    fbsd_13_2: vendor/github.com/docker/docker/pkg/chrootarchive/diff_unix.go:45:8: undefined: goInChroot

We might need upstream changes in moby. Looks related to https://github.com/moby/moby/pull/44210 (cc @corhere)

[akhramov]

@crazy-max confirming.

Created a simple patch to fix that: https://github.com/moby/moby/pull/45724

[tonistiigi]

So this is blocked on Moby after the chroot updates there? Do we need to remove it from the milestone?

[crazy-max]

So this is blocked on Moby after the chroot updates there? Do we need to remove it from the milestone?

Yes I guess we need to move this out of the milestone while waiting for upstream changes in moby.

[tonistiigi]

Assuming we would follow up with the correct fix, I think I would be ok with switching from chrootarchive to the unsafe archive pkg with a build tag if you like. The FreeBSD support would be experimental and not supported anyway. Looks like the only place is https://github.com/moby/buildkit/blob/master/solver/llbsolver/file/unpack.go#L38

[akhramov]

I think I would be ok with switching from chrootarchive to the unsafe archive pkg with a build tag if you like.

We could do that. But I think we're pretty close to reaching a proper solution in https://github.com/moby/moby/pull/45724

[crazy-max]

@akhramov https://github.com/moby/buildkit/pull/4052 got merged if you want to do a rebase.