build/common: replacing mount_unionfs with mount_nullfs

[kubatyszko]

Unionfs isn't a "jail friendly" filesystem and it can't be mounted inside jails. Simply replacing the mount_unionfs with mount_nullfs helps to resolve this issue. Yes, I'm aware that mount_unionfs had a '-o below' option and mount_nullfs doesn't support that, but that's the compromise. So far I haven't had any other issues running the build system (make serial) in a jail other than the mount_unionfs problem. Using jail requires several security settings, such as 'amount.mount', 'allow.mount.nullfs', 'enforce_statfs=1' to allow mounting filesystems inside.

[fichtner]

Thanks for the suggestion. I agree with this, yet the "compromise" is the main issue: we should provide a read-only copy of the git tree to the build. I'm not sure if nullfs can already do that and if it can do that the ports directory probably at least needs a "safe" distfiles directory to load its distfiles from or dump them (this also includes making distfiles.sh caching work properly). I don't have enough time to chase this, but I'm happy to review and discuss to bring this to a mergeable state.

Cheers, Franco

[kubatyszko]

Ah, so that's the reasoning behind unionfs. Well, in a jail the requirement for read-only ports tree doesn't matter that much, so maybe some switch statement detecting we're in a jail and use nullfs, otherwise use unionfs ?

[fichtner]

# mount -r -t nullfs /usr/ports /usr/foo
# mkdir /usr/foo/distfiles
mkdir: /usr/foo/distfiles: Read-only file system

TBH, this is ok for me, but we need to address the distfiles dir use. Not much else should require write access except for debugging perhaps.

I just don't like that the write default creates and manipulates files in the target:

# mount -t nullfs /usr/ports /usr/foo
# mkdir /usr/foo/distfiles
# touch /usr/foo/distfiles/foobar
# umount /usr/foo
# ls /usr/ports/distfiles/foobar
/usr/ports/distfiles/foobar

Our future selves may thank us for it. :)

Cheers, Franco

[fichtner]

To give further pointers we need a DISTDIR assignment like WRKDIRPREFIX here:

https://github.com/opnsense/tools/blob/639306b756bf2ff3cca9fa2c038a753c178da418/config/25.7/make.conf#L40

And use it consistently and that should be all there is to it?

[kubatyszko]

My main intention behind using jail was to minimize impact to the base OS, one such impact is pkg version (I believe I was on 2.2.2 and Opnsense wanted 2.1.9 if not older), there's probably plenty more to keep track to build opnsense... FreeBSD itself I believe uses poudriere to manage jails for build systems if I'm not mistaken.