Add Debian Packaging for Linux

[umsaleem]

Motivation and Context

Add Debian Packaging for Linux

Description

Currently, the Debian packages are generated from ALIEN that converts RPMs to Debian packages. This commit adds native Debian packaging for Debian based systems from ZFS on Linux.

Proper Debian style packaging can:

1. Make things easier for developers to have an actively managed packaging for Debian based systems.
2. Ease the testing and development of ZFS packages on Debian based systems.
3. It may also serve as a starting point for downstream projects to base their packaging on and add their own customization.
4. Current .deb packages do not include debug symbols and information to facilitate debugging process.

How Has This Been Tested?

Built the packages from new packaging. Installed the packages on Debian 11 and ran the test suite.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Performance enhancement (non-breaking change which improves efficiency)
• [ ] Code cleanup (non-breaking change which makes code smaller or more readable)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• [ ] Documentation (a change to man pages or other documentation)

Checklist:

• [x] My code follows the OpenZFS code style requirements.
• [ ] I have updated the documentation accordingly.
• [x] I have read the contributing document.
• [ ] I have added tests to cover my changes.
• [x] I have run the ZFS Test Suite with this change applied.
• [x] All commit messages are properly formatted and contain Signed-off-by.

[gdevenyi]

Can you please add a complimentary PR to https://github.com/openzfs/openzfs-docs to update the docs for debian based systems?

[umsaleem]

Can you please add a complimentary PR to https://github.com/openzfs/openzfs-docs to update the docs for debian based systems?

@gdevenyi Sure, will do.

[rincebrain]

Quite a long way to just fix debuginfo generation, heh. Well done.

My only suggestion is that it might be good to acknowledge that this is a minimally modified copy of the existing Debian maintainers' packaging for OpenZFS somewhere other than debian/copyright? I don't know that it's necessary, but I had to go looking to confirm you hadn't just deleted all references to them, since you emptied the changelog, and I'd rather acknowledge people too many times than too few. :)

[behlendorf]

@happyaron @xnox @aerusso @cdluminate would you mind commenting on this? We'd like to (finally) add proper debian style packaging to the upstream repository to make it easier to build real packages to test with. However, I want to make sure we avoid stepping on any toes or causing unnecessary problems downstream. What's the right way for us to go about this?

[happyaron]

One of the possible scenario I can think of is people who run a custom build of debian package would like to upgrade to a distro build at a later time, or vice versa. Some care of the package names and dependencies should be taken to make sure they are compatible with the distro ones, so that a valid upgrade path exists.

[pzakha]

I'm curious, did you use existing Ubuntu ZFS packages as reference?

[cdluminate]

Ubuntu ZFS package is pulled from Debian with slight modification. I'm continuously synchronizing Ubuntu modifications back to Debian package. But in fact most of the changes made by Ubuntu are Ubuntu-specific, like zsys support. They are not enabled on Debian.

[cdluminate]

I currently have several concerns regarding this: (1) is it possible to not place the debian directory in root directory? That conflicts with our workflow and will force us to modify the upstream source tarball. (2) how do we synchronize changes? (3) are your sure the 2.1.99 version number in debian/changelog is intended? Namely, a locally built, say 2.1.2 in fact will always suppress debian's 2.1.X release because dpkg --compare-versions 2.1.99 gt 2.1.X is always true.

[cdluminate]

Another debian package contributor is also around: @nabijaczleweli

[nabijaczleweli]

Yes, and I consider putting packaging in upstream sources idiotic at best, especially Debian-style packaging that takes over the source directory, especially especially if the downstream heavily patches this and has a particular binary package layout. If it were up to me I'd close this as severely out-of-scope; but it isn't, so I didn't post here at all.

But (a) if you put debian/ in the root then Debian will have to repack, so (b) it's best to move rpm -> contrib/rpm and put this in contrib/debian (or package/ if the Laboratory actively supports the RPM packaging), but then (c) it won't work OOB, obviously, so (d) don't inline Debian packaging.

If you want to build a Debian package off trunk for whatever reason then (a) configure for srpm and make deb-pkg or however we're spelling it via alien or (b) make disting and gbp import-orig in the salsa checkout (or apt source zfs-dkms; gbp import-dscs zfs*.dsc) like usual.

Don't @ me here again, thanks.

[umsaleem]

I'm curious, did you use existing Ubuntu ZFS packages as reference?

@pzakha I used existing ZFS on Linux packaging for Debian, that is available here.

[umsaleem]

I currently have several concerns regarding this: (1) is it possible to not place the debian directory in root directory? That conflicts with our workflow and will force us to modify the upstream source tarball. (2) how do we synchronize changes? (3) are your sure the 2.1.99 version number in debian/changelog is intended? Namely, a locally built, say 2.1.2 in fact will always suppress debian's 2.1.X release because dpkg --compare-versions 2.1.99 gt 2.1.X is always true.

@cdluminate

1. I think Debian packaging requires that debian directory should be placed on the root of the source directory. If we want to change the location of this directory and we would have to place it on root afterwards when we would want to build. Please let me know if we would want to do that.
2. We would either be required to synchronize this as is, or we can take it from this point and continue to maintain this.
3. Actually I had used 2.1.X as version number in debian/changelog but I changed it to 2.1.99 to make it consistent with META. If you recommend, I can change it back to 2.1.X.

[cdluminate]

(1): True. But as said by more than one person that, once debian is imported to the root of upstream source, we (debian) are forced to modify the upstream (openzfs) tarball when importing it to our workflow (salsa.debian.org) due to file conflict. The conflict means openzfs debian directory with old files will overwrite Debian's debian directory with our (debian) own new changes.

Let me say this again: if debian is imported to the source directory of openzfs/zfs, it will cause trouble for us (Debian).

[cdluminate]

(2) I can't agree with this as a good engineering practice. Assume Debian's packaging commit A has a bug. This snapshot is also imported to openzfs upstream. Is this bug debian bug or upstream bug? As time goes by, we debian fixed that bug and commited as B. But openzfs/zfs still use debian packaging commit A. Is this bug debian bug or upstream bug then? Debian packaging bug DO EXIST.

With alien, such packaging bugs are uniform across distribution. But if native debian packaging is imported, a bug will involve too much participants and may even be distribution-specific. Please be aware of such implication on maintenance burden for openzfs/zfs upstream. I think the upstream don't have to be distracted by such issue.

[umsaleem]

We do agree that this is not a good approach. Moving debian to contrib and rename the packages to openzfs-* can help clarify that this is not expected to be Debian zfsonlinux package set. I am working on that front. This can hopefully make things easier for many developers to have an actively maintained packaging for Debian based systems.

For synchronizing changes, any changes downstream that seem generally useful may be contributed back to upstream since git allows to merge into a sub-directory.

[cdluminate]

We do agree that this is not a good approach. Moving debian to contrib and rename the packages to openzfs-* can help clarify that this is not expected to be Debian zfsonlinux package set. I am working on that front. This can hopefully make things easier for many developers to have an actively maintained packaging for Debian based systems.

For synchronizing changes, any changes downstream that seem generally useful may be contributed back to upstream since git allows to merge into a sub-directory.

In that case please specify Conflicts+Replaces in debian/control to make sure locally-packaged openzfs-* packages cannot coinstall with zfs-*. We will have to add that from Debian side as well. https://www.debian.org/doc/debian-policy/ch-relationships.html

[Fabian-Gruenbichler]

as a former occasional Debian zfs contributor, and one of the downstream maintainers of ZFS packages derived from Debian's - another option instead of fully renaming the packages is to use a version suffix but keep the names/contents broadly in sync. we do this in our downstream distros (which are using the main Debian repositories and a downstream specific one, with the latter shipping higher versions of packages also shipped by Debian). Debian backports does the inverse (using a suffix that sorts lower than the actual package, so that upgrading from N-backports to N+1 works), Ubuntu uses a similar scheme for some packages.

obviously, this requires some amount of coordination but not that much:

• patches cherry-picked for compat/bug fix reasons in Debian obviously shouldn't be included upstream, but patches which exist to smooth building as part of the package should
• conflicts arising from upstream changes can be fixed in upstream packaging and Debian notified (since they likely need to be included when the upstream version including these changes is included in Debian)
• bugs in packaging fixed in Debian should be pushed upstream as well (either to the release branch only or both master and release branches, depending on where they are relevant)

IMHO the most important requirement is that there needs to be someone available for reviewing changes to the upstream/forked packaging that (really) understands how Debian packaging works, and has a good working relationship with downstream (Debian/Ubuntu) maintainers. not sure whether such a person exists or whether any of the existing downstream maintainers want/have capacity to fill that role.

dpkg-buildpackage operates on unpacked source trees, so if debian/ is not included directly in the source tree some sort of wrangling needs to be employed by the Makefile (e.g., copying the debian/ dir to the top level as part of building the packages), but that doesn't hurt that much and is probably preferable to annoying downstream packagers ;)

[umsaleem]

We do agree that this is not a good approach. Moving debian to contrib and rename the packages to openzfs-* can help clarify that this is not expected to be Debian zfsonlinux package set. I am working on that front. This can hopefully make things easier for many developers to have an actively maintained packaging for Debian based systems. For synchronizing changes, any changes downstream that seem generally useful may be contributed back to upstream since git allows to merge into a sub-directory.

In that case please specify Conflicts+Replaces in debian/control to make sure locally-packaged openzfs-* packages cannot coinstall with zfs-*. We will have to add that from Debian side as well. https://www.debian.org/doc/debian-policy/ch-relationships.html

Sure, will do.

[cdluminate]

Let me add my disclaimer here: As an active downstream Debian ZFS package maintainer, I will not keep track of the status and bugs of this copy of imported debian packaging. When new commits are made on the Debian side, I'll not inform the upstream to update the code copy. As long as (1) packaging incorporation does not interfere with the downstream (Debian) workflow, (2) copyright, changelog, credit to contributors and license are intact, I'm not going to stand in the way of anyone.

[cdluminate]

Another detail -- the debian packaging is licensed under GPL-2+ due to historical reasons: https://salsa.debian.org/zfsonlinux-team/zfs/-/blob/master/debian/copyright#L194-204 -- please be aware.

[xnox]

If motivation is described a bit better, different solutions and approaches can be done to make things easy to achieve the desired goal and make it easy to share packaging.

Are you planning to start providing snapshot / daily builds of openzfs for debian/ubuntu? Or is it an advanced-user / developer enhancement to allow one to self-build .debs and install them to see how things are interacting?

From Ubuntu point of view, it would be nice to use different package names, and maybe even have extra conflicts. This is because systems that are installed with zsys feature is currently trivially to irreversably make unbootable, and loose automatic snapshots and rollbacks. Thus ideally the packaging upstream should intentionally use different package names. Also in Ubuntu, zfs.ko is signed to support secureboot with enforced module signature checking. Care should be taken to ensure that packages provided here, do not allow installing dkms based zfs.ko module without signatures from an enrolled MOK key.

[umsaleem]

If motivation is described a bit better, different solutions and approaches can be done to make things easy to achieve the desired goal and make it easy to share packaging.

Are you planning to start providing snapshot / daily builds of openzfs for debian/ubuntu? Or is it an advanced-user / developer enhancement to allow one to self-build .debs and install them to see how things are interacting?

From Ubuntu point of view, it would be nice to use different package names, and maybe even have extra conflicts. This is because systems that are installed with zsys feature is currently trivially to irreversably make unbootable, and loose automatic snapshots and rollbacks. Thus ideally the packaging upstream should intentionally use different package names. Also in Ubuntu, zfs.ko is signed to support secureboot with enforced module signature checking. Care should be taken to ensure that packages provided here, do not allow installing dkms based zfs.ko module without signatures from an enrolled MOK key.

The need for proper Debian style packaging was felt to:

(1) Make things easier for developers to have an actively managed packaging for Debian based systems. (2) Ease the testing and development of ZFS packages on Debian based systems. (3) It may also serve as a starting point for downstream projects to base their packaging on and add their own customization. (4) Current .deb packages for OpenZFS do not include debug symbols and information to facilitate debugging process.

Ease of development, testing and debugging for Debian based systems is the only goal here.

[cdluminate]

As for (4). Debian packages built using the standard process do not include debug symbols neither. Debugging information will be split into -dbgsym packages. Why not try to prevent rpm/alient from stripping binaries at the first place? But prefer adding 3000+ lines of code?

[umsaleem]

As for (4). Debian packages built using the standard process do not include debug symbols neither. Debugging information will be split into -dbgsym packages. Why not try to prevent rpm/alient from stripping binaries at the first place? But prefer adding 3000+ lines of code?

I did take that route first. But after discussing it on the slack channel for OpenZFS, it was recommended we take this as a shorter path than trying to convince alien to emit debugging information. And that we should try to switch to a less alien based generation system.

[cdluminate]

I see. Sounds good to me.

[cdluminate]

My mind has changed a little bit. For serious debian packaging bugs I'll notify the upstream if this PR eventually gets merged and suggest fix. Minor ones do not worth that and you may cherry pick by yourselves.

[rincebrain]

As the one who last submitted a patch when he found a bug in alien causing problems, +1 on finding a solution for generating deb packages that is not reliant on alien, be it actual native packaging format or (though it doesn't seem likely now) if we couldn't find a solution that wouldn't cause Debian's maintainers more work, I believe at least one person maintains their own alternate scripting for generating OpenZFS debs from specfiles that isn't in front of me at the moment...

[Conan-Kudo]

As for (4). Debian packages built using the standard process do not include debug symbols neither. Debugging information will be split into -dbgsym packages. Why not try to prevent rpm/alient from stripping binaries at the first place? But prefer adding 3000+ lines of code?

I did take that route first. But after discussing it on the slack channel for OpenZFS, it was recommended we take this as a shorter path than trying to convince alien to emit debugging information. And that we should try to switch to a less alien based generation system.

It's not a question of convincing alien to emit debugging information. You just have to tell rpmbuild to not strip the symbols before alien runs on the built packages. --define '__strip /bin/true' would have been enough.

[Conan-Kudo]

That said, Debian generally prefers that upstream projects not have a debian folder, because it's a pain for them to overwrite it when they have to mangle the source tree for their own packaging.

Additionally, I do have a way to build Debian packages from an RPM spec file natively, as I've been doing this for years. I haven't contributed it upstream because I wasn't sure if people wanted it.

[umsaleem]

Since including Debian packaging in upstream is not a good practice and it creates a over head for maintainers, I have decided not to pursue this path further.

Thanks to @Conan-Kudo, debug symbols can be included in existing packaging.