Added debian contents for building a deb package

[twojstaryzdomu]

Debian build directory contents, tested as valid for the latest pulseaudio release 15.0.

Using the debian contents, building a set of packages should take these steps, run from repository root:

# apt-get install -y build-essential bzr-builddeb dh-exec
# dpkg-checkbuilddeps 2>&1 \
| sed -E 's| \([^)]+\)||g' \
| grep -Po 'dependencies: \K.*' \
| xargs -r apt-get install -y
# bzr init
# bzr add debian
# bzr builddeb -v --dont-purge -- -nc -us -uc

NOTES: This PR includes two patches:

1. enabling SO_REUSEADDR for non-multicast addresses, which is necessary with pulseaudio started by a systemd socket
2. fixing pulseaudio includes path, which is needed to use PA libs installed via deb packages rather than compling PA from the ground up

Pulseaudio roc modules are built against the PA version currently installed in the environment. During the build, PA isn't fully compiled from the downloaded sources, only configured to generate the config.h file.

Caveat: requires openfec source to be installed in the OS. The debian package build configuration assume those are available (presumably installed as native package).

[gavv]

Hi! That's cool, thanks!

Regarding SO_REUSEADDR: I'll push a fix to master in upcoming days, so the patch wont be needed.

Regarding pulseaudio modules. In upcoming weeks, I plan to do the following:

• finish roc-pulse repo (including the issues you've opened and docs)
• remove pulseaudio modules from roc-toolkit repo
• merge develop into stable/master branch (it has quite a lot of changes)
• tag 0.2.0 release
• also make a release of go and java bindings and roc-pulse

I hope to finish this is in September of beginning of October. In this regard, what do we want to do with the part of debian package that handles pulseaudio modules? I wonder if you could open a PR to roc-pulse and move this part there?

[gavv]

Also, do you have any suggestions how we could easily publish prebuilt debian packages for users? (And preferably do it automatically in CI).

[gavv]

Another option: instead of having debian/ in both repos (toolkit and pulse), we can create a single repo for all debian packages for all projects. Let me know if you think this can work better. Though, probably this is not the best idea because every project can have its own versioning.

[twojstaryzdomu]

Hi, It makes sense to remove the patches, I'd been considering that since those are only interim code until it gets implemented or as workaround, like in the case of the other patch for the pulseaudio libs.

To your question on a CI workflow for building native debian packages, I have just the thing. It would works nicely without any adjustments if the code to build comes from the same repo as the debian directory.

The one thing I'd ask of you is if you'd be happy to fix the library discovery. Dependencies that are compiled at build time are a problem, you'd need to package them one by one, which is a waste of effort when you may simply use the ones that come with the OS. The reason being if any of those are missing at package install time, the code won't work and likely fail to install. Debian is notorious for having outdated libraries so using the latest releases of dependencies will likely lead to incompatibilities.

With that in mind, roc relies on openfec, which isn't available within debian. That package will need to be built first as a prerequisite. In summary, a user wanting to use pulseaudio roc module will need to install at least 3 packages: libopenfec, libroc & pulseaudio-module-roc.

A separate repo, say roc-debian, is the most sensible option if it would work with a CI workflow and want the roc-toolkit repo uncluttered and free of OS-specific code.

[gavv]

The one thing I'd ask of you is if you'd be happy to fix the library discovery.

Sure, I'll fix it!

With that in mind, roc relies on openfec, which isn't available within debian. That package will need to be built first as a prerequisite. In summary, a user wanting to use pulseaudio roc module will need to install at least 3 packages: libopenfec, libroc & pulseaudio-module-roc.

Given it a second thought, it seems that libopenfec will be never added (and should not...) to official debian / ubuntu repos for reasons described in https://github.com/roc-streaming/roc-pulse/issues/11. In short, the upstream is a kind of abandoned and we run unofficial low-maintained fork with several critical bug-fixes, just for roc.

Thus, maybe we want to treat openfec specially, as "internal" dependency of roc? And never ever try to search it on system. BTW openfec even doesn't have correct installation step actually :)

I could do the following:

• add openfec as submodule to roc
• build system will automatically clone this submodule for convenience
• unfortunately, standard Source code (zip) tarballs that github attaches to releases don't contain submodules; however, I can configure CI to attach extra asset to each release that will contain roc source code + all dependencies (say, roc-toolkit+dependencies.tar.gz) - you can find example of such "extra assets" here - and then you can use those tarballs for fully offline and "self-sufficient" builds

What do you think?

A separate repo, say roc-debian, is the most sensible option if it would work with a CI workflow and want the roc-toolkit repo uncluttered and free of OS-specific code.

What do you prefer? From my side, I don't have any problem with adding OS-specific code to this repo, so this is up to you.

[twojstaryzdomu]

Your openfec code being a fork is still fine, perhaps not for PPA but it will suffice, I'd keep it where it is. Submodules seem hack-y and bundling openfec within the libroc package will be a no-no.

But, I can save you the effort to integrate it. I've already thought of a way to solve the dependency problem as part of a single CI workflow:

1. Install any build dependencies.
2. Checkout your fork of openfec. Build libopenfec first. Install libopenfec-dev locally. Building this is quick.
3. Checkout roc-toolkit. Build libroc. Install the libroc-dev
4. Checkout roc-pulse. Build pulseaudio-module-roc.

Note we avoid building pulseaudio. The debian build config for roc-pulse will download the pulseaudio sources and configure the sources to generate config.h. No packages will be built, as pulseaudio is quite time consuming to build.

One caveat is roc-pulse will be tied to the major.minor release of pulseaudio available within the debian/ubuntu distibution used to build out the packages on github. IIRC debian doesn't update their pulseaudio releases so that's fine. I don't know about ubuntu. Do you think this would work?

Now, there remains the $1M question where the debian directories and the CI workflow will reside.

• On github, debian directories are often contained within the sources themselves (so that any user that clones the code repo can run dpkg-buildpackage). I'd be inclined to use that for simplicity.
• I'm also open to having roc-debian and openfec-debian. solely to keep the debian build configuration separate. This is also what debian does internally, their debian build code is kept separate from the app sources (e.g. http://deb.debian.org/debian/pool/main/p/pulseaudio/pulseaudio_14.2-2.debian.tar.xz). Building the code is more tricky, a user needs to use the so called merge mode for dpkg-buildpackage as the sources are contained separate from the source code repo.

A CI workflow may reside wherever it is most appropriate. Which repo would you like the deb assets to be published in? That will be the repo where the CI workflow will remain.

It doesn't matter from the point of view of the CI workflow how many repos need to be checked out.

What I don't have for the moment is the debian configuration specifically for roc-pulse and one for roc-toolkit that does not include pulseaudio parts. The existing debian configuration from the PR would result in libroc, libroc-dev & pulseaudio-module-roc being built.

I have the following:

1. debian contents for your fork of openfec & roc-toolkit. That may be submitted to a repo.
2. A working github actions runner code to build out packages for non-native ARM. I think it may be adapted easily as a x86_64 runner that is understandably less complex. I'd prefer to use a github actions runner than include the build code in the CI as it would make it unnecessarily complex and have redundant code to build out 3 packages. Runners are a great way to offload that part to a code contained in a separate repo.

[gavv]

Your openfec code being a fork is still fine, perhaps not for PPA but it will suffice, I'd keep it where it is. Submodules seem hack-y and bundling openfec within the libroc package will be a no-no.

OK, let's keep openfec separate package then.

But, I can save you the effort to integrate it. I've already thought of a way to solve the dependency problem as part of a single CI workflow: [...]

Sounds good. One notice: when we check out repos, I think we should always choose last release (tag), not just last commit.

Note we avoid building pulseaudio. The debian build config for roc-pulse will download the pulseaudio sources and configure the sources to generate config.h. No packages will be built, as pulseaudio is quite time consuming to build.

Yep. I'll take a look, probably I can change roc-pulse so that it wont require even running ./configure in pulseaudio (a possible approach was discussed on tracker earlier).

One caveat is roc-pulse will be tied to the major.minor release of pulseaudio available within the debian/ubuntu distibution used to build out the packages on github. IIRC debian doesn't update their pulseaudio releases so that's fine. I don't know about ubuntu. Do you think this would work?

Can we just use docker inside github actions to run desired debian version? This will also allow to build multiple packages for testing, stable, oldstable, if desired.

Now, there remains the $1M question where the debian directories and the CI workflow will reside. [...]

After more thought, it seems that it makes sense to create a single separate repo for all debian packages (e.g. roc-streaming/debian-packages), like it's done here.

Why:

• the user who wants to build deb packages can just clone this repo, run one script and immediately get all (three) needed packages; no need to do it three times for three repos (and later we can add more)

• as you described above, our packages depend on each other (pulse -> toolkit -> openfec) and should be built in correct order; thus, it's logical that there should be a single place where we know about all packages and their relations; and it would be a little strange if that place would be roc-toolkit or roc-pulse;

If you like the idea, I can create the repo so that you could create a PR there.

Which repo would you like the deb assets to be published in? That will be the repo where the CI workflow will remain.

If we go with the idea above, we can publish assets in roc-streaming/debian-packages.

I wonder if you have plans to work on something beyond publishing assets on github, like a PPA? I have no idea how much effort does it need though. If we have no plans for PPA (or something) soon, than just publishing assets on github should be a good start!

In this case, one question is how should we do it: publish each package as separate asset, or maybe publish archive with all packages, for convenience? Or maybe publish each package as separate asset and provide a script / a command example in README that downloads them all.

What I don't have for the moment is the debian configuration specifically for roc-pulse and one for roc-toolkit that does not include pulseaudio parts. The existing debian configuration from the PR would result in libroc, libroc-dev & pulseaudio-module-roc being built.

Would you like to merge the existing version first, or first switch it to roc-pulse?

[...]

I'd prefer to use a github actions runner than include the build code in the CI as it would make it unnecessarily complex and have redundant code to build out 3 packages. Runners are a great way to offload that part to a code contained in a separate repo.

Sorry, I don't get the last one. What do you mean by CI when you say that "github actions runner" is preferred over "CI"? I mean, isn't github actions a CI?

[twojstaryzdomu]

Yep. I'll take a look, probably I can change roc-pulse so that it wont require even running ./configure in pulseaudio (a possible approach was discussed on tracker earlier). Excellent, if it could be done it would reduce the complexity of the code that went into the debian build configuration.

Can we just use docker inside github actions to run desired debian version? This will also allow to build multiple packages for testing, stable, oldstable, if desired.

The runner I'm using runs straight on the OS copy github provides. I'd have to explore how to tie a github runner within docker.

If we go with the idea above, we can publish assets in roc-streaming/debian-packages.

Regarding the naming I'd pick something applicable to the platform at hand. roc-debian would be in line with the way the different repos are called, reflecting their use. It's a good idea to have a clear naming convention.

I wonder if you have plans to work on something beyond publishing assets on github, like a PPA? I have no idea how much effort does it need though. If we have no plans for PPA (or something) soon, than just publishing assets on github should be a good start!

I'd like to explore that possibility. Though I've not yet had much headway in that direction.

Sorry, I don't get the last one. What do you mean by CI when you say that "github actions runner" is preferred over "CI"? I mean, isn't github actions a CI?

Github actions runner is code to perform repeatable activities tied into an item in the CI that acts on input and produces some output, similar to github's checkout or release actions. They make code to run so much easier to maintain. Here's one example that runs code on emulated ARM arm-runner-action. I've written another one of my own specifically for building packages.

[gavv]

Regarding the naming I'd pick something applicable to the platform at hand. roc-debian would be in line with the way the different repos are called, reflecting their use. It's a good idea to have a clear naming convention.

Sure, roc-debian sounds good! So do you agree with the idea of the single separate repo, should I create it?

I'd like to explore that possibility. Though I've not yet had much headway in that direction.

Great. See also roc-streaming/roc-pulse#11, a similar effort, with a PPA, but without pulseaudio modules.

Github actions runner is code to perform repeatable activities tied into an item in the CI that acts on input and produces some output, similar to github's checkout or release actions. They make code to run so much easier to maintain. Here's one example that runs code on emulated ARM arm-runner-action. I've written another one of my own specifically for building packages.

Ah, I see now. So you've created your own github action that builds debian package, and want to reuse it for all three packages, right? (I though that thing is called just "github action", without "runner").

Sounds good, the only concern is that the user can't run it locally, I guess?

[twojstaryzdomu]

Sure, roc-debian sounds good! So do you agree with the idea of the single separate repo, should I create it?

I'd like to explore that possibility. Though I've not yet had much headway in that direction.

Great. See also roc-streaming/roc-pulse#11, a similar effort, with a PPA, but without pulseaudio modules.

The one problem I'd likely encounter is managing dependencies. With roc-pulse yet in flux, it's somewhat difficult to get the build configuration right (and believe me, it took a lot of trial and error to get through the build hurdles in roc-toolkit/roc-streaming before the rename).

When publishing a debian package on PPA, another likely problem is licencing for openfec. For non-public builds, it's a non-issue but once it's public it becomes one. It's not clear which parts of code are covered by a specific licence. I understand it's a mixture of 4 licences. Debian packaging requires For any publication, the licencing needs to be flawless. Each package requires a boilerplate debian/copyright to cover licencing for each file in the source code. The last I looked at it, it was difficult to determine the parts not covered by CeCILL-C/CeCILL licences. The format for the file is defined here if you want to take a look at it.

Github actions runner is code to perform repeatable activities tied into an item in the CI that acts on input and produces some output, similar to github's checkout or release actions. They make code to run so much easier to maintain. Here's one example that runs code on emulated ARM arm-runner-action. I've written another one of my own specifically for building packages.

Ah, I see now. So you've created your own github action that builds debian package, and want to reuse it for all three packages, right? (I though that thing is called just "github action", without "runner").

Sounds good, the only concern is that the user can't run it locally, I guess?

You're absolutely right, they call it a github action, I called it a runner to distinguish it from a workflow that merely uses github actions.

Anyway, the github action to build out is complete and will happily build a package or a set of that may be subsequently published.

No, github actions aren't executable by a user from the context of a sequence of 'run' steps. But I'm sure it can be worked around. The ARM package builder (not public yet due to unclear licencing) was one attempt that worked.

Meanwhile, take a look at build-openfec.yml. This short workflow does everything to build and publish the package using the debianise github action.

[gavv]

The last I looked at it, it was difficult to determine the parts not covered by CeCILL-C/CeCILL licences.

You can grep for "Radford M. Neal" and "Luigi Rizzo" (names are from here). Example files:

• https://github.com/roc-streaming/openfec/blob/b6d32edc75e66d5e036c9346fd0852be09863e8e/src/lib_common/linear_binary_codes_utils/binary_matrix/of_matrix_convert.h
• https://github.com/roc-streaming/openfec/blob/b6d32edc75e66d5e036c9346fd0852be09863e8e/src/lib_stable/reed-solomon_gf_2_m/galois_field_codes_utils/algebra_2_8.c

Meanwhile, take a look at build-openfec.yml. This short workflow does everything to build and publish the package using the debianise github action.

Nice!

[gavv]

I've finished roc-pulse, except documentation.

Its build system now has two modes:

• download and build dependencies automatically (roc, pulseaudio, libtool)
• don't manage dependencies, instead configure paths to them in command-line

I've implemented the method suggested in roc-streaming/roc-pulse#13. With this, in both methods we don'y need to build or configure PulseAudio anymore. Only source code is need.

So how one could package roc-pulse:

• install roc headers and libs

• install libtool (ltdl) headers

• download pulseaudio soruce code

• run cmake as follows:

  cmake .. \
    -DDOWNLOAD_ROC=OFF \
    -DDOWNLOAD_PULSEAUDIO=OFF \
    -DDOWNLOAD_LIBTOOL=OFF \
    -DPULSEAUDIO_DIR=<...> \
    -DPULSEAUDIO_VERSION=<...>
  

You can find some usage examples in README: https://github.com/roc-streaming/roc-pulse

Now I have to switch to freelance work for about two weeks and then I'll work on SO_REUSEADDR problem, merging develop into master, and making 0.2.0 release.

Please let me know if there are some quick fixes that you would like to have.

[gavv]

Hi,

I've squashed and cherry-picked this commits to develop: 774e45f85b9ea4fca7711cdfed683ad5e6f2c5cd

And pushed two follow-up commits:

• 089d197a2a1432ce4b6fd85273cb4082cadcf0d5
• 70d6aaa65f8acc68b7b7ea7e710fc9898f81fc7d

Now debian packages will be automatically attached to roc releases on github. Thank you so much!

What I changed:

• updated dependencies
• renamed packages to the same naming scheme as in pipewire-debian PPA
• update copyrights (they are now generated automatically)
• revised what files are included in libroc vs libroc-dev
• removed patches (they are not needed now)
• removed .docs

Instead of arm-runner step, I've integrated your "debianise" action (my fork of it) to build packages for amd64, to start with something more straightforward. The fork was needed to build package under debian stable instead of ubuntu, I've sent you a PR. If you decide to merge that PR, I'll switch back to the original action.

If you would like to proceed with the arm packages, I'll be happy to review it in a new PR. One thought: I'd prefer to hide that code in a github action, instead of adding a huge script to roc's CI. Hopefully reuse "debianise" action or maybe create a new one.

For now debian package uses --build-3rdparty=openfec because it's easier to implement and at the same time more convenient for users (they'll need to download package only from one source). However if you or someone else have a need to build the package differently, I'll be happy to accept PRs.

Thanks!