Decouple Regolith from Debian package management

[marc0der]

• [x] I've run a search on the existing enhancements to make sure I'm not filing a duplicate (you can use the search box at the top left corner)
• [x] I've read the Regolith documentation and didn't find any relevant topic
• [ ] I tried asking on Slack (especially for configuration related issues)

Is your feature request related to a problem? Please describe. I love using Regolith, but like many other users, do not feel the same way about Ubuntu. At the moment, Regolith is very tightly coupled to the Debian package management, which makes porting Regolith to other distributions very difficult and cumbersome.

Describe the solution you'd like It would be fantastic if all Regolith configuration lived in a separate repo, decoupled from all the Debian packaging config. This would allow other spins (such as an Arch or Fedora spin) to be created from the centralised configuration. The current Arch (and Manjaro) packages rely on transforming the Debian artefacts, which works but is an error-prone and labour intensive process.

Describe alternatives you've considered I've been using the regolith-de package provided by @gardotd426 (which is a great first attempt to get this working, thank you!), but this package tends to fall behind and not on the Arch AUR. Even though this works in the short term, it is not an optimal solution and will require lots of effort to be maintained in the future.

Proposed solution Having a separate repo for the Regolith configuration could allow this project to have multiple automated builds that produce releases for all popular Linux distros. This would definitely further the reach of this fantastic project and help in adoption by many tiling window manager fans like myself who do not want to be tied to Ubuntu.

I am willing to get involved in such a project and would love to know your thoughts around such a proposal. It certainly means quite a bit of work, but could potentially open up many new avenues and bring lots of growth for this project in the future.

[kgilmer]

Hi @marc0der , I completely agree with your points. We look to address the existing coupling to Debian packaging in Regolith 2.0. We don't have much in the way of details yet on this particular item but I believe this will be in scope for the next major release. There are two issues ~ reducing the coupling between packages such that a smaller subset can be more easily ported to another Linux distro without having to "boil the ocean" and port all packages to have a working system. Second, (and more minor I believe but things always start out seeming to be minor) is to remove any existing logic that refers to dpkg, apt, or anything else specific to Debian (or Ubunutu).

I would greatly appreciate your help! What package manager are you familiar with? The question on the top of my mind is, do all popular package management systems support the following features:

1. "virtual" packages...multiple packages can satisfy a given dependency
2. "one and only one" of a given virtual package
3. specifying that a virtual package must be installed and if one currently isn't, specifying a default
4. soft dependency declarations...a transitive package is installed along with a specified package, but the transitive package can be cleanly removed by the user without breaking the dependency relationships in the package manager.

Regolith relies on all of these features from apt/dpkg. In fact, a primary configuration mechanism that Regolith utilizes is the package manager itself. For example, we have compositor packages and they contain package metadata such that one compositor must be installed, and all of the status indicators exist as packages that inject conf.d style configuration snippets at install time.

Much of the coupling I mention above is due to the monolithic nature of the i3 config file. As-is, any program referenced in that file becomes a hard dependency of Regolith. Another theme in 2.0 is adding a simple pre-processor that will generate the i3 config file from conf.d style partials. This allows us to break up these dependency relationships such that porting can start small, be functional, and be incrementally augmented over time.

[kgilmer]

Oh and I happen to be an sdkman user, thanks for your efforts @marc0der !

[marc0der]

Thanks for the reply @kgilmer! Sounds great as a target for Regolith 2.0, and I totally agree with your suggested approach.

I've worked with several package managers in the past. I did some packaging work for some Ubuntu/Debian projects but have now been a longtime Arch (and Manjaro) user. I've also done some work with rpm, and I'm sure I'd be able to pick that up again too.

I hear what you are saying about all the virtual packaging considerations, but this does lead to a very Debian-centric way of thinking about packaging. Have you ever considered a radically different approach? I'm a guy who loves automating things, so have you ever considered using a build tool/system to do the packaging for you?

Consider if you had all the configuration for Regolith in a single repo (neatly modularised), and had some build tool that sat on top of this. You would be able to run different targets to produce whatever packaging you liked. You could produce a set of Debian/Ubuntu .deb packages, some .rpms (that might even have a completely different granularity or dependency graph to your .debs), or produce a PKGBUILD that produces a single package to be pushed to the AUR. Delegating this to a build tool could even eventually lead to the building of custom ISOs that represent different spins of Regolith. I know it's radical, but stay with me :smile:

Along these lines, have you heard of jdeb? In case you haven't, it's a Java library that provides the ability to create Debian packages through an Ant task (that can easily be used from within Gradle or Maven through plugins). This is a way to get around having to use the entire Debian toolchain and be able to build packages on a CI environment. Similarly, there is an RPM builder that can be used from Maven to build .rpms. Of course, these don't give you all the bells and whistles that the native tooling provides, but you would still be able to build a fairly complex DAG of packages.

Another benefit of using a build tool is the power of templates. For instance, it could be possible to use Gradle to prepare a template of the i3 config custom for a given distro. This ties in nicely with what you mentioned about the generation of the i3 config file from partials.

These are all just thoughts and ideas, so take them with a pinch of salt. I'm sure you've already pondered many of these ideas too. But maybe some of these could lead to something useful that will help bring Regolith to the masses through many different Linux distros.

[kgilmer]

Are you saying that the notion of a Virtual Package is a Debian-only concept @marc0der ?

Regarding pivoting to a development model in which the Regolith DE is expressed in package-independent way and that a build system takes that as input and produces sets of packages for target Package Managers and Distributions is something I've briefly thought about. I find myself sticking to well-beaten paths simply as a matter of coping with a project of relatively large scope yet little time to devote to (being a side project). As I only know (and honestly not very well) how Debian packaging works I personally don't feel the risk is acceptably low that I would be able to produce a package abstraction and build system out of the gate that would fit well with other package systems, simply because I do not know how they differ from Debian.

I have briefly looked around to see how other DEs may handle this (package/distro agnostic) but haven't found anything that is general purpose in the sense that I'd be able to use use something out of the box. I've played around with the Opensuse build system which claims to offer this but didn't find documentation that I could get to work.

One approach I plan to look into once 2.0 is stable is fpm which is able to use the Debian packaging format as a source model and generate packages in other target forms. I've done a bit of testing with this and it seems to work.

Another concern w/ a custom build solution is that it would require would-be contributors to learn whatever abstract packaging format that was used, vs Debian which is at least popular and well documented (assuming you're reading the right docs).

And lastly, there is just a lot of long tail stuff that ends up being hard to do. By using Debian packaging I can usually find someone else having a similar problem and am able to adapt something like their strategy.

But, yes, the general idea makes sense. I just don't have the time to build it myself and haven't found anything that would be suitable for Regolith that doesn't have a steep learning curve and/or large risk that it ends up just not working. Last year I devoted a lot of time to unraveling the mystery of how Ubuntu LiveCD installers are produced, and ultimately threw in the towel with nothing to show for it. I'm still somewhat recovering from that and focusing more on goals with higher impact w/ less risk.

[marc0der]

Are you saying that the notion of a Virtual Package is a Debian-only concept

By no means, but it is handled a bit differently by other distributions. For instance, an Arch package in the AUR can't depend on other AUR packages (you can only specify dependencies on packages in the main repos). However, on Arch, we can have a PKGBUILD that builds multiple packages (like regolith-de does), so closer to how a build script would build artefacts. In fact, the command to build the PKGBUILD is called makepkg. You then need to install these packages individually. You will notice that @gardotd426 has far fewer packages than your debs because of this flat dependency graph.

As you can see, this is a departure from Ubuntu/Debian where a PPA's packages become part of the greater pool and allows for far more complex DAGs. The package management on Arch is purposefully simpler in keeping with their philosophy of KISS.

Thus, making Ubuntu the base packaging enforces the Debian packaging onto other package management systems that aren't really compatible, or have very different ways of achieving their purposes.

So my personal gut feel is that:

source -> deb -> rpm
              -> pacman
              -> xxx

feels more error-prone, and prescriptive than:

source -> deb
       -> rpm
       -> pacman
       -> xxx

Then again, I'm known to be a purist :grin:

I also hear your concerns about this stuff being very time-consuming, but here I'd be happy to help by getting involved (and I'm sure others are also keen to get in). I don't think anyone expects you to do all this work by yourself or get on top of all the types of packaging systems out there. I'm also not saying that what you've done so far isn't great work, because it's awesome! I'm merely saying that the current setup does limit the scope of who the project could reach going forward.

My idea of using a build tool for the packaging is exactly for the reason that you mentioned before. I don't want to know the intricacies of multiple platform packages. Others have done the work already. I would rather work on top of what they have already achieved. Gradle with the Netflix nebula ospackage plugin could save lots of time. I'm sure there are many other alternatives too.

Here's my suggestion: why don't we do a small spike to test viability? I would choose some fairly isolated packages that can be drawn together via a virtual package (maybe themes are a good candidate, but whatever you would suggest).

• import the repo(s) into a new repo via git submodules.
• use a build tool in this new repo to try to build .deb packages.
• new packages should be mostly equivalent to the ones that you are currently producing.
• add the ability to produce the rpm analogues of the debs

If we could initially succeed in doing a simple like-for-like, it would prove the point. This would also give us a starting point to begin iterating on. I have a deep dislike for big bang changes, so I'm not proposing anything radical here. I would do something like this in a very sensible and gradual approach.

I hope you now have a clearer understanding of where I'm coming from and that I'm not thinking of any gun-slinging cowboy stuff :smile:

[kgilmer]

Thanks for the kind words and well reasoned argument @marc0der :smile: . I think I have a clear understanding of where you're coming from and will take you up on your offer to collaborate! However I think the current work I'm doing in greatly reducing coupling across all packages needs to be stable before getting too far ahead, as the package layout change is drastic. Would you be able to do testing for AUR? Maybe some one else can cover RPM?

Let's move this over to the wiki: https://github.com/regolith-linux/regolith-system/wiki/Package-Manager-Agnostic-Build-System

@gardotd426 are you interested in getting involved?

[marc0der]

No probs and glad that you understand my intentions here.

However I think the current work I'm doing in greatly reducing coupling across all packages needs to be stable before getting too far ahead

You are so right. Decoupling package management will give us a far better foundation to work from.

Would you be able to do testing for AUR?

Absolutely, yes. I would love to help in any way possible.

[marc0der]

Just dropping a note here to say that regolith-de has not seen any subsequent releases since 25 December (1.5 was the last) and has still not been released to the AUR. Also, no further updates from @gardotd426 on this. I would love to see this move forward while we don't have the newly proposed solution in place yet.

What can I do to make this happen? Should we fork the regolith-de repo as an official part of regolith and start making it part of the regular releases? That would also involve publishing it to the AUR when it lands. As always, I'd be happy to get my hands dirty on this.

[gardotd426]

Hey guys, sorry I had some equipment issues that kept me out of it for a while, but everything is now sorted and yeah Ken, I'm more than happy to get involved, this sounds like a good idea if you're able to do it. 

Also @marc0der, maybe contacting me on the regolith-de repo would have answered your questions, and the repository absolutely has had updates in since 12/25, actually 13 days ago was the last update, look for yourself. I don't see a reason to do GitHub "Releases" for an Arch Linux PKGBUILD, that seems nonsensical. But the repo has been getting updated as needed. And at this stage, everything is to the point where the only things that will be changing are the PKGBUILD and then maybe little patches here and there to fix Arch/Ubuntu differences, but they're rare and I've been able to remove most of them by now. So yeah, weekly/bi-weekly updates to package version numbers will be the only thing changing from here on out.

A user kindly put together a python script to help me automatically update the PKGBUILD with new .deb package versions, so I don't have to manually update it, because trying to track 40-something packages for version changes and then individually change them is insanely tedious. I'm looking to set up something that can automatically run the script at a set time (probably weekly) and notify me when there's updates so I can merge the PKGBUILD files and push the update to the repo. So I guess I anticipate a move to the AUR in the next week, as if it's this stable and only needs to be changed to track package versions (for the most part, unless something breaks where I fix it before pushing the update), then why not have it in the AUR? The main reason I've waited is for more testing, as 3-4 people isn't enough for a desktop environment for me to feel comfortable with stability, but the past few weeks I've had more and more testers, to where I'm around 20 at this point, and I feel comfortable pushing to the AUR.

But yes, a distro-agnostic build system would be a dream come true for me at this point, and am happy to help however I can.

[marc0der]

@gardotd426 Well, this is fantastic news, and I'm thrilled to hear that regolith-de is still alive and kicking! I'm also happy to join your testers and will certainly engage with your regolith-de repo going forward.

It would be super nice to have some tags corresponding to the official regolith releases though, no need for full-blown Github releases. I think we are currently standing at 1.5.3, and it would be good to track this in your repo too. Also, great to hear that you are about to go live to the AUR! Have you considered using simple Github actions to automate this process? I'm happy to help you get this up and running.

But let's move over to the regolith-de repo for further discussion.

[gardotd426]

Release is on 1.5, I see nothing on Regolith's website about 1.5.3, and I've had my PKGBUILD versioned 1.5 for weeks (since Regolith moved to 1.5 from 1.4.3 or 1.4.1 or whatever).

I see no point in tagging the github repo according to Regolith release version numbers, the only real important thing is to have the Arch PKGBUILD match up with the Regolith version its taken from, and it already does that. This is intended to be an AUR package.

Furthermore there's no real way to even automate versioning and I don't plan on constantly checking the Regolith home page to find out what the latest release one the "Release" PPA is. Do you? I was hoping there was a package included such as regolith-desktop-complete that came with a version in-line with Regolith's overall version (btw @kgilmer is there some reason why that's not a thing? Normally all DE packages share a version with the DE itself, you can look at Plasma as an example), but that's not the case, there are no 1.5 (or 1.5.3) versioned packages on regolith's release PPA. Well, regolith-system is on 1.5.0-1 for 18.04, but not for any other version, so idk if that's relevant or not.

But yeah, I'd written a script to handle the automatic updating of the PKGBUILD and include an updating of the pkgver, but there's no package version that corresponds so there's no easy way to do this, so for now I'm just tracking 1.5 as the documentation says.

But yeah feel free to open an issue on my repo if you want to discuss anything about regolith-de.

[kgilmer]

Hey guys :smile: . I try and keep the github release tags up to date here: https://github.com/regolith-linux/regolith-desktop/releases Release notes are here: https://regolith-linux.org/docs/reference/releases/regolith-1.5.3-release-notes/

You can follow the @RegolithL twitter account or the regolith mailing list to get announcements for releases, if you'd like.

I was hoping there was a package included such as regolith-desktop-complete that came with a version in-line with Regolith's overall version (btw @kgilmer is there some reason why that's not a thing? Normally all DE packages share a version with the DE itself, you can look at Plasma as an example)

Well, the DE is a set of many packages. For the full ISO, a package regolith-system provides that version mapping from package to DE version. However in PPA form that package is not needed. The version of packages changes frequently and without cadence w/ the release version. The closest I can give you here is the /etc/regolith/version file which is shipped here.

Also, a bit of an update from my end, the 2.0 work is coming along well. My focus has been on reducing the package graph as much as possible to aid in porting as well as just enabling users to have light setups. Regolith 2.0 can now run on i3 or i3-gaps, and the package manager will handle all the compatibily details if a user switches between them. Also, it's running on Debian in 281Mb cold start. There is still some work remaining on the desktop metapackages but overall I think the scheme is in good shape as a PoC. I will add some documentation to the wiki and call for comments from users in the next week or so regarding the general design.

[kgilmer]

So, about the build system...is anyone of a general purpose package model in a modern format?

[marc0der]

@kgilmer Awesome news about the 2.0 work, It sounds like you are making great progress. Also, thanks for clarifying about the versioning. This makes perfect sense. We can pick this discussion up again on the regolith-de repo.

So about the build system. I have been giving this some more thought and have some ideas about how we can do this in the simplest possible way. My initial thoughts about using Gradle and plugins have changed somewhat. Instead, I'm thinking of using something more native. I thought of using make in conjunction with Docker to build native targets per Linux distro. For instance, this would allow us to continue using the build-essential tooling provided by the Debian stack to build packages inside a mounted Docker container. Likewise, we could use makepkg for preparing Arch packages in an Arch container. We could repeat this for as many distros as we would like to support. What do you think of this approach?

Anyway, these are just thoughts for now, but I will report back when I have something more solid up and running over your 2.0 work.

[gardotd426]

Hey @kgilmer, someone just made me aware that you aren't packaging rofication on launchpad the same way you have it on the regolith-roficiation repo. They simply pointed out that you aren't including the setup.py file, but you're also not including the i3xrocks module file, 80_rofication.

This actually means that I have to wait until it's fixed to put it live on the AUR, or just remove rofication and use dunst or some other notification manager. The problem is, AUR packages can't (at least are absolutely not supposed to) require user intervention, but the way things are right now, rofication won't work unless the user pulls the setup.py script from your repo and runs it. So we're at a bit of an impasse. I can still leave up my PKGBUILD repository, but I can't put it on the AUR until this is sorted.

[kgilmer]

They simply pointed out that you aren't including the setup.py file

Hmm, yes looking at the output setup.py (dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb | grep setup) is not there. I checked a few other packages in Regolith for that file, and none of them have it. For example, python3-i3ipc. Why is this file required only on this one repo? I can look into fixes but python/debian packaging seems like a wonderland unto itself and I simply don't have time to become an expert on this. Is there more specfiics as to what you require other than "setup.py"? Say I make a change, how can I test that my update satisfies the requirements of Arch policy or build logic?

I'm guessing this is a case where a package agnostic build approach would shine. I'm guessing your porting is constrained by what your native packaging system needs and what the debian packages provide.

you're also not including the i3xrocks module file, 80_rofication.

This file is now shipped in the binary package i3xrocks-rofication defined in regolith-i3xrocks-config. This was done to decouple rofication from i3bar.

[gardotd426]

Hmm, yes looking at the output setup.py (dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb | grep setup) is not there. I checked a few other packages in Regolith for that file, and none of them have it. For example, python3-i3ipc. Why is this file required only on this one repo? I can look into fixes but python/debian packaging seems like a wonderland unto itself and I simply don't have time to become an expert on this. Is there more specfiics as to what you require other than "setup.py"? Say I make a change, how can I test that my update satisfies the requirements of Arch policy or build logic?

Well, that's the thing. Without running the setup.py script, I'm not really sure how it's even being correctly installed. Because just copying the files from the .deb to /usr/bin (which is literally all that dpkg does too) should fail with a Module not found: rofication/no module named "rofication" Python error. So that's why I was confused, and no, you don't need to become an expert on it, I'll explain it this way: Imagine if to install a package from a PPA, instead of just adding the PPA, sudo apt update-ing and sudo apt install PPAPACKAGE-ing, you had to do all that, then cd into some directory and run an additional script. That wouldn't generally be cool, right? Same thing for AUR packages. But see, you're not doing it w/ dpkg, so it shouldn't be necessary on Arch. The only thing we need to figure out now is why it's not working on Arch without the setup.py script running. But I know you're busy, so I'll look into it, I can spin up some Ubuntu VMs and add the PPAs and see what happens, and then I can just compare the differences.

It's not anything like python-i3ipc, I made sure to include that in the dependencies for the PKGBUILD, so that's getting installed already. But yeah the fact that you don't include it or anything special on Ubuntu (or Regolith Proper) is more than enough to get me on the right track, so don't worry about it.

This file is now shipped in the binary package i3xrocks-rofication defined in regolith-i3xrocks-config. This was done to decouple rofication from i3bar.

Ah cool, I noticed you'd added a couple new modules lately, I'll go ahead and add that to the i3xrocks package then.

I'll let you know once I figure it out, but I already have a couple ideas. Thanks for the guidance

[kgilmer]

My apologies if I came across defensive or weird there, it wasn't intentional. Somehow I have a very low patience threshold when dealing w/ python issues in general. I have no good reason, just a personality quirk I guess. In any case, thanks for your clarifications. Yes, it's true that despite the deb package simply saying it's shipping the python file, actually quite a bit more comes along for the ride:

$ dpkg -c ~/Downloads/regolith-rofication_1.3.1-1_amd64.deb 
drwxr-xr-x root/root         0 2020-12-27 11:27 ./
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/bin/
-rwxr-xr-x root/root       417 2020-12-27 11:27 ./usr/bin/rofication-daemon
-rwxr-xr-x root/root       169 2020-12-27 11:27 ./usr/bin/rofication-gui
-rwxr-xr-x root/root      1157 2020-12-27 11:27 ./usr/bin/rofication-status
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/
-rw-r--r-- root/root       414 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/__init__.py
-rw-r--r-- root/root      1536 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_client.py
-rw-r--r-- root/root      3345 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_dbus.py
-rw-r--r-- root/root      3665 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_gui.py
-rw-r--r-- root/root       214 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_metadata.py
-rw-r--r-- root/root      1260 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_notification.py
-rw-r--r-- root/root      3908 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_queue.py
-rw-r--r-- root/root      2893 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_server.py
-rw-r--r-- root/root       297 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_static.py
-rw-r--r-- root/root      1092 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/_util.py
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/
-rw-r--r-- root/root       163 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/__init__.py
-rw-r--r-- root/root      1042 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication/resources/_static.py
-rw-r--r-- root/root       533 2020-12-27 11:27 ./usr/lib/python3/dist-packages/rofication-1.2.2.egg-info
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/
-rw-r--r-- root/root      1121 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/changelog.Debian.gz
-rw-r--r-- root/root      1231 2020-12-27 11:27 ./usr/share/doc/regolith-rofication/copyright

Again, I can't say if this should be sufficient for a working Python module, but perhaps this provides some help. I do know that a python-specifc build system is used to generate the package, and this is integrated w/ the debian packaging. I'm not sure but maybe this is doing some magic that does whatever setup.py is doing in other packages..?

If you come up w/ anything specific you'd like to try, I'm more that happy to build some test packages w code changes, just LMK :smile:

[gardotd426]

Oh no worries man, it's all good.

But yeah I know about the other files aside from what's in /usr/bin, I install the whole package, I'm not just pulling deb files. There's a script in the PKGBUILD like this:

extract_deb() {
    local tmpdir="$(basename "${1%.deb}")"
    rm -Rf "$tmpdir"
    mkdir "$tmpdir"
    cd "$tmpdir"
    ar x "$1"
    tar -C "${pkgdir}" -xf data.tar.xz
}

And then for each .deb, I just have to run extract_deb "${srcdir}"/pkgname-pkgver.deb, blah blah blah. So I'm installing all those files too, I was just simplifying things earlier. I get everything that's included in the .deb package, but I don't get the setup.py because that's not with the .deb package, it's in the upstream original dev's repo, so obviously we shouldn't need it. I'll figure it out, no worries. It's probably something stupid, like I said I already have a couple ideas.

Also if necessary I could potentially just break regolith-rofication into its own separate PKGBUILD instead of being part of a meta-package, and instead of using the deb, just clone the repo, because I can clone the repo and build and install from within the PKGBUILD just fine, it's just, it's hard to explain but the way PKGBUILDs work for meta-packages, defining one git repo for one package can mess all the rest up, but honestly that's not even a remotely difficult thing to do, it would literally just be mostly a copy and paste thing, and I'd just have two PKGBUILD files.

But yeah, I think debhelper-python is what's doing it for you on the Ubuntu side, which makes a ton of sense. No worries, I have like 5 different solutions, I'm just researching which one would be the "best" one, or at least the one I'm likely to get the least complaints about lol.

[gardotd426]

I got it figured out. I just had to dig deep into some Arch documentation about including Python packages in PKGBUILDs, because the normal procedure/recommendation is to use PyPi/Pip, but they do have procedures for PKGBUILDs for Python packages that aren't available there. I just tested it and it works great, I'll be updating the PKGBUILD in a few.

[kgilmer]

@marc0der,

We could repeat this for as many distros as we would like to support. What do you think of this approach?

I have started working working on something similar to this in order to support Debian and Ubuntu. It's nothing fancy, just some bash code and a JSON document that describes all packages necessary to build the Regolith DE. It's somewhat simplified at the moment as it's only building a subset. There is a repo that contains the scripts (mainly just a convention): https://github.com/regolith-linux/pkgrepo-lib and then "child" repos (currently only https://github.com/regolith-linux/pkgrepo-ubuntu-focal) that include the scripts, add their own specific package models (Debian generally needs more than Ubuntu) and any custom scripts. Rather than just straight docker, each "child" repo uses gh actions to essentially do the same thing ~ create a container and do some work. The results are committed back to the source repo in my current case, as I'm able to use the Github Pages feature to host the repositories. Unsure if this is how things will be for production but works well enough for me now.

I plan to use this approach to completely automate generation of Ubuntu Bionic, Focal, Groovy and Debian Buster, Bullseye package repos.

What do you think? Is there anything about the system I describe here that would be prohibitive to non Debian based systems?

[kgilmer]

FYI I've been adding issues to give a little more insight as to what I'm doing: https://github.com/regolith-linux/pkgrepo-lib/issues

[marc0der]

Hi @kgilmer! Firstly, a big apology that I haven't replied to you sooner. I've been a bit swamped with work the last while, so I haven't had much chance to look at this stuff again. That said, I hope to have a closer look over the weekend. Happy to have a play and give some feedback regarding this.

[kgilmer]

I'm suffering from a bad case of day-job myself recently, no worries @marc0der . Come back to this when you have the time and interest :smile:

[kgilmer]

Things have moved around a bit now that the names matter..

• Generalized model and scripts: https://github.com/regolith-linux/regolith-creche
• Semi working example: https://github.com/regolith-linux/repo-ubuntu-bionic (I have yet to make the necessary model transforms for the bionic version of gnome-flashback, but it installs)

[neevparikh]

Hey all, any updates on moving this forward since March? I'm already testing regolith-de on arch, @gardotd426, and it's working totally great.

[a-crate]

I'd like to chime in as well here, I'm looking at porting regolith to gentoo, and the first hurdle from my perspective is determining how and when releases are cut. I'm under the impression that @gardotd426 has been unpacking .deb files from the PPA but that doesn't really work for me on a source based distribution. I've had a look around and while the package model in https://github.com/regolith-linux/package-repo is great for determining where I can grab sources and what sources I need, it doesn't really help me with versioning, nor is there a consistent source of truth for what source files were used to build which version. Is there any way to add a commit field in the package model, and possibly have something like a github release on version update to show the package model for each release?

[gardotd426]

Yeah I struggled for quite a while with trying to figure out how exactly to version stuff (especially since with metapackages in the AUR, there can only be one package version), before I moved most of the non-Regolith-specific packages over to Arch repo packages (like i3-gaps, picom, etc), so my PKGBUILD mostly just provides regolith-specific stuff, so I landed on using the current Regolith version. So right now it's 1.6. But yeah there seems to be little rhyme or reason on the versioning on source repos, I'm in the process now of moving whatever I can over to being built from source instead of just extracting .debs.

@a-crate I got your email, I'll email you back shortly answering some of your questions

[kgilmer]

Hi @a-crate , that's exciting news. I learned a lot from running Gentoo for years and years. It would be amazing to have Regolith run there as well.

Preamble: Only the release stage in Regolith is versioned. stable and unstable lack a cross-package versioning scheme.

the first hurdle from my perspective is determining how and when releases are cut

Things are a bit weird right now due to the transition from Regolith 1.x to 2. This major version bump constitutes the change to clean things up and do things in simpler ways while breaking with the old, launchpad.net-based build/repo model. The 2.x build stuff in part is to make it easier for efforts such as yours to port the packages to other distributions. 2.0 is still in development and so if there are ways that this approach can be improved this is certainly the time to make the right changes.

As such, you are not finding any versioning stuff because there has yet to be a version of Regolith 2.x. The general workflow for how packages go from source in git repos to installed packages on user's systems is described here.

In summary, the "version" of Regolith 2.x is the set of git URLs and branch/tag names specified in the root package model + any distro specific overlays (ex).

nor is there a consistent source of truth for what source files were used to build which version

Again, since we are before an explicit version, this information does not yet exist. My thinking here is that rather than having a single package model + overlays, rather these will be maintained per distro. So the file package-repo/regolith-2.0-package-model.json would move to something like package-repo/stages/unstable/regolith-2.0-package-model.json. And package-repo/distros/debian/bullseye/* -> package-repo/distros/debian/bullseye/unstable/*. This would essentially tie package model snapshots to particular stages. When releases are cut, the corresponding release tags will snapshot these models to a particular version. (Yes there is a problem of package models specifying (changing) branches over static tags that will need to be addressed. Each regolith package will need a release tag and the release models updated with that tag before release).

Is there any way to add a commit field in the package model

Yes the current field branch should be updated to be more general and work with branches, tags, or commits. (TBD)

and possibly have something like a github release on version update to show the package model for each release?

Yes, package-repo will have tag snapshots for public releases. I'd like to tag repos with release versions as well but am unsure of how I can easily automate that and will not be doing it by hand.

Hopefully that answers your questions to some degree @a-crate . FWIW if you were to provide a GitHub action like this for your target distro, all of the versioning would generally be handled and you'd just* need need to build and stage packages for your target distro. EG when Regolith 2.0 had it's first release, your action would be called just like the Debian and Ubuntu actions.

(*: of course it's never that simple and I'm sure there are things I'm entirely unaware of that need to be solved.)

[kgilmer]

More details here (I just refreshed this page): https://github.com/regolith-linux/regolith-system/wiki/Package-Build-Automation