Integration in bloom, and binary package repository

[bchretien]

Now that the whole system is fairly stable and has been working well for years, we can consider integrating the work that was done here inside bloom, the ROS release automation tool, and try to establish a repository with binary packages as is done for Debian (and probably Fedora). The first task would be to list the requirements for this endeavor. Because of the rolling-release style of Arch, there might be some key differences in the way we approach this compared to Debian/Ubuntu. The question of dependencies would also need to be resolved: should we package them as well? Or find a TU willing to handle them in [community]?

Concerning bloom, I found the POC I was investigating 2 years ago. I can update it with what I ended up doing in arch-ros-stacks after, and see how far I can go.

cc @v01d, @dirk-thomas

Related discussion on ROS answers here.

[bchretien]

Comment from @dirk-thomas on the ROS answers discussion:

Yes, ros_buildfarm would drop the binary packages into an Arch repository (however that looks like). I guess the same server as for the Debian/Ubuntu apt repo can be used. The buildfarm_deployment repo is used to provision the server.

Arch repositories are fairly simple to set up (see the Arch wiki), everything appears to be done with the repo-add script (provided by pacman), so the workflow would just be:

• Copy binary tarball to the server,
• Call repo-add to register the new package/package version.

Having a server available solves most of the problem (dependencies aside). The main difficulty will probably be to add Arch support to ros_buildfarm.

[protobits]

Hi, good thing about bloom. I can try to help there but I'm not very proficient in python. I will try to look into ros_buildfarm and see if I understand what would be needed there.

Regarding dependencies. At first maybe it would be better to rely on pacaur/yaourt as part of the build process. If this gets going I guess some people may be willing to support the main AUR dependencies .

[protobits]

One question, though. What is the relation between what bloom does and ros_buildfarm does? From what i understand, bloom is who creates the so-called release repository and ros_buildfarm reads this to know what to do, right? So, more or less, the PKGBUILD would be embedded in this release repository and the buildfarm will mostly do the makepkg, right?

[bchretien]

That's right, there are 2 distinct steps:

• creating the package description (PKGBUILD/.SRCINFO), which could be done by bloom,
• providing binary releases of these packages, which could be done by ros_buildfarm.

For dependencies, not having binary packages available means the build farm would need to download the source and recompile everything. If something depends on PCL for instance, merely handling the dependencies is likely to take like an hour... That's not gonna cut it. If we have a TU backing us up, that part could be done in [community], or we need to setup another repository.

[dirk-thomas]

The build farm will take the release repo as input and produce a binary for each package. It will not build any dependencies - they much have been either available from somewhere else or built by the build farm from other release repos. The build farm will make sure to invoke all jobs in the correct topological order as well as will perform rebuild of downstream packages (since we can't assume ABI compatibility).

[protobits]

@bchretien I'm currently setting up a private gitlab server with CI where I have managed to set a job for every push where an ubuntu package is generated and stored in a repository, which allows to handle dependencies when compiling other projects in the server. For this, I'm using bloom-generate and then compiling with fakeroot (think of a poor man's, but simpler, ros_buildfarm).

I was wondering if the scripts in this repo could be used in the same way, at least rudimentary, to generate a PKGBUILD from a local catkin package, or if this functionality is only developed by pulling package information from rosdistro. I'm asking since if I set this up, I could also try to get a start with porting this functionality to bloom-release.

[bchretien]

@v01d nice, this sounds promising!

The script used here is self-contained and grew over time, so it is a bit messy, and it would just be a temporary solution to validate the approach for you. It could be extended to take a path to a local package.xml rather than fetching it automatically. The problem is that the PKGBUILD still requires a remote tarball/repository. Here, we actually rely on bloom's source tarball (generated by RosReleaseGenerator).

That's the main difference with bloom's deb/rpm generators. The source code should not be bundled with the PKGBUILD (although that's possible AFAIK). If we can rely on RosReleaseGenerator to generate the source tarball, and know the URL where it will be stored, then we can use the extra metadata to fill the PKGBUILD (but that would probably be quite different from the current deb/rpm generators). Note that if the tarball is available locally, it does not even need to be available online at that time (makepkg detects that the tarball is available and that the checksum matches).

[protobits]

Hi, thanks for the info. If I wanted to modify the scripts to work on a local package.xml, where should I start?

When you say "bloom's source tarball" you mean the GBP repository, right? In my case, what is only needed is to replicate bloom-generate's functionality, which assumes the source is already present in the current directory. So retrieving the source would not be needed in my case, but only to generate the PKGBUILD. For the case of a local package.xml the PKGBUILD could directly point at the local source directory.

In the long-term I guess the correct approach is to generate the GBP repository, which is the current ROS standard for dealing with this. But I would directly add that functionality to bloom instead of extending the current scripts. In any case, I think the functionality I'm needing now (which mimics a part of bloom, the bloom-generate script) would be a first step towards solving the present issue.

On Tue, Mar 8, 2016 at 12:12 PM, Benjamin Chrétien <[email protected]

wrote:

@v01d https://github.com/v01d nice, this sounds promising!

The script used here is self-contained and grew over time, so it is a bit messy, and it would just be a temporary solution to validate the approach for you. It could be extended to take a path to a local package.xml rather than fetching it automatically. The problem is that the PKGBUILD still requires a remote tarball/repository. Here, we actually rely on bloom's source tarball (generated by RosReleaseGenerator).

That's the main difference with bloom's deb/rpm generators. The source code should not be bundled with the PKGBUILD (although that's possible AFAIK). If we can rely on RosReleaseGenerator to generate the source tarball, and know the URL where it will be stored, then we can use the extra metadata to fill the PKGBUILD (but that would probably be quite different from the current deb/rpm generators). Note that if the tarball is available locally, it does not even need to be available online at that time (makepkg detects that the tarball is available and that the checksum matches).

— Reply to this email directly or view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-193818458 .

[bchretien]

That's correct. Relying on the source tarball from the GBP repository means that people could easily rebuild the packages themselves (i.e. exactly what we currently have on the AUR). If this is not possible or too complicated, a "dirty" PKGBUILD relying on local files could probably work (and in that case we assume that people simply get the Arch binary releases anyway). I guess this is what you currently have in your proof of concept, right?

Now, in that case, the PKGBUILD generated by import_catkin_packages.py would need to be adapted a bit more, e.g. a LocalPackage that inherits from PackageBase and provides an adapted PKGBUILD template (which is similar to bloom's empy templates). But then I guess dusting my old bloom fork with early PKGBUILD support might take just as long. What stopped me then was that unavailable source tarball problem.

[bchretien]

You can take a look at my bloom fork here. I tested it on catkin, and you can see the resulting PKGBUILD there. There's still work to do, but it's a good start.

[protobits]

Great! Any pointers of what things would require improvement or should I simply start testing it with different packages and see what issues come up? I can fork your fork and send you PRs for any changes.

On Tue, Mar 8, 2016 at 2:36 PM, Benjamin Chrétien [email protected] wrote:

You can take a look at my bloom fork here https://github.com/bchretien/bloom/tree/topic/pkgbuild-support. I tested it on catkin, and you can see the resulting PKGBUILD there https://gist.github.com/bchretien/37ae45fba68886c3e5ae. There's still work to do, but it's a good start.

— Reply to this email directly or view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-193882744 .

[bchretien]

I tested on catkin since it does not depend on any ROS package. If it depends on ROS packages, there's a rosdep check that currently fails. Maybe we need to add Arch to the release platforms in rosdistro (see here).

Also, AFAIK, a PKGBUILD's changelog is supposed to reflect changes to the Arch package, not changes to the original project. Then again, nearly nobody uses it, especially now that PKGBUILDs are hosted as Git repositories, so checking the Git history gives much more information anyway. Should we just ignore the changelog?

Moreover, the original commit for PKGBUILD support was based on what was done for Debian/Fedora, and there has probably been changes for them that I now need to apply.

[protobits]

I'll take a look at the ros dependency handling. I fail to see where is dependency information actually stored inside the rosdistro repo, though.

I vote for leaving unessential features as the last step. I nice milestone would be to have all packages in AUR auto generated from bloom, and compiling OK with the produced PKGBUILDs, even if they're ugly or not feature-complete.

Maybe you can try rebasing your branch on top of master and see what comes up.

On Tue, Mar 8, 2016 at 10:13 PM, Benjamin Chrétien <[email protected]

wrote:

I tested on catkin since it does not depend on any ROS package. If it depends on ROS packages, there's a rosdep check that currently fails. Maybe we need to add Arch to the release platforms in rosdistro (see here https://github.com/ros/rosdistro/blob/master/indigo/distribution.yaml#L5 ).

Also, AFAIK, a PKGBUILD's changelog is supposed to reflect changes to the Arch package, not changes to the original project. Then again, nearly nobody uses it, especially now that PKGBUILDs are hosted as Git repositories, so checking the Git history gives much more information anyway. Should we just ignore the changelog?

Moreover, the original commit for PKGBUILD support was based on what was done for Debian/Fedora, and there has probably been changes for them that I now need to apply.

— Reply to this email directly or view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-194052890 .

[bchretien]

The problem with the in-source approach (PKGBUILD that is supposed to be inside the source directory) is that this is just not AUR-friendly, so the PKGBUILDs generated by my bloom fork are not gonna cut it for now. Since we can add CLI options to bloom-generate arch, we can have 2 PKGBUILD templates: one for the in-source deb-style approach (what we have on my fork), and one that relies on the source tarball (as we do on the AUR), and add a --from-tarball option to select the latter.

My branch is already up-to-date with master, there's just some code duplication.

[bchretien]

@v01d: To test my bloom fork locally with ROS dependencies, I used rosdep-generator, modified the generated yaml to add arch, and ran rosdep update. I guess there's a smarter/easier way to do this, but I couldn't find the information. Still, for std_msgs, running bloom-generate rosarch --ros-distro indigo gives this PKGBUILD, and it builds just fine. There's still a few things to do (add changelog, format the raw description, provide a way to set the Python version, remove the sed block for Python 3, see if adding support for Arch variants such as Chakra makes sense etc...).

@dirk-thomas What is the procedure to add a new distribution to rosdistro? Do we only need to add an entry to rosdistro/.../distribution.yaml? Is that doable before having the whole "bloom pipeline" ready?

[dirk-thomas]

I don't know how it affects bloomwhen a new releases platform is added to the distribution file. Maybe @wjwwood can clarify.

[wjwwood]

Adding an os (platform) to the distribution file does not affect bloom unless you have a generator which asks what the default distros are for that platform/os. See our recent changes to bloom which "add" support for Debian Jessie: https://github.com/ros-infrastructure/bloom/commit/5305e9749e17a2162939c4f29d2400e216fe94cb

In that case we just slightly modified the debian generator to allow for an empty distros (in the cases where Debian is not setup for older ROS distros) and added an action to the default bloom actions which attempts to generate packaging for Debian Jessie.

[protobits]

Hi, I would like to see if we can pick up this a bit. As I understand this could be gradually implemented.

• [ ] add "arch" to distribution files (trivial?)
• [ ] with this, we could test the bloom fork and see if we can get it to a basic usable state (for a proof of concept at least)
• [ ] locally generate all PKGBUILDs with bloom fork and locally compile (I have a nice script for building a bunch of interdependent packages on a directory: http://github.com/v01d/aap)
• [ ] revise all required rosdep keys for archlinux to allow correct generation of dependencies in prior step
• [ ] see about external/internal source issue for PKGBUILD
• [ ] see into required modifications in buildfarm

To actually have this going we need to:

• [ ] get non-ros AUR dependencies into an arch repository (maybe we can host this somewhere until they can be adopted by a TU)

My only question is: how to deal with the rolling release aspect of Arch? I mean, right now when for example boost is updated, all AUR packages need to be recompiled. In this case, how would the ros buildfarm notice this?

Finally, I think that once all of this is going, there's no point in having all ROS packages in the AUR. So I don't think it is necessary to keep compatibility for this use case.

I can start with some of the initial steps, I would only ask for some guidance or help in this quest with bloom and buildfarm modifications.

[bchretien]

get non-ros AUR dependencies into an arch repository (maybe we can host this somewhere until they can be adopted by a TU)

Indeed, we should start with this. Would you be able to manage that repository?

My only question is: how to deal with the rolling release aspect of Arch? I mean, right now when for example boost is updated, all AUR packages need to be recompiled. In this case, how would the ros buildfarm notice this?

We'd need a way to manually trigger bumps of relevant packages when rebuilds are required. I don't know if this is possible/compatible with the buildfarm.

I can start with some of the initial steps, I would only ask for some guidance or help in this quest with bloom and buildfarm modifications.

I'm quite busy at the moment, but I can provide information on what I've done so far.

[dirk-thomas]

You won't need to bump the package versions. A simple manual retriggering of the binary jobs should be sufficient. Each package has the timestamp it was build on appended.

[protobits]

@bchretien @dirk-thomas can you confirm that the first step is ok? If so, I can make a pull request to rosdistro repo and have that out of the way. After that I can test the bloom fork and re-generate and locally compile pkgbuilds. We can see from there.

Indeed, we should start with this. Would you be able to manage that repository?

I can get the ball rolling and put the necessary precompiled packages up somewhere to test the system. But I'm not sure if I can get commit in the long term to do so. I have the hope that if we have a working system it will be easier to convince a TU to maintain them. By the way, can you think of a simple command to list all non-official (aur) non-ros packages which would need to be provided?

You won't need to bump the package versions. A simple manual retriggering of the binary jobs should be sufficient. Each package has the timestamp it was build on appended.

But does the buildfarm check if for example boost was updated and triggers a rebuild? Or is there an API which should be used to externally trigger the rebuild? In the latter case, I understand that it would be ArchLinux's build system's responsibility to invoke this API.

[dirk-thomas]

@bchretien @dirk-thomas can you confirm that the first step is ok? If so, I can make a pull request to rosdistro repo and have that out of the way. After that I can test the bloom fork and re-generate and locally compile pkgbuilds. We can see from there.

I am not sure if adding it to the public rosdistro would have any effect. Since you only need this for trying to build local packages I would recommend to fork the rosdistro repo, apply the change their, and then use this fork for your local package tests.

[dirk-thomas]

But does the buildfarm check if for example boost was updated and triggers a rebuild? Or is there an API which should be used to externally trigger the rebuild? In the latter case, I understand that it would be ArchLinux's build system's responsibility to invoke this API.

No, the build farm does not trigger any jobs based on third party packages. It is assumed that all external dependencies are ABI compatible and don't require a rebuild of downstream packages. There is no custom API in ros_buildfarm. But you can use the Jenkins API to trigger specific builds if necessary.

[protobits]

I am not sure if adding it to the public rosdistro would have any effect. Since you only need this for trying to build local packages I would recommend to fork the rosdistro repo, apply the change their, and then use this fork for your local package tests.

I understand. I wanted to clear the simpler tasks. Local compilation is just a first task. The idea is to eventually have this officially support.

I can test the bloom fork locally. I guess this simple addition into rosdistro can be proposed if we get a pull request for bloom adding support for arch.

No, the build farm does not trigger any jobs based on third party packages. It is assumed that all external dependencies are ABI compatible and don't require a rebuild of downstream packages. There is no custom API in ros_buildfarm. But you can use the Jenkins API to trigger specific builds if necessary.

For a rolling release distro as arch ABI compatibility can't be ensured, so this rebuild process would be necessary. I don't know much about Jenkins.

Do you see the whole system feasible? I would like to work on this but I'm unsure about the reception of the addition of support for Arch.

[dirk-thomas]

Adding a trigger mechanism to ros_buildfarm to rebuild a set of jobs sounds not too difficult. I don't see a reason why this wouldn't get merged.

[bchretien]

By the way, can you think of a simple command to list all non-official (aur) non-ros packages which would need to be provided?

I made this simple script (it depends on pactree).

[protobits]

Nice!

Here's the output for ros-indigo-desktop-full:

collada-dom console-bridge gazebo ignition-math ignition-transport libccd log4cxx pcl poco python2-catkin-pkg python2-empy python2-rosdep python2-rosdistro python2-rospkg sdformat tango-icon-theme urdfdom urdfdom-headers uuid

By the way, do you know of a free hosting service where I could put up a temporary repository to hold these packages? El jun 27, 2016 9:24 AM, "Benjamin Chrétien" [email protected] escribió:

By the way, can you think of a simple command to list all non-official (aur) non-ros packages which would need to be provided?

I made this simple script https://gist.github.com/bchretien/c77d5774a5c781fc7dd95f0333dfd584 (it depends on pactree).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-228730685, or mute the thread https://github.com/notifications/unsubscribe/AAJ3qoWFpJkSNRONSC2bCVHcDa-kDZtJks5qP8DrgaJpZM4HX4nh .

[protobits]

@bchretien which branch of your bloom fork repo should I use?

[bchretien]

@v01d it should be topic/pkgbuild-support.

[protobits]

@bchretien I've downloaded your fork and tried the pkgbuild-support branch. I'm using a rosdep entry generated by rosdep generator with arch included for kinetic. I used bloom-generate on one of my own packages (with many ros deps) and created a functional PKGBUILD which also builds fine. Besides any other aspects you mentioned which could be polished, it would seem that I can reproduce the test you performed.

Now, I'm a bit lost on how to proceed. Is this enough on bloom side? I mean, does the buildfarm actually use bloom-generate and then invokes the required build commands (makepkg in this case)? I've looked a bit into ros_buildfarm but is seems incredibly complex to trace where bloom is invoked and how would I start adding support for building for arch. Moreover, testing ros_buildfarm on my own computer also seems very difficult to me (I don't know much about jenkins, however I do handle docker).

If buildfarm does indeed invoke bloom-generate, I understand that it would be required to generate a PKGBUILD which gets the sources from the GBP repository and not the sources locally. Is that so? In this case, I understand that for Debian a "source package" is first generated and then it is built into a binary package. I'm not sure how would this work for arch. I would imagine that building a source package is nothing more than generating the PKGBUILD, right?

[dirk-thomas]

The build farm does not run bloom. The developer runs bloom to populate the release repo and register the package version in the ROS distribution files. The build farm then pull the sources and Debian control files from the release repo and produces Debian packages.

[protobits]

The build farm does not run bloom. The developer runs bloom to populate the release repo and register the package version in the ROS distribution files. The build farm then pull the sources and Debian control files from the release repo and produces Debian packages.

Ok, now I understand. Thank you.

Definitely for me, modifying the buildfarm code feels out of my capabilities at the moment. So I hope someone else can take this task.

I've been working on getting all non-ROS AUR dependencies compiled and up in a testing repository. I've created a github org github.com/ros-archlinux where all packages are stored. These are forks of AUR git repos. I'm working on "rosarch" branch on each repo. I've built a system which builds packages using TravisCI and uploads them automatically to the testing server. For this I'm using a docker-based archlinux build image (github.com/v01d/archlinux-build). The server URL is http://rosarch.h0l.3s (replace the 0 and 3 with an 'o' and 'e'). This is a free hosting service so I will likely eventually run out of bandwidth if it is used frequently.

I'm still finishing building the required packages but it seems to be going well (after fighting a bit with docker and Travis). By the way, @bchretien, I've left you a comment on AUR, regarding some packages (gazebo dependencies) which I cannot clone. If you can look into it, I can also get that built.

EDIT: all deps for desktop built and uploaded. if the missing repos can be retrieved, I can do the same up to desktop-full.

By the way, @bchretien, maybe it would be better for my to adopt these packages in AUR and push also there to avoid duplication. It would be nice to receive pull request for out-of-date packages to help me maintain these.

[bchretien]

@v01d I've added you as a co-maintainer for the ROS dependencies I currently maintain. That should help setting everything up.

[protobits]

Cool. If you agree I will then merge my rosarch branch to master and push that to have the same state as in GitHub.

On Sun, Jul 3, 2016 at 3:51 PM, Benjamin Chrétien [email protected] wrote:

@v01d https://github.com/v01d I've added you as a co-maintainer for the ROS dependencies I currently maintain. That should help setting everything up.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-230168929, or mute the thread https://github.com/notifications/unsubscribe/AAJ3qge-SncOuTanN06ywI9kpdLvAi6-ks5qSAS3gaJpZM4HX4nh .

[bchretien]

What are the differences on those rosarch branches? Simply version updates?

[protobits]

They're mostly the same, with a few exceptions. I've updated poco to latest stable and also PCL. Moreover, I disabled the apps on PCL to shorten the build. I also patched log4cxx to build with gcc6. Also, all of these repos have a .travis.yml for the build system I mentioned. But that should not bother anything, I think.

On Sun, Jul 3, 2016 at 4:09 PM, Benjamin Chrétien [email protected] wrote:

What are the differences on those rosarch branches? Simply version updates?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bchretien/arch-ros-stacks/issues/50#issuecomment-230169835, or mute the thread https://github.com/notifications/unsubscribe/AAJ3qs2hHzcZ28jibGmxLjXuo62Lk6xCks5qSAjugaJpZM4HX4nh .

[bchretien]

Ok, that sounds reasonable to me! Feel free to apply your changes on the AUR.

[gdlg]

I wrote my own fork of Bloom to support Arch https://github.com/gdlg/bloom but it's very similar to what has been done here. When I started it, I wasn’t aware of this thread. I have posted a message here http://discourse.ros.org/t/who-is-using-ros-on-archlinux/388/6.

For the sources, I moved all of them in a subdirectory alongside the PKGBUILD. I think that’s less messy that referencing the parent directory.

So far, to test it, I forked rosdistro (https://github.com/gdlg-rosarch/rosdistro) and the *-release repositories into https://github.com/gdlg-rosarch. I generated a new “rosarch” track for Arch which essentially duplicates the “kinetic” track and uses the last released version. I could maintain a script which detects new releases in the “kinetic” track and automatically releases new version for the “rosarch” track.

Locally, I had to bump the version of urdfdom and urdfdom-headers to 0.4.2. I also bumped console-bridge to 0.3.2.

I would like to issue a PR to Bloom. Hopefully, Arch could be released for Lunar Loggerhead.