sonic-pi
sonic-pi copied to clipboard
sonic-pi-net
(Linux) Packaging
I'm currently trying to package sonic-pi for Arch Linux' [community] repository. However, this proves extremely painful, because of
- conflicting installation information and about a dozen different "package scripts" for various distributions (that all seem slightly different and/or outdated)
- four (!) different make systems (qmake, erlang, something that should probably be ruby bundler and cmake) in use to create (but not install!) this environment
- no FHS compliance (which is bad as distribution usually try to avoid dumping applications in
/opt) - no XDG desktop entry integration
This all makes it very hard to package sonic-pi without major headache.
It would be very beneficial to have a Make-like system, that takes care of all of the building and installing, as maintaining a package with a lot of customization and workaround steps, such as this, is very time-consuming and therefore off-putting to any packager.
There are a few open pull requests, that should be followed through with (e.g. #218, but especially #1291), as they improve on exactly this issue. However, I don't think the latter approach should be chosen for e.g. building the Qt interface. If a standard GNU make is chosen, this can use rake, build the GUI and place things where they should go.
According to FHS, all things below etc should actually end up in /usr/share/sonic-pi, the executables below app in /usr/bin and so on.
I'm currently trying to find a way to make sonic-pi FHS compliant, but I'm not sure it's possible in its current form. I'd definitely appreciate any help and pointers!
@dvzrv you make some excellent points :-)
It's also important to point out that Sonic Pi has never really had a strong focus on making things friendly for developers. All my energy has gone towards making the end user experience as simple, intuitive and as fun as possible. Given that the majority of Sonic Pi users aren't developers, this felt (and continues to feel) a the best focus to have.
However, this has meant that the development side of things has a lot of room for improvement. I'm totally in favour of this - and if you're able to help out in any way, that would be amazing.
Let me address each of your points individually:
conflicting installation information
Currently we don't have the resources to support generic Linux distros. We are able to support Raspberry Pi because it is a single hardware and software platform and there are fewer moving parts. As you can see with all the conflicting installation info - Linux isn't consistent across distros which gives rise to these issues. What you see is the result of lots of people trying to get things working on their particular linux distro and the resulting info is a bit of a mess. I'd love to hear any thoughts you might have on unifying and simplifying this in a way that works for all linux users.
Multiple make systems
As mentioned previously, simplifying the creation of the system has never been a goal. The goal was always to have a single point of installation that anybody could install. On macOS this is an app bundle and on Windows it's the msi installer. How that is created can be magic from the perspective of the user. I'm happy to look into solutions for this, but it would have to be something that was cross platform and worked just as well on Windows and macOS.
No FHS compliance
Again, generic linux has never been a priority. I hadn't even heard of FHS before. I believe @hzulla did some work on this for the previous Debian release - although I'm not sure how that related to the current file structure.
No XDG desktop entry integration
Also, I have no idea what this is and due to Linux not being a target platform it was never a priority. If you see a way of integrating this without affecting current platforms then I'd love to know what might be involved.
In conclusion, I completely take your points and would love any and all help improving these matters!
@samaaron thanks for getting back on this so quickly!
Well, I guess there are many possible entry points (apart from a make-system automating the installation process later... and if you use cmake, you can do this cross platform!).
The first one to tackle is probably to have a look at util.rb and see how paths can be separated out and/or made global (as especially on Linux, the philosophy is to not bundle everything, but make software as generic and reusable as possible). It would be great to find a way to do this, without patching! This would mean to set global paths for the Qt GUI, sox (which is not directly mentioned as a dependency btw), osmid (now packaged for Arch at least) and a bunch more I probably forgot. Generally, I think that the application should be more reliant on PATH (as this should be set for all sane Linux distributions).
Writing this, I realize, that one (very easy) way to get around the whole "multiple installation instructions" mess is to only provide a single location in the repository, where dependencies (and their minimum versions) are listed. Installation instructions should be moved to a wiki. This would already minimize the noise significantly.
The XDG stuff is basically only a bunch of flat files and pictures, that have to be installed to certain locations (e.g. /usr/share/applications/sonic-pi.desktop, /usr/share/pixmaps/sonic-pi.png, etc.). So that's straight forward and can just be not installed on macOS/Win.
The Qt GUI should get an install target. I'm not a huge fan of cmake, but I think it will get the job done (for all platforms). Another option could be meson. In any case, it would be great to drop qmake!
Install targets have to be applied to all components of sonic-pi. This means, that a make system will take care of building (also help files) and placing of built files, while discarding the (unneeded) source files. Currently everything is moved (which is messy and unneeded).
This would make all install scripts obsolete (e.g. those in app/gui/qt)!
To really have a deeper look into this, I need to understand a couple of things first though:
- why is there a part written in erlang? Was this not possible in ruby? It needs compilation and then be moved somewhere, where executables reside (e.g.
/usr/bin), while settings its PATH in util.rb. - What is the usual entry point for the application? I assume it's the Qt GUI, right? To be able to place it below
/usr/bin, its internal pointers to the ruby part of the application need to be made absolute (btw. in mainwindow.cpp I wonder why most of the includes from mainwindow.h are done again and then mainwindow.h is included?!) before compiling it. - How does
app/gui/htmlrelate?
There are probably more things, but sieving through this project is a tough cooky! I need a break ;-)
In any case: Using a cross-platform make system such as cmake or meson would be of benefit and improve the current situation a lot.
Hey there. I did address some of these things in the Debian packaging. You can see my packaging choices described in the Debian source package's README.
After the Debian packaging, I had tried to make all compontents of Sonic Pi using a Rakefile to simplify building all dependencies. That effort ultimately failed, partly due to the rapid speed of the development compared to my ability to get the Rakefile done.
Sonic Pi is a collection of a wide range of software components (core in ruby and erlang, gems in ruby and C, synths in clojure, GUI in C++/Qt, help documents from Markdown, etc.), some of which require specific patches curated by @samaaron.
Distributions usually think differently about software packaging, they try to make everyone use standard packages wherever possible and reject those who don't follow that advice. My Debian packaging attempt took most time where I tried to get Debian to upgrade outdated software dependencies, de-couple the ruby gems required by Sonic Pi, use those delivered by Debian instead and/or to convince Debian that Sonic Pi requires a specific earlier variant of a gem.
Looking back, my best guess would be to avoid using traditional distribution packaging rules altogether and move to Snap / Flatpack instead. (That still wouldn't help on Raspbian, though.)
All in all, Linux packaging is terribly frustrating for a large and interwoven software package like Sonic Pi. It requires bureaucratic processes and the willingness to adhere to each distribution's specific quirks and rules. There's a reason why it's easier to get software distributed to Win10 / Mac.
As mentioned in https://github.com/samaaron/sonic-pi/pull/1291 I'd argue for using a Rakefile as a viable cross-platform build tool. It works on all platforms, you can adapt it like hell, it is written in ruby, which is a scripting language that you will install on your computer anyway if you want to build Sonic Pi, so it is a very small dependency compared to cmake / GNU Make / autotools / whatever.
@hzulla Thanks a lot for the README pointer! I'll take that into account when trying to build my package!
That being said, my suggestion with cmake/meson is my personal bias of things I've seen and worked with so far. I honestly can't evaluate, if ruby rake makes more sense in this context, as I don't really use ruby. But it sounds like a great approach! Anything that betters the modularity and lowers the complexity of the project is good! :)
I'll have to evaluate, if it makes sense for me to patch the installation into position (time wise probably not ;-) ).
I have to add: Many other projects, that I have packaged so far have managed to achieve decent cross-platform builds (e.g. supercollider). It is possible! It's just not very easy, when the project was not started that way (which is not to say, that it shouldn't be tried!).
@dvzrv do note that my Debian packaging was made before the Erlang component became part of Sonic Pi.
@hzulla noted!
@samaaron which components below etc are in use in version 3.1.0?
@dvzrv most of the stuff in etc is used at runtime:
- buffers - contains a wav file that drives the random system
- doc - only used at build time
- examples - also only used at build time
- samples - all the built-in audio samples
- snippets - hidden feature, but planned to be used in the future
- synthdefs - contains both compiled synths (required at runtime) and the designs to create them
- wavetables - not yet used (@xavriley was looking at getting these working a while back)
@samaaron thanks for the info!
I'll check, how much work it'll be to do the patches, and if they can be easily transferred to other distributions (which I think they should).
Okay, my first attempt can now be used on Arch Linux. I have only done the patching for the GUI and the ruby part (and most likely I forgot stuff), but no real testing (in a use-case situation) yet.
As you can see in the first patch: It's not that much. The second one however is a little more extensive and it would be great, if the paths could be set in a more useful and unified location.
It would also be great, if the main ruby scripts could be moved to /usr/bin, so the PATH environment variable can be used, instead of a long hierarchy of folders to find things.
For "kind of" FHS compliance, I had to move files to other directories than intended by the project, which can be seen in the package() function. Those make some base directories obsolete and move everything to /usr/share/sonic-pi or /usr/share/doc/sonic-pi
@dvzrv - this is great.
Do you have any suggestions on how we could modify the current directory hierarchy to make FHS compliance a little easier?
Maybe we could have some config file to specify where files and folders are located?
@samaaron I have moved most things to /usr/share/sonic-pi
and have not yet factored out the scripts for example...
ls -l /usr/share/sonic-pi
book
buffers
help
html
images
lang
samples
server
snippets
synthdefs
theme
As the GUI went to /usr/bin/sonic-pi, I have moved the ruby folder contents to the server folder above.
I guess the book folder should actually go to /usr/share/doc/sonic-pi instead (will fix that in another release).
The same might be the case for the help and html folders...
ls -l /usr/share/doc/sonic-pi
cheatsheets
examples
synthdefs
tutorial
Note, that the server directory might actually be located below /usr/lib on other distributions. This is just my wild guess as to where to put things for now.
Apart from the launch popup window that never closes, everything seems to work though!
@SunderB that's not the worst idea. I wonder though, how to incorporate this.
Currently the installation requires for etc/buffers/rand-stream.wav to be in the set location and only after that, I can patch all the ruby files. This is another thing to decouple!
In any case, further moving towards better distributability would be a great thing, as the above mentioned patches (most likely) have to be rebased on every new version.
FWIW, I'm currently also trying to create a package for FreeBSD and running into some of these same issues.
@swills feel free to check out my PKGBUILD and the accompanying patches. Some of the choices should be applicable for FreeBSD as well.
@swills What's the status? I was wondering recently as well if I can run this wonderful piece of software on FreeBSD.
Not really specific to FreeBSD, but in answer to the original issue, I'm working on a reorg of the source tree and cmake based build system for everything, the current WIP is in my fork of the repo.
Hi! Sorry I've not been active in a while, been a bit busy with stuff and occupied by other projects. Anyway, based on my fairly small amount of dev experience with Sonic Pi, here's some ideas for sorting some of these points:
-
Conflicting/out-of-date build scripts (Linux especially) The rakefile idea seems like a good one to me. I've been working on Rakefiles to build SP (based on/inspired by hzulla's work), which is in my fork of SP in the branch
patch-4-rakefile. (It's an early WIP and I'm not sure of the conventions of Rakefiles, so it might not be great at the moment)Maybe we could have separate make files for the server (there's already one to run the server tests), and the Qt GUI? Then we could even build them separately!
-
No FHS compliance This might require a bit of work, as you can probably tell! Maybe you'd have to specify different root paths (ie. for server ruby files, config files, gui etc.) for different OSes (i.e. one set for Windows & Mac, another set for Linux); maybe that work has been started by the people who've packaged SPi before.
Perhaps could give some options when compiling/installing: FHS compliant; all in one style (like what it is now on other platforms like Raspberry Pi and Windows); or even a portable Linux version (with config files and runtime dependencies bundled in the SP folder, a bit like the windows portable ver)?
-
No XDG desktop entry integration Here's the .desktop file I made for Sonic Pi on my computer:
[Desktop Entry]
Name=Sonic Pi
GenericName=Sonic Pi
Comment=The live coding synth for everyone.
Exec="/home/sunder/Documents/Coding/SonicPi/sonic-pi-3.1.0/run-debian-app"
Icon=sonic-pi
Type=Application
Terminal=true
Categories=AudioVideo;Audio;Development;
X-Desktop-File-Install-Version=0.23
We could make a ruby file to generate a desktop file like this (with the chosen installation path) and put it in the right place (based off https://developer.gnome.org/integration-guide/stable/desktop-files.html.en, I think that's /usr/share/applications for a whole system install, or ~/.local/share/applications when installing for one user, for GNOME at least), and also put the icon in the correct place, then we have Sonic Pi in application menu with an icon :)
(based off https://developer.gnome.org/integration-guide/stable/desktop-files.html.en, I think that's
/usr/share/applicationsfor a whole system install, or~/.local/share/applicationswhen installing for one user, for GNOME at least)
I think it is that for most desktop environments out there. Certainly the default for GNOME, KDE Plasma, Cinnamon, and I'm fairly sure also Xfce and MATE. Confirmation for other desktops would be nice.
and also put the icon in the correct place, then we have Sonic Pi in application menu with an icon :)
Application icons are usually provided by icon packs that either come with the DE or are installed by the user. Nonetheless, the .desktop file could include a path to a PNG or SVG icon file, should one not have been provided.
Just my 2 cents ;)
I can help get the desktop file correct - I've been working with these for some time. A 3rd party application should ship its own icons and install them in the 'hicolor'theme.
On Tue, 18 Dec 2018, 21:52 outfrost <[email protected] wrote:
(based off https://developer.gnome.org/integration-guide/stable/desktop-files.html.en, I think that's /usr/share/applications for a whole system install, or ~/.local/share/applications when installing for one user, for GNOME at least)
I think it is that for most desktop environments out there. Certainly the default for GNOME, KDE Plasma, Cinnamon, and I'm fairly sure also Xfce and MATE. Confirmation for other desktops would be nice.
and also put the icon in the correct place, then we have Sonic Pi in application menu with an icon :)
Application icons are usually provided by icon packs that either come with the DE or are installed by the user. Nonetheless, the .desktop file could include a path to a PNG or SVG icon file, should one not have been provided.
Just my 2 cents ;)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/samaaron/sonic-pi/issues/1864#issuecomment-448366104, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXVi7bSeG4UT4k5tO8xM-XzjO0_47W0ks5u6VWngaJpZM4S9wQ2 .
About the old Linux build instructions, do you think we should shift them to the wiki? I've looked at the supercollider repo and it seems like they have build instructions in readme files and the wiki.
Apologies for dragging up an old issue again - I'm personally interested in hearing about whether this is still something worth discussing. @samaaron - you've mentioned you're not keen on affecting the build process, and yes, we don't officially support Linux - but you did seem open to suggestions made above. Shall we all continue the above discussion still? @SunderB - I'm wondering about the same thing. Perhaps the wiki is useful as the primary location for troubleshooting, development and contribution information... it does mean it's a bit more fiddly to manage though IMO! 😅
and yes, we don't officially support Linux
Raspberry Pi OS is Linux and it is offered on the website in the download section. A streamlined build and installation route is beneficial to all sides :)
Tomayto tomahto 😂 Yes, I didn't make that specific clarification 🙂 The difference is that Raspberry Pi OS is a single OS is run on a single computing platform that is much easier to target than X number of Gnu/Linux variants and hardware platforms 😉 And yes, I agree, streamlined is always better 😄
Hi @dvzrv, to be clear, whilst Raspberry Pi OS is a variant/distribution of Linux it isn't actually Linux - there are many other distributions too. Unfortunately we don't currently have the resources to support all of them and are focussing on the Pi for now.
We do have a streamlined build and installation route and Sonic Pi is easier than ever to build. Currently you just need to install a bunch of deps, then run our build scripts. There's still a little bit of tidying to be done to the instructions prior to the release of v4, but this will get you very far and we'd be happy to hep out wherever possible: https://github.com/sonic-pi-net/sonic-pi/blob/dev/BUILD-LINUX.md
there are many other distributions too. Unfortunately we don't currently have the resources to support all of them and are focussing on the Pi for now.
I understand. However, (given a build process that does not vendor and is generic) a Linux distribution will do this work for you :) That's where packagers for these distributions come in and provide binary packages to their users. I absolutely do not expect any project to understand any Linux distribution's packaging process in detail (unless the project's build system interferes with it in some fundamental way).
We do have a streamlined build and installation route and Sonic Pi is easier than ever to build. Currently you just need to install a bunch of deps, then run our build scripts. There's still a little bit of tidying to be done to the instructions prior to the release of v4, but this will get you very far and we'd be happy to hep out wherever possible: https://github.com/sonic-pi-net/sonic-pi/blob/dev/BUILD-LINUX.md
With streamlined I am referring to the use of e.g. cmake for the entire build and installation process. Currently the build steps include the execution of linux-prebuild.sh which builds vendored versions of libraries (that are available on any Linux distribution), vendors ruby libraries, builds erlang code. Afterwards linux-config.sh is called and configures cmake for the further build process. Finally, there is no installation target (the reader is left to execute a binary in the build directory).
I'm certainly not blaming you here, but I would like to point out that those scripts make the build process very intransparent and all of this can and should be achieved using cmake directly (if you choose to use cmake as your main build system), as it provides the cross-platform flexibility that has been re-implemented and hardcoded in those scripts (from what I can tell). Calling per-OS scripts and cmake is not a streamlined build process unfortunately.
If I find time this week I'll prepare a PR for the possibility to devendor some libraries via options to cmake (see https://github.com/sonic-pi-net/sonic-pi/issues/2278#issuecomment-939495796), which can already enable me to drop two patches, but to make the configuration of paths etc. more configurable (see initial description), more work is needed.