3.2.0: building GUI with cmake fails, force vendors libqscintilla2_qt5

[dvzrv]

Hi! I'm packaging sonic-pi for Arch Linux.

While building 3.1.0 has already been challenging (#1864, #1865, #1908), building 3.2.0 is currently completely impossible.

The cmake setup forces me to build qscintilla-qt5 unconditionally which is already available as a library in the repositories (it was previously just linked against using the qmake setup and that was that). This adds immense turnaround time for building and debugging the build of sonic-pi!

When looking only at the required steps to build the GUI, I'm currently doing the following:

    cd app/gui/qt/
    cp -vf utils/ruby_help.{tmpl,h}
    ../../server/ruby/bin/qt-doc.rb -o utils/ruby_help.h
    cmake -DCMAKE_INSTALL_PREFIX=/usr \
          -B build \
          -S .
    make VERBOSE=1 -C build

This leads to:

[ 90%] Automatic RCC for info_files.qrc
/usr/bin/cmake -E cmake_autorcc /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build/CMakeFiles/sonic-pi_autogen.dir/AutoRcc_info_files_EWIEGA46WW_Info.json
AutoRcc: Generating "SRC:/build/sonic-pi_autogen/EWIEGA46WW/qrc_info_files.cpp", because it doesn't exist, from "SRC:/info_files.qrc"
/usr/bin/rcc -name info_files -o /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build/sonic-pi_autogen/EWIEGA46WW/qrc_info_files.cpp /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/info_files.qrc
AutoRcc: Writing settings file "SRC:/build/CMakeFiles/sonic-pi_autogen.dir/AutoRcc_info_files_EWIEGA46WW_Used.txt"
make[2]: *** No rule to make target '../lang/sonic-pi_pt_BR.qm', needed by 'sonic-pi_autogen/EWIEGA46WW/qrc_SonicPi.cpp'.  Stop.
make[2]: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'
make[1]: *** [CMakeFiles/Makefile2:98: CMakeFiles/sonic-pi.dir/all] Error 2
make[1]: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'
make: *** [Makefile:84: all] Error 2
make: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'

sonic-pi-3.2.0-build.log

After devendoring qscintilla-qt5 from the CMakeLists.txt, I end up with a similar problem, just in a different location (as I don't have to build qscintilla):

[  8%] Automatic RCC for info_files.qrc
/usr/bin/cmake -E cmake_autorcc /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build/CMakeFiles/sonic-pi_autogen.dir/AutoRcc_info_files_EWIEGA46WW_Info.json
AutoRcc: Generating "SRC:/build/sonic-pi_autogen/EWIEGA46WW/qrc_info_files.cpp", because it doesn't exist, from "SRC:/info_files.qrc"
/usr/bin/rcc -name info_files -o /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build/sonic-pi_autogen/EWIEGA46WW/qrc_info_files.cpp /build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/info_files.qrc
AutoRcc: Writing settings file "SRC:/build/CMakeFiles/sonic-pi_autogen.dir/AutoRcc_info_files_EWIEGA46WW_Used.txt"
make[2]: *** No rule to make target '../lang/sonic-pi_pt_BR.qm', needed by 'sonic-pi_autogen/EWIEGA46WW/qrc_SonicPi.cpp'.  Stop.
make[2]: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/sonic-pi.dir/all] Error 2
make[1]: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'
make: *** [Makefile:84: all] Error 2
make: Leaving directory '/build/sonic-pi/src/sonic-pi-3.2.0/app/gui/qt/build'

sonic-pi-3.2.0-devendor-build.log

I don't really understand how the generation of the *.qm files works, but they are definitely not being generated below app/gui/qt/lang/.

AFAICS there is no CI pipeline to build the GUI once a pull request was merged. This would already greatly improve the understanding of how to actually build sonic-pi from scratch (on any OS).

I hope any of the developers can shed some light on this, as the time to debug the build system of sonic-pi is in no relation to the time I spend actually using it :(

[samaaron]

Hi, wonderful that you're looking to package Sonic Pi for Arch Linux. That's amazing, thank-you!

The cmake stuff is definitely very much work-in-progress. It's currently only really stable for building on Windows, and whilst some people have reported some success on various linux distros, it's not something I've spent any time on (yet) and is properly documented (yet).

You're therefore walking in unchartered terrain.

Currently there are some unix-* scripts in the qt dir. These are generically named as they're likely to be fairly close to accurate to both linux and macOS, although it's even more likely they'll need to branch out to macOS-* and linux- and also potentially specific linux distros. Hopefully we can keep as much generic as possible to reduce duplication.

The principle author of the cmake stuff is @cmaughan who has been nothing but amazing and helpful. He's mainly a Windows developer, but I'm sure can help point you in the general direction of cmake-specific fixes. @rbnpi and @SunderB have also been working with these cmake scripts and may be able to help.

I also share your pain and sadness about the time of development of Sonic Pi vs using it. For me that ratio is currently truly awful. Hopefully when we start to stabilise all the build using cmake things will get a lot smoother for future devs.

Thanks for being patient!

[dvzrv]

@samaaron Thanks for the reply! Maneuvring such a large project with different languages surely isn't easy.

I have found the culprit for the above mentioned build steps:

lrelease SonicPi.pro

Needs to be run before running cmake to generate the translation files.

Couldn't this be run from cmake? I think all the shell scripts can then just be dropped (instead of creating more, which makes things more complicated). CMake is able to generate a lot on its own and can also (re)create the build directory. I think it's absolutely fair to depend on a more recent version of cmake though.

Meanwhile I've spent some time on devendoring qscintilla-qt5 as well (this can be made optional by flag). I can open a pull request for this as soon as I figured out whether the package now behaves as it should. The same goes for boost btw. It's typically around on all distros and can easily be found (in specific versions) using cmake.

[samaaron]

Sounds good, although in general, I don't think there's a huge fuss in running a "get things ready to build" shell script before then running cmake.

Still, always interested in streamlining things. The crucial thing is to make sure that any changes to cmake don't break the Windows appveyor build. It'll be great to have macOS and some Linux distro over on appveyor soon too.

I'm still yet to move over to cmake for macOS builds at this stage. I wanted to just get the release out. I'm now working on a bug-fix release which fixes a number of bugs specifically on Windows and some memory leaks which I'm still frantically tracking down. Once this bug-fix release is out, I can then start to spend quality time with cmake on macOS which should help unify things further.

[rbnpi]

I did a patch on the scripts as they were at the tagged 3.2 release, which works for Raspbian and Ubuntu 18.04.4. Since then @SunderB has added a flag to allow libaubio5 build to be switched in or out as some distro have a suitable package already (in a pending PR). I guess something could be addedas a switch for qscintilla2, although there have been issues in the past in using different versions of lib-boost and it may make sense to go with the bundled lib-boost and qscintilla building currently in the script which is what I do. I found it useful to study the windows build doc to help get the linux one going. The build scripts don't need SonicPi.pro any more (just used currently by MacOS build). Basically at present I build the vendor gems with compile-extensions.rb then run unix-prebuild.sh (patched) followed by unix-config.sh followed by cd build and make --build. (I have added the erlang script compile commands to (my) unix-prebuild script). NOTE a commit since the release v3.2 tag also now requires bundler to build the vendor gem extensions)

[dvzrv]

The crucial thing is to make sure that any changes to cmake don't break the Windows appveyor build. It'll be great to have macOS and some Linux distro over on appveyor soon too.

@samaaron That's what CI is for. It would indeed be great to test against a more diverse set of OSes.

@rbnpi Could you please elaborate on and link to what you mean by

has added a flag to allow libaubio5 build to be switched in or out as some distro have a suitable package already (in a pending PR)

and

I have added the erlang script compile commands to (my) unix-prebuild script).

In regards to:

I guess something could be addedas a switch for qscintilla2, although there have been issues in the past in using different versions of lib-boost

The idea is of course to provide user/packager choice. AFAICS lib-boost is not required, but the boost headers are (during build time).

[ethancrawford]

@dvzrv - where does this issue stand now that the most recent release of Sonic Pi is v3.3.1, soon(ish) to become v4.0? Are your questions for @rbnpi still relevant? 🙂 (Just doing a sweep of issues that have been sitting around for a while. Hopefully we can resolve this one for you in some way soon 😄 )

[dvzrv]

I am not sure the questions are still of relevance (or whether the answers are of relevance for the topic at hand).

At any rate, I am still devendoring libqscintilla2_qt5: https://github.com/archlinux/svntogit-community/blob/a8e8f0f6f6cf6c6ae06de87dbbcd588c38b6107e/trunk/sonic-pi-3.3.1-devendor_qscintilla-qt5.patch I am also now devendoring boost, because it is provided system-wide during build time: https://github.com/archlinux/svntogit-community/blob/a8e8f0f6f6cf6c6ae06de87dbbcd588c38b6107e/trunk/sonic-pi-3.3.1-devendor_boost.patch

I think it would be fairly easy to provide a PR for this to become a cmake option.

[ethancrawford]

Sure, thanks for the feedback. Looking forward to comments from Robin or Sam 🙂

[samaaron]

@dvzrv - very happy to see a PR that keeps the build as it currently stands but also adds options to use system qscintilla and boost

[samaaron]

Hey @dvzrv - is this still an issue considering all the changes in the build system in v4?

[dvzrv]

Not sure yet. I have not yet upgraded. Will update this ticket as soon as I do! :)

[lilyinstarlight]

Hi @dvzrv! I've put work into being able to de-vendor dependencies specifically for Linux packaging (including qscintilla, rtmidi, aubio, etc). It's still far from ideal from a packager's perspective, but the basics needed for packaging should now be accessible until stuff can be reorganized to suit developers' and packagers' needs better without making it overly complex

I meant to prepare documentation on it since v4 came with a whole bunch of build-time Elixir dependencies and vcpkg and stuff, but I haven't finished it yet

You can review my NixOS package, however, which should have the big picture of what is needed and I'm happy to answer any questions: https://github.com/NixOS/nixpkgs/blob/e939d920d5f430147b127d9f2e3e6b7dc926b9c3/pkgs/applications/audio/sonic-pi/default.nix

[dvzrv]

@lilyinstarlight Awesome! That is much appreciated! I'll try to get back to you on this as soon as I can. I'm currently buried under several other package upgrades.