Proposal for a mechanism to install and manage multiple versions of binary tools

[ilg-ul]

Version 4.x of GNU MCU Eclipse plug-ins will soon be out. In addition to ARM, it'll also support RISC-V devices.

For future versions I'm considering adding a mechanism to install/update/manage multiple-versions of ARM & RISC-V toolchains, OpenCOD, QEMU, etc.

Both command line and GUI solutions should be considered, with the CLI version having the top priority.

The tools should be very simple to use, and prevent mistakes, like using a 64-bits binary on a 32-bits system.

The planned interface is inspired by npm. The descriptions for the binary tools can be stored as npm packages on the npm public registry.

The typical use case might look like:

$ xpm install @gnu-mcu-eclipse/arm-none-eabi-gcc

which will install the latest version of the ARM toolchain, in a default location.

A specific version of the binaries can be installed by explicitly giving the version, using the npm syntax:

$ xpm install @gnu-mcu-eclipse/[email protected]

The result of the install is a 3 levels hierarchy, <scope>/<name>/<version>, for example gnu-mcu-eclipse/arm-none-eabi-gcc/6.2.1 for the above command.

For more generality, the xpm tool should be able to install binaries in 3 locations:

• in the current folder (for example when used with a studio like application)
• in a folder below the user home (for example ${HOME}/opt)
• in a system folder (for example /usr/opt/)

The supported binaries are compressed archives, stored on any public server, for example the ARM server, or GitHub Releases. The actual npm packages do not need to include the binaries, links to the actual addresses should be enough.

Any comments/suggestions are highly appreciated.

@jantje?

[jantje]

The folder structure looks really similar to what arduino does. Arduino stores their stuff in /users/[user]/AppData/Local/Arduino15. This is a system central place. I don't like this because

1. Arduino does not support multiple versions of the same tool making it impossible to have 2 locations using a different version. (Sloeber allows you to use different versions of the same platform->tool)
2. I don't like the "it is somewhere on my system and I don't know where.
3. Downgrading is impossible.
4. Upgrading centrally enforces you to upgrade everything. Once there you have packages[provider]\tools[tool name like arm-none-eabi-gcc][version] Sloeber puts the data in [Eclipse install]\arduinoPlugin and then follows the Arduio folder names.

The install goes different in Arduino world. The biggest question is "what do I need to install". The nice thing of starting from boards is that you can tell which "toolchain" you need. So a platform uses tools (of course with versions) and the tools are defined in a json as follows

      "tools": [
        {
          "version": "5_2-2015q4",
          "name": "gcc-arm-none-eabi",
          "systems": [
            {
              "url": "https://github.com/adafruit/Adafruit_nRF52_Arduino/releases/download/gcc-5_2-2015q4/gcc-arm-none-eabi-5_2-2015q4-20151219-mac.tar.bz2",
              "checksum": "MD5:603bcce8e59683ac27054b3197a53254",
              "host": "i386-apple-darwin11",
              "archiveFileName": "gcc-arm-none-eabi-5_2-2015q4-20151219-mac.tar.bz2",
              "size": "96372129"
            },
            {
              "url": "https://github.com/adafruit/Adafruit_nRF52_Arduino/releases/download/gcc-5_2-2015q4/gcc-arm-none-eabi-5_2-2015q4-20151219-linux.tar.bz2",
              "checksum": "MD5:f88caac80b4444a17344f57ccb760b90",
              "host": "i686-linux-gnu",
              "archiveFileName": "gcc-arm-none-eabi-5_2-2015q4-20151219-linux.tar.bz2",
              "size": "92811866"
            },
            {
              "url": "https://github.com/adafruit/Adafruit_nRF52_Arduino/releases/download/gcc-5_2-2015q4/gcc-arm-none-eabi-5_2-2015q4-20151219-linux.tar.bz2",
              "checksum": "MD5:f88caac80b4444a17344f57ccb760b90",
              "host": "x86_64-pc-linux-gnu",
              "archiveFileName": "gcc-arm-none-eabi-5_2-2015q4-20151219-linux.tar.bz2",
              "size": "92811866"
            },
            {
              "url": "https://github.com/adafruit/Adafruit_nRF52_Arduino/releases/download/gcc-5_2-2015q4/gcc-arm-none-eabi-5_2-2015q4-20151219-win32.tar.bz2",
              "checksum": "MD5:32d950225b6c7c886f6225c1fc578934",
              "host": "i686-mingw32",
              "archiveFileName": "gcc-arm-none-eabi-5_2-2015q4-20151219-win32.tar.bz2",
              "size": "102981732"
            }
          ]
        },

As you can see from this abstract versioning and platforms have been taken into account. As this is from package_adafruit_index.json the tool will be installed at [eclipse install]\arduinoPlugin\packages\adafruit\tools\gcc-arm-none-eabi\5_2-2015q4 As Sloeber starts from boards which are from platform which are defined in the json files sloeber knows what to add to the path before building/uploading

[ilg-ul]

ok, very nice, if the Arduino world is happy with this, fine, but to me this sounds very specific.

as a generic tool, xpm installs any generic binaries, in a given location. actually installing these binaries is only the first and simplest xPack application, since I plan to use xpm to install source packages and compose projects from components.

somehow similar to npm which manages node.js modules, xpm will manage xPacks. and by manage I mean it will handle dependencies, since packages can ask for other packages to be present, with specific versions or version ranges. packages content is unspecified and can be anything (source files, metadata, tools, etc), all that matters is to have all dependencies satisfied. a higher level tool is expected to make something useful with the content of the xPacks.

in practical terms, what can be used from the Arduino/Sloeber design in the xPacks design?

[jantje]

in practical terms, what can be used from the Arduino/Sloeber design in the xPacks design?

The concept of a json file defining the packages that need to be known by CDT. (decide what to install) After all once the packages are installed CDT needs to know which packages(that is inclusive version) to use for which project and needs to know where to find these. The concept of platform.txt because you need to specify which commands you need to run to build.(tell the higher level tool how to make something useful with the content of the xPacks) The concept of menu's to influence the build/upload/run/debug or whatever you need eclipse to do. The concept of putting this information in the environment variables and configure cdt from environment variables. Whether you download and extract a archive or run a xpm command is a pure implementation matter. Extending Sloeber to also support xpm commands could be a win win.

Looking at this from a conceptual point of view, I think that as a IDE provider there are 3 main strategies to go around the problem of multi toolchain development.

1. Don't care (this is what CDT does -eclipse legal enforced-)
2. Take control and become the single point of contact to extend/change the toolchains (this is what platform io does)
3. Have a documented way of doing things and let others provide toolchains (That is what Arduino/Sloeber does)

The advantage of 3 is that "others do the work" the disadvantage is that "you need to specify and document how they can do the work" This means figuring out a framework, document it and "making it work" The "target" framework needs to satisfy following requirements (in Arduino there is also a library framework.) I'm using target as it maps very well on the target as defined in CDT with the work done by dough for his arduino implementation.

1. Let the end user select targets to install/remove.
2. The tool will Install/remove tools based on the installed targets
3. Let the user create a project based on a target. (note in cdt it can also be a configuration)
4. The tool will select the toolchain and build commands based on the target
5. let the end user select build options (limited by the selected toolchain)
6. The tool will execute the build taking into account the build commands and build options using the correct toolchain.

I think you realize this is a lot of work so reuse is good advice :-)

[ilg-ul]

you realize this is a lot of work so reuse is good advice :-)

reuse is always a good advice, as long as the desired functionality is covered. and the trick is to figure out how to do it. ;-)

The concept of a json file defining the packages that need to be known by CDT.

if you make a step back and try to see the broader picture, this is the concept of some metadata associated with the useful data available in the package.

npm defines a package as a folder or an archived folder that has a package.json file with a few mandatory properties, like name and version.

in a true object oriented way, xpm packages are npm packages with a mandatory xpack property. it's as simple as that. the complicated part is the semantic behind it...

platform.txt ... menus ... target

these are related more to project management and to the builds than to the packages themselves.

fyi, for builds, I consider a tool called xmake, which should be part make, part cmake, part wh knows what. an experimental xmake is https://www.npmjs.com/package/xmake. I started to implement features to import/export Eclipse projects, but I had to delay it for a while.

xpm, as an xPack package manager, should be be able to manage generic packages, and preferably delegate the project and build management to other tools.

similarly, npm does a fairly good job on managing node.js modules, including executable modules; npm itself is such an executable module, and xpm (https://www.npmjs.com/package/xpm) follows the same path.

neither npm nor xpm are specific to an IDE, and even less to Eclipse CDT.

3 main strategies to go around the problem of multi toolchain development.

the multi toolchain problem is a specific version of the multi-tool problem, for example there can also be multiple versions for the OpenOCD binaries, QEMU binaries, SEGGER J-Link binaries, and so on.

if projects are also xPacks, it shouldn't be difficult to add explicit dependencies to specific tools to be used during the project lifetime using the xpm dependency mechanism. this dependency mechanism can be direct, i.e. the project can have an explicit dependency to a specific tool, or indirect, for example the project can have a dependency to another xPack which defines a class/group/category/etc the package adheres (like Arduino platforms, targets, etc), and that package to have explicit dependencies to different tool (like toolchains).

I'll try to prepare some more detailed specifications for the xpm install command, and see how much of the Arduino/Sloeber experience can be reused.

[jantje]

I see lots of ideas used in the Arduino way coming back in your story. I'm looking forward to your detailed specifications :-)

[jantje]

As you can see here in the gui sloeber supports multiple versions of platforms. here you see 3 installed avrdude versions

I do not support multiple versions of libraries because Arduino libraries are selected based on include which does not support multiple versions. However this is just me who opted for this limitation.

[mgiaco]

Why not realize something like virtualenv does for python. http://blog.conan.io/2016/08/04/Conan-virtual-environments-Manage-your-C-and-C++-tools.html

[ilg-ul]

yeah, I did not use python lately, but I read that virtualenv finally allowed to use different version. I guess it is functional, but it is still a later addition to python.

xPack is more like npm, which by design handles multi-versions quite well.

[jantje]

I still have this on the back of my mind. I propose the following: If you provide a number of "install packages" and their "compile commands" I'll try to get it to work in Sloeber.

[ilg-ul]

I'm glad you asked, since I'm working on it, but unfortunately it is not the top priority.

for now I have a preliminary tool (xpm install) that installs the RISC-V toolchain (https://github.com/gnu-mcu-eclipse/riscv-none-gcc-xpack).

$ xpm install @gnu-mcu-eclipse/riscv-none-gcc --global

the idea is that all packages are always downloaded via a cache to a central repository in the home folder.

• some tools may be thought to inspect this central folder, it is not a big deal to process some json files
• xPack based projects will have explicit references to given versions, and in this case links to the binaries will be added to the referring project, for convenient usage.
• if really needed, similar links can be added to a system folder (like /usr/local/bin, so the given version of the tool becomes visible for all.

my first use for this tool is to install the toolchain and qemu on travis, to test projects generated by a template project (https://github.com/micro-os-plus/sifive-templates-xpack)

hopefully these days I'll test it on other platforms and publish the result.

[jantje]

To handle multiple versions Sloeber uses fqn all the time. This makes there is no path setup needed and this helps a lot on windows (where tools like cygwin can cause lots of troubles) I'll first try to install this xpm package in the sloeber tree as show above.

[ilg-ul]

I'll first try to install this xpm package in the sloeber tree as show above.

which package? what exactly do you mean by this?

and please be warned, the xPack tools are still experimental. for now I only used them on macOS, did not try on other platforms, most probably fail.

and the sifive-template-xpack is currently half xpack, half npm module (it handles dependencies via npm instead of xpm, since xpm is not ready).

I'll try to improve xpm this weekend.

[jantje]

which package? what exactly do you mean by this? I'm assuming xpm install @gnu-mcu-eclipse/riscv-none-gcc --global installs a "package". I assume it is a gcc compiler so in Sloeber language I'm installing a tool.

I'm running on windows and my first concerns are

• can I run the install command
• can I redirect the install to where I want it
• Can I run the tool in a cdt toolchain For all these actions I don't need anything that "works". Basically it is a feasibility study to see how easily I can extend the current system to install and use tools and toolchains. I won't have lots of time till the end of the year for this, so don't stress.

[ilg-ul]

yes, xpm install should install a generic package. if the package refers to some binaries, they are also downloaded (the toolchain case).

can I run the install command

yes, you should be able to run it, xpm is actually a npm command line application, and should run on any platform

can I redirect the install to where I want it

generally everything is configurable. there are to xPack specific folders, configurable with environment varibles, the repository folder and the cache folder

all archives are dowloaded to the cache, then expanded in the repository folder. this is where multi-version packages sit together in separate folders like .../xPacks/@gnu-mcu-eclipse/riscv-none-gcc/7.2.0

if you start xpm install in a standalone folder, you get a copy of the package there.

[jantje]

I had some time playing around with this. xpm install @gnu-mcu-eclipse/riscv-none-gcc --global installed in C:\Users\jan\AppData\Roaming\xPacks and the binaries are in C:\Users\jan\AppData\Roaming\xPacks\@gnu-mcu-eclipse\riscv-none-gcc\7.2.0-1.2\.content\bin So that looks good. I also tried xpm install @gnu-mcu-eclipse/riscv-none-gcc -C C:\test Which did installl some stuff in c:\test but no binaries. What I'm trying to say is that I used the wrong command or there is something wrong (like my understanding). One of the things I realized when reading through all the doc is that xpm uses a package.json and Arduino uses a package-index.json (in other words a list of package.json's). The reason for this is that you (in Arduino language the provider) must specify which package.json's he is "providing". Providers can then add their index.json here https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls. Can you share your thought on the index concept?

[ilg-ul]

What I'm trying to say is that I used the wrong command or there is something wrong (like my understanding).

no, you did use the right command, and yes, there is something wrong: the xpm install command is not yet fully implemented :-(

Can you share your thought on the index concept?

well, as long as you use it, and are happy with it, I guess it is ok.

however, npm (and xpm) uses a different approach, each package is independent and has its list of dependencies.

when you install an npm package (if you installed xpm, you just did it), you get not only the desired package code, but a folder called node_modules, where all dependencies are installed; each dependency is also independednt and has its dependencies, and this goes as deep as it needs (and works very well).

the interesting part is that in this hierarchy of dependencies, you can encounter multiple versions of a package in different locations, and the JavaScript engine can perfectly make sense of them.

also interesting is how npm isolates each package at its level when dealing with executable tools: each package has a subfolder npm_modules/.bin where are added links (or short .cmd scripts on windows) to the actual tools located in subfolders of node_modules. before executing any sub-process, npm adds npm_modules/.bin in front of the PATH, so local binaries are prefered.

xpm takes a slightly different approach. it also adds a xpacks/.bin subfolder, and adds it in front of the local PATH, but the dependent packages (actually all packages) are installed in the central repository, each version in a separate folder. in the project folder, by default, xpacks includes links to the required versions in the repository, not the packages themselves.

for short source libraries this does not make much difference, but for large binary packages, like toolchains, it saves quite a lot of space.

xpm install is not yet fully functional, it does not install dependencies, so you cannot thest this, but imagine we have a small package with a blinky project, a src folder with a main.c and possibly an include with some header. in the package.json are dependencies to many other packages with source code that are required for a successful build; but not only, there are also dependencies to binary packages, like a toolchain, OpenOCD, possibly a binnary version of make on Windows, etc.

right now, if you use xpm install you get the small blinky package installed in the central repository and into the local folder.

when ready, xpm install will also install all dependencies (sources and binaries) in the central repository, add links in xpacks and, for binaries, add links in .bin. if you run build in the project folder (with something like xpm build), you get two more folders, something like build/something-debug and build/something-release, where the debug/release builds are performed.

I guess this helps you better understand the concept used by npm (and xpm).

as for the index, Arduino may use an index to refer to related packages, but you still need to maintain pages with urls to these index files, then you have to promote these pages, since if people do not know your page with urls, they have no chance to install your packages.

npm fully automates this, all packages are published onto the central server (https://www.npmjs.com), and, if you need a package, you simply search for it, it is exists, it must be there. (well, to be fully correct, npm can also install packages from other sources, like git repos, archives, and even private servers that implement the npm protocol, like bintray, but I would consider them special use cases).

npm is a very powerful solution for JavaScript modules. I'm trying to make xpm an even more powerful and generic solution, mainly for C/C++ projects, but I guess it can be used for other languages too.

of course your milage may vary, Arduino uses a different approach, and there are very elaborate tools, including your plug-ins, that make great use of them.

however, I consider the npm/xpm mechanism more generic, and I plan to further develop the xpm tools. unfortunately I cannot dedicate as much time as I would like to this project, but more work is planned for the next few weeks.

as an immediate use of this technology, I plan to use it to run the µOS++ tests on CI servers, like Travis; I don't know if you can create Arduino projects from a command line terminal, but with xpm it is quite simple, all you have to do is

$ xpm install xxx
$ cd xxx
$ xpm test

or xpm build if you want to run only the compile step.