Linux releases are missing

[kristofmulier]

I've very happy to find the latest releases here:

https://github.com/avrdudes/avrdude/releases

Thanks a lot @mariusgreuel and @dl8dtl for your awesome work! Unfortunately, I can't see any Linux releases. The Linux releases are available in the nightly builds though:

https://github.com/avrdudes/avrdude/wiki/Getting-Nightly-Builds-for-AVRDUDE

It would be great if they would also be available on the release page. I know that usually the Linux distros package their own binaries to fit into their package systems (RPM, dpkg, ...). But some people prefer to get the builds directly from GitHub. It would also be helpful for our usecase: we'd like to download AVRDUDE directly to the computer of our users (the users of Embeetle IDE, a new microcontroller IDE) without going through a package manager.

Thanks a lot ^_^

[mcuee]

The current github action build of avrdude may not be suitable as a Linux release, lower/older version of Linux [eg: Ubuntu 18.04 or Debian 8) may be better. And different Linux distro may have different dependancies. So I am not sure how practical this is. Ref: https://github.com/avrdudes/avrdude/blob/main/.github/workflows/build.yml (Ubuntu:latest and Debian 11).

For example, we only provide Windows binary for libusb projects.

[ygramoel]

I just managed to build avrdude on a virtual machine running Debian Jessie.

It did require a recent version of cmake (3.21.4) that I had to install from source as well. Except for that, the standard instructions on github worked:

$ sudo apt install flex bison ...
$ git clone https://github.com/avrdudes/avrdude
$ cd avrdude
$ cmake -D CMAKE_BUILD_TYPE=RelWithDebInfo -B build_linux
$ cmake --build build_linux

For distribution, I checked the dependencies using ldd:

$ ldd build_linux/src/avrdude
	linux-vdso.so.1 (0x00007ffdc0de5000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff785365000)
	libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007ff78515c000)
	libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007ff784f44000)
	libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007ff784d3d000)
	libftdi.so.1 => /usr/lib/x86_64-linux-gnu/libftdi.so.1 (0x00007ff784b36000)
	libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff7848ec000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff784541000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007ff784332000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff784115000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff783f0d000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff783ce3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff785666000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff783adf000)

For maximum compatibility with different Linux distros, I am tempted to link these libraries statically:

/usr/lib/x86_64-linux-gnu/libusb.a     
/usr/lib/x86_64-linux-gnu/libusb-1.0.a
/usr/lib/x86_64-linux-gnu/libhidapi-libusb.a
/usr/lib/x86_64-linux-gnu/libftdi.a

Would you suggest any other libraries for static linking? For example, libudev or libreadline?

[dl8dtl]

OTOH, if you have an old reference system already available, could you just give the nightly builds a try?

If they work even on your relatively old system, I'd just pick those. I don't think the APIs of the libraries you mentioned have been changed much lately.

The bigger issue is probably that the "modern" build might require a certain LIBC_foo version symbol, for whatever reason.

[mcuee]

@ygramoel I think it is enough to statically link to the following.

/usr/lib/x86_64-linux-gnu/libusb.a     
/usr/lib/x86_64-linux-gnu/libusb-1.0.a
/usr/lib/x86_64-linux-gnu/libhidapi-libusb.a
/usr/lib/x86_64-linux-gnu/libftdi.a

[ygramoel]

Linking the four libraries mentioned above statically works for me. The others, including libudev, are linked dynamically:

$ ldd avrdude
	linux-vdso.so.1 (0x00007ffc1d244000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fddcebd2000)
	libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007fddce988000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fddce76b000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fddce55c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fddce1b1000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fddcdf87000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fddceed3000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fddcdd7f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fddcdb7b000)

This is the link command I used:

/usr/bin/cc -O2 -g -DNDEBUG -L/home/johan/local/openssl/lib -L/lib/x86_64-linux-gnu -Wl,-rpath,/home/johan/local/openssl/lib CMakeFiles/avrdude.dir/main.c.o CMakeFiles/avrdude.dir/term.c.o CMakeFiles/avrdude.dir/whereami.c.o -o avrdude  libavrdude.a -lm /usr/lib/x86_64-linux-gnu/libusb.a /usr/lib/x86_64-linux-gnu/libusb-1.0.a /usr/lib/x86_64-linux-gnu/libhidapi-libusb.a /usr/lib/x86_64-linux-gnu/libftdi.a /usr/lib/x86_64-linux-gnu/libreadline.so -lpthread -ludev

I had to sudo apt install libudev-dev first; your mileage may vary.

It is most probably not a good idea to link libdl or libc statically. I am less certain about the others.

[dl8dtl]

I just tried it on an elderly Ubuntu 16.04LTS:

7% ldd avrdude
./avrdude: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./avrdude)
        linux-vdso.so.1 =>  (0x00007ffecb7c1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f895a1f7000)
        libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f8959fdf000)
        libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007f8959dd6000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f8959bbe000)
        libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007f89599b7000)
        libftdi1.so.2 => /usr/lib/x86_64-linux-gnu/libftdi1.so.2 (0x00007f89597a9000)
        libreadline.so.8 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89593df000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f895a500000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f89591c5000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f895a63c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8958fa8000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8958da0000)

As you can see, it's not the libusb or libftdi stuff that is causing any trouble. It's that "minimum version" symbol in libc, and libreadline.

[kristofmulier]

I just tried the nightly build on this old Debian Virtual Machine - Debian GNU/Linux 8.11 (jessie). Apparently, two libraries are missing:

johan@debian:~/avrdude_nightly$ ./avrdude
./avrdude: error while loading shared libraries: libftdi1.so.2: cannot open shared object file: No such file or directory

johan@debian:~/avrdude_nightly$ ldd avrdude
./avrdude: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./avrdude)
	linux-vdso.so.1 (0x00007ffdfcef8000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbe8139f000)
	libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007fbe81189000)
	libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007fbe80f80000)
	libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007fbe80d68000)
	libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007fbe80b61000)
	libftdi1.so.2 => not found
	libreadline.so.8 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe807b6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbe816a0000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fbe805a7000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbe8038a000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbe80182000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbe7ff7e000)

[dl8dtl]

OK, so your system's libftdi1 is even older than Ubuntu 16, it seems. ;-)

The biggest issue is libc. Everything else, you can just ship the .so that has been used to build along with the binary, and set LD_LIBRARY_PATH. (If you ship your binary without the .so files, it e.g. won't find an appropriate libreadline on a more modern system.)

[dl8dtl]

Try adding librreadline to your above list of static libraries, and see whether the result will run on a current Linux distribution.

[ygramoel]

Linking readline statically works; I had to add -ltinfo to the link command.

/usr/bin/cc -O2 -g -DNDEBUG -L/home/johan/local/openssl/lib -L/lib/x86_64-linux-gnu -Wl,-rpath,/home/johan/local/openssl/lib CMakeFiles/avrdude.dir/main.c.o CMakeFiles/avrdude.dir/term.c.o CMakeFiles/avrdude.dir/whereami.c.o -o avrdude  libavrdude.a -lm /usr/lib/x86_64-linux-gnu/libusb.a /usr/lib/x86_64-linux-gnu/libusb-1.0.a /usr/lib/x86_64-linux-gnu/libhidapi-libusb.a /usr/lib/x86_64-linux-gnu/libftdi.a /usr/lib/x86_64-linux-gnu/libreadline.a -ltinfo -lpthread -ludev

It runs on my Linux Mint 19 Tara, but not on a more recent Ubuntu 21.10. The problem is libtinfo:

$ ./avrdude 
./avrdude: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

[ygramoel]

The original version, with both libreadline and libtinfo linked dynamically, doesn't work on a recent Ubuntu either. However, linking both libreadline and libtinfo statically works.

[kristofmulier]

I'm Johan's (@ygramoel) colleague. Here is the binary he just made:

https://new.embeetle.com/downloads/beetle_tools/linux/avrdude_7.0.0_feaa1c6_64b.7z

It's a 7z compressed folder containing two files:

• avrdude (the executable)
• avrdude.conf

Can you please give it a try on your Linux machine(s)?

[dl8dtl]

Works here, customized Ubuntu 20.04. Can test that old Ubuntu 16.04 tonight.

[mariusgreuel]

@kristofmulier Just to add some history and reasoning to this discussion, even if it may be redundant: The current CMake setup and Github Actions do not support building Linux binaries that are suitable for download, which is why they do not show up in the release artifacts. You can tell from the discussion that your requirement is not just a matter of including the build artifacts in the release, but actually quite a bit of work to get it right.

Technically speaking, the whole build pipeline consists of three things: a) the CMake build system b) the GitHub build actions c) the GitHub release actions

Regarding a) The current CMake setup is designed to support the two most common scenarios: 1) Building Un*x binaries on a defined target system using dynamic linking, suitable for package managers etc., and 2) Building Windows binaries suitable for download using static linking, supporting Windows 7 and up.

See also: https://github.com/avrdudes/avrdude/pull/960#discussion_r872641070

Regarding b) The GitHub build action for Linux is merely a build check, with the intention that the build works on a current Linux system and toolset, throwing no errors and warnings, etc, for the scenario outlined in a). With that reasoning in mind, I picked the latest Debian distro (i.e. the VM 'ubuntu-latest') with the latest GCC toolset and the latest libraries.

Regarding c) As the Linux binaries are not meant for downloading, only the Windows release artifacts (the MSVC builds with the custom libraries) are included in the release.

Now, we have a few new requirements here, such as the Arduino team wishing for statically linked binaries that they can ship with the Arduino IDE (https://github.com/avrdudes/avrdude/discussions/946#discussioncomment-2721265), which is probably what you expect from a Linux binary offered on the release page, or others finding value in building dynamically linked binaries for Windows (https://github.com/avrdudes/avrdude/issues/954). All these requests are related, because they all require changes in a), b), and c).

You are welcome to submit a PR that addresses your needs, however, I would like to point out that the requirements for building Linux binaries suitable for download conflict with the requirements that I outlined above (because you want static linking, support for older distros, etc.). So when you submit a PR, please keep the existing build and CI actions intact, and add a extra jobs that builds the required outputs.

[dl8dtl]

Works on Ubuntu 16.04 as well.

[mcuee]

Interesting from here: *https://github.com/avrdudes/avrdude/discussions/946#discussioncomment-2721265

From the above comments, Arduino has just come out with their version of avrdude 7.0 test release for Linux/macOS and Windows. https://github.com/arduino/avrdude-packing/releases

Their patches on top of the official avrdude 7.0 release. https://github.com/arduino/avrdude-packing/tree/7.0-arduino.1/patches

[kristofmulier]

Hi @facchinm , Please join the discussion.

[umbynos]

Hello guys, @arduino maintainer here, I'm the guy responsible for https://github.com/arduino/avrdude-packing. I would like to answer the questions posted by @kristofmulier here for completeness.

I assume this new repository is the official Arduino repo for AVRDUDE releases.

Yes, as I explained here the new avrdude-packing repo under arduino org contains only the patches that could not be upstreamed, and the release workflow. The source code will be only on the official avrdude repo. Currently, we are internally testing if there are regressions with the new version (7.0-arduino.1).

Which libraries are statically and which are dynamically linked?

As I wrote here these are the dynamic dependencies required at runtime by the binary produced by our CI:

ldd avrdude
	linux-vdso.so.1 (0x00007ffd66d20000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0d66fd4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0d66db5000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0d669c4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0d67372000)

we are statically linking against:

• hidapi
• libftdi1-1.4
• libusb-compat-0.1.5
• readline-8.0
• eudev-3.2.10
• libelf-0.8.13
• libusb-1.0.20
• ncurses-5.9

In order to do so I have prepared a docker container able to crosscompile. Inside it, there are all the toolchains and dependencies required to produce that binary. As of now, the container is not yet public. But I'm planning to open it shortly. So that you guys can look how it works and, if you want, even use it in your CI.

Did you build the releases on an old Linux system, such that it can run on both modern and old systems?

Definitely yes. Our main goal is to have a binary that runs on almost everything. I used the following toolchains for the linux binaries:

• x86_64-ubuntu16.04-linux-gnu-gcc
• i686-ubuntu16.04-linux-gnu
• gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu
• gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf

By the way, the work I've done regarding the build is here if you are curious.

Also, the release are there for testing purposes. So feel free to test it out.

@mariusgreuel

Now, we have a few new requirements here, such as the Arduino team wishing for statically linked binaries that they can ship with the Arduino IDE (https://github.com/avrdudes/avrdude/discussions/946#discussioncomment-2721265), which is probably what you expect from a Linux binary offered on the release page, or others finding value in building dynamically linked binaries for Windows (https://github.com/avrdudes/avrdude/issues/954). All these requests are related, because they all require changes in a), b), and c).

Regarding this, feel free to integrate the work I already did here into your system. Removing the parts that you don't need. When the container will be made public, it will be possible to do so.

[mcuee]

@umbynos

hidapi

What is the version used? Hopefully you are using the release here and not the old one from the signal11 repo. https://github.com/libusb/hidapi/releases

libftdi1-1.4

Can you upgrade to libftdi1-1.5?

libusb-compat-0.1.5 libusb-1.0.20

Wow, that libusb-1.0.20 is really old. It may be okay under Linux but I tend to think you may encounter users under macOS and Windows. 1.0.23 or 1.0.26 would be better (you can probably skip 1.0.24 and 1.0.25). https://github.com/libusb/libusb/blob/master/ChangeLog

[umbynos]

hidapi

I was using 0.11.2

I'm currently trying to update libusb to 1.0.26, libusb-compat to 0.1.7, libftdi to 1-1.5 and hidapi to 0.12.0 I'll keep you posted.

PS since libusb-compat is a mess to compile for win32 I'm using a precompiled version from libusb-win32 project (version 1.2.6.0)

[mcuee]

hidapi

I was using 0.11.2

I'm currently trying to update libusb to 1.0.26, libusb-compat to 0.1.7, libftdi to 1-1.5 and hidapi to 0.12.0 I'll keep you posted.

That is great.

PS since libusb-compat is a mess to compile for win32 I'm using a precompiled version from libusb-win32 project (version 1.2.6.0)

Yes I agree that this is good.

[umbynos]

In the end I managed to:

• bump libusb-compat to 0.1.7
• bump libusb to 1.0.26
• bump hidapi to 0.12.0

libftdi1-1.5 was a pain to update and I gave up. They changed something in the build process and it does not build correctly with mingw and osxcross. So I gave up.

Here's my progress and here a test release

[mcuee]

libftdi1-1.5 was a pain to update and I gave up. They changed something in the build process and it does not build correctly with mingw and osxcross. So I gave up.

Thanks for updating libusb-compat, libusb-1.0 and hidapi. Just wondering why you prefer to use cross-compiler. I also hear the same feedback by OpenOCD side that cross-compiling of libftdi-1.5 is problematic.

If you use MSY2 under Windows (But there is no issue to build with MSYS mingw32/mingw64 -- they actually provide the latest dependancies. It is also used by the avrdude github action. https://github.com/avrdudes/avrdude/blob/main/.github/workflows/build.yml#L232

Version of MSYS2 packages: Ref: https://packages.msys2.org/package/mingw-w64-x86_64-libftdi (1.5) https://packages.msys2.org/base/mingw-w64-hidapi (0.12.0) https://packages.msys2.org/base/mingw-w64-libusb (1.0.26) https://packages.msys2.org/base/mingw-w64-libusb-compat-git (0.1.7)

@umbynos FYI, I regularly build libftdi for Windows and I provide the package for download here since libftdi-0.18 back in 2010. Last time it was difficult to build libftdi so there were many websites referring to my Sourceforge site for the libftdi Windows binary download links. Now it is much easier using MSYS2 and people do not really need to use my build any more. https://sourceforge.net/projects/picusb/files/

[mcuee]

@umbynos Please refer to the fix of libftdi-1.5 cross build here by an openocd developer and check if you can leverage the fix for your cross-build. https://github.com/tarek-bochkati/openocd/commit/16f0cca75959b730979c1faf059f103c22886236

[umbynos]

Just wondering why you prefer to use cross-compiler.

Because we cannot build natively on GitHub actions CI, or at least we could, but the VMs GitHub offers are pretty updated (unfortunately) and do not allow us to build binaries that will work even on older OSes. Furthermore, there is no other way regarding arm and aarch64 architectures.

[umbynos]

Ps I just opened the repo we used to build https://github.com/arduino/crossbuild, if you want to give it an eye

[umbynos]

And also thanks @mcuee for the useful pointers and links. As of now, I'm a bit swamped work-wise. But when I'll have more bandwidth, I'll definitely give libftdi-1.5 a try.

[mcuee]

Just tried out the Arduino release of avrdude under macOS (mac Mini M1) and it seems to work fine. https://github.com/arduino/avrdude-packing/releases/tag/7.0-arduino.2

[kristofmulier]

Thanks a lot @umbynos for the latest release: https://github.com/arduino/avrdude-packing/releases/tag/7.0-arduino.2

Is there a reason why there's no 64-bit version for Windows? I suppose the 32-bit version works just fine on Windows, though.

[umbynos]

Is there a reason why there's no 64-bit version for Windows?

Yeah the reason is that a 32-bit binary will work just fine on 64-bit systems, so why building for a new platform?