IOTstack when building with syncthing the doker-compose.yml throws an error

within the syncthing section of the docer-compose.yml somehow the yml gets written to: "x-port: -...... -...... " instead of just "port: -..... -..... "

When removing the "x-" manually after building and starting the stack again, it works.

Jun 13 '23 18:06 PhilBrk8

First off, if you want to run your system that way, please do so. You don't need to change anything.

However, the x- prefix in the template is not a mistake. I think the most likely explanation for why you're seeing a problem is because your docker-compose is out-of-date.

You can check this by running:

$ docker-compose version
$ docker compose version

What you should see is:

$ docker-compose version
Docker Compose version v2.18.1
$ docker compose version
Docker Compose version v2.18.1

What I suspect you will see will be something like 1.29.2 or earlier for the first command, and possibly Docker saying that "compose" isn't a command for the second.

Please also run:

$ docker version -f "{{.Server.Version}}"
24.0.2

If you see a version string like "20.10.5+dfsg1", the "+dfsg1" bit means your Docker version is pinned and isn't being upgraded on routine apt upgrade.

It might also be useful to confirm Buster vs Bullseye and 32-bit vs 64-bit mode via:

$ grep "PRETTY_NAME" /etc/os-release 
$ uname -a

Everything I'm writing here assumes Bullseye. I don't think the 32- vs 64-bit matters but it's a long time since I tested any scripts on Buster so, if you're still running Buster, please be cautious.

If I have guessed correctly about either docker-compose being out-of-date (ie not v2.18.1) and/or not having been installed correctly (ie not getting the same answer for both commands), and/or Docker being pinned or otherwise out-of-date then I recommend reading PiBuilder's reinstallation document, and doing what it says.

There's a weird history behind all of this which I don't pretend to understand fully but it goes something like this:

Way back in the mists of time, both docker and docker-compose were installed via apt, and docker-compose was a compiled binary. That combo leads to the "pinned" version of docker and neither it nor docker-compose is ever upgraded by routine apt upgrade.
Then came an interim version of docker-compose (1.29.x) which was a Python script. If you installed docker using the official "convenience script" and docker-compose with pip, then docker would be upgraded via routine apt upgrade, and docker-compose could also be upgraded by pip (not that it changed much).
The situation right now is, if you run the official "convenience script" you get docker and docker-compose-plugin, both of which are upgraded by routine apt upgrade. This is "modern" docker-compose which is a compiled binary. However, the convenience script only gets you the docker compose command. It doesn't also set up the docker-compose command, and its absence breaks a lot of scripts and means instructions in documentation don't work as written.

The purpose of the reinstallation document and the scripts it tells you to run is to try to sort out any historical mess and get your system to the point where you have up-to-date docker and docker-compose-plugin, both of which will be kept up-to-date via routine apt upgrade, plus a symlink to make the docker-compose command work.

So, how does all of the above answer your original question? Well, the relevant lines in the syncthing service definition are:

  x-ports:
  - 8384:8384 # Web UI
  - 22000:22000/tcp # TCP file transfers
  - 22000:22000/udp # QUIC file transfers
  - 21027:21027/udp # Receive local discovery broadcasts
  network_mode: host

What the x- prefix means is "this entire clause is commented-out". It's exactly the same as doing this:

  # ports:
  # - 8384:8384 # Web UI
  # - 22000:22000/tcp # TCP file transfers
  # - 22000:22000/udp # QUIC file transfers
  # - 21027:21027/udp # Receive local discovery broadcasts
  network_mode: host

Using x- is an easy way of accomplishing a couple of goals:

Make it simple to switch between host mode (the default for syncthing) and non-host mode:

 ports:
 - 8384:8384 # Web UI
 - 22000:22000/tcp # TCP file transfers
 - 22000:22000/udp # QUIC file transfers
 - 21027:21027/udp # Receive local discovery broadcasts
 x-network_mode: host

Just a subtle rearrangement of x- prefixes!

Document the actual ports in use even when a container is in host mode. Once you learn to interpret it, it's actually telling you that, when syncthing is running in host mode, the container will bind to all those right-hand-side ports (ie the "y" in "x:y"). Not every container has a 1:1 mapping where x=y so this can be a handy reference. Otherwise, you often have to ferret about to find this information (usually when you're trying to solve a port conflict).

However, support for x- depends upon "modern" docker-compose. That's why your question makes me suspect your version is out of date.

In turn, the need for either wholesale commenting-out of ports or something like the x- prefix, was all brought about by modern docker-compose insisting that a service definition could have either a ports list or a network_mode: host clause, but not both.

"Pre-modern" docker-compose cheerfully accepted both an active ports list and and active host-mode statement, with the latter taking precedence. I have no idea why the maintainers decided to make it an either/or choice in "modern" docker-compose. They just did.

Last thing. You might find it helpful to also read IOTstack menu maintenance because you might run into that problem as well.

Hope this helps.

Jun 14 '23 00:06 Paraphraser

I´m pretty new to docker ( - and yaml - as you also might have guessed correctly ;-) ) Thanks for your explanation, I just learned a TON.

Fresh install of 64-bit raspbian, even updated the kernel to 6.1.21, so you guess it´s up to date...

But the docker-compose is at 1.25
And docker pinned to 20.10.5

Thanks again, would have probably taken me quite a lot of time to get to that knowledge on my own which you just provided me with your detailed explanation.

Jun 14 '23 04:06 PhilBrk8

You're welcome. So, with 64-bit Bullseye, you should be safe to run the uninstall scripts for both docker and docker-compose, then the PiBuilder 04 script (which could also be called the "install docker and modern docker-compose" script)m after which your Pi should take care of itself with routine apt update.

Jun 14 '23 05:06 Paraphraser

Thanks again. One more thing: I'm using Fedora on my main machine. Would the scripts break due to "dnf" when they don't find an "apt"?

On Wed, 14 Jun 2023, 07:20 Phill, @.***> wrote:

You're welcome. So, with 64-bit Bullseye, you should be safe to run the uninstall scripts for both docker and docker-compose, then the PiBuilder 04 script (which could also be called the "install docker and modern docker-compose" script)m after which your Pi should take care of itself with routine apt update.

— Reply to this email directly, view it on GitHub https://github.com/SensorsIot/IOTstack/issues/710#issuecomment-1590486701, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV6EJVBOOJUGG5UDUFEFIALXLFCY5ANCNFSM6AAAAAAZFIS5OE . You are receiving this because you authored the thread.Message ID: @.***>

Jun 14 '23 05:06 PhilBrk8

Sorry, no idea. I'm most familiar with systems that have apt, and have some understanding of apk but I've never used a system that uses dnf.

The best advice I can give you is to study the PiBuilder scripts and make parallel changes where possible or find workarounds. When it comes to the Docker "convenience script" (downloaded and invoked by the PiBuilder 04 script), you'll have to consult the Docker documentation to see whether the script copes with Fedora or if Docker has an alternative.

Jun 14 '23 06:06 Paraphraser