minisatip and James Donkey HD DUO

[lars18th]

Hi @catalinii and others,

I've now one unit of the cheap James Donkey HD DUO. And after a lot of testing over several weeks I can finally run minisatip on this device, albeit with fairly limited capabilities. So first of all I want to share the facts I have discovered:

• The unit can run minisatip if you install the last version of the OpenATV firmware version 6.4. One relevant aspect is to use the driver version: 20210607 (kernel 4.4.35).
• To run minisatip you need to use the option -1 1 because the stream can only be readed from the /dev/dvb/adapter0/demuxX and not the common /dev/dvb/adapter0/dvrX.
• The "streaming" mode requires to be enabled using echo 1 > /proc/stb/parameters/ts_tap (already set by the script /etc/init.d/dinobot-pvr.sh).
• The firmware has some bug regarding the background splash image. Therefore you need to kill the showiframe process that is consuming a lot of CPU resources.
• Current minisatip ARM binaries can't be run because a mismatch with the GLIBC version. You need to recompile it with an older toolchain with ./configure --enable-static --host=arm-linux-gnueabihf --disable-dvbca --disable-dvbcsa and make EXTRA_CFLAGS="-DNEEDS_SENDMMSG_SHIM".
• The pid filtering implemented inside the dvb driver is very buggy. With the simple dmesg command you can see a lot of errors. And now I discovered that the driver only supports up to 10 "service" pids plus another 10 "psi" pids. So you have to request the opening of the pids in this order: first you can request up to 10 pids from the range 32-8190, and then continue with another 10 pids from the range 0-31. This is very crazy. But if you request the opening of the pids not using this technique, then you can open only a small number of pids. If you request for example: 100,0,101,1,102,16 then you will recive only pids 100 and 0, and nothing else! So the golden strategy is to open (0-10)service_pids, (0-10)psi_pids.
• But anything else seems to work very well. I can use the two tuners independently, and stream multiple pids with a bandwith of 30mbps from each tuner without troubles.

Therefore, I request if you can help to improve the support for this device:

1. The first "simple" question is to provide a compatible binary. I suggest to add to the releases another ARM target compiled with an older toolchain, and with EXTRA_CFLAGS="-DNEEDS_SENDMMSG_SHIM". I feel several low cost devices with ARM cores will agree. You accept this @catalinii ?
2. The second question is how to handle the "limited pid list". My first idea is to ordering the received pid list, split it in two groups, and send it in the correct way limiting to 10 pids per list. However, a lot of edges cases can appear. So any idea to almost improve the current behaviour with minimal changes?

Please, comment anything you want, and let me know if you want me to perform any concrete test.

[catalinii]

Hey @lars18th,

IMHO we should not really support such devices. There are few reasons for that:

1. the driver is buggy and we will have to cary those patches for a long time which will complicate minisatip and potentially break also this device in the long run.
2. Basically there is no open source version of any of the drivers so we can improve those.
3. Those devices will not get an update. Kernel 4.4 is no longer supported for more than 1 year.

I honestly feel disappointed by my gigablue (quad 4k) that takes 1s to set a pid filter, which makes tuning to a channel a 10s deal.

I understand that the device is cheaper than a regular one (50EUR vs 100-150EUR), but you will pay 10x that difference in time spent to make it work and keep supporting it. IMHO I am ok if we do not support devices for which the company that creates them does not do the minimum of providing a device that can handle the DVBAPI.

[Jalle19]

I agree with @catalinii , some devices are just not worth the trouble. It's already somewhat of a pain tl support AXE devices, and those at least have some unique capabilities (cheap, fast quad tuner with internal switching matrix, custom OS). But an aliexpress dual tuner, meh 🤷

Providing a pre-built binary on the other hand could be done, especially if there are other old Enigma device that would benefit from that. Feel free to make the necessary changes to the GitHub workflow.

[lars18th]

Hi,

Thank you for your comments. Regarding this:

Providing a pre-built binary on the other hand could be done, especially if there are other old Enigma device that would benefit from that. Feel free to make the necessary changes to the GitHub workflow.

OK, and I can do it. However, before changing the /workflows/binaries.yml, we need to update the container catalinii/minisatip-build-image:latest to add the "legacy" toolchain for ARM. The one that I've used searching from the Internet is this: http://tropical.jungle-team.online/utilidades/toolchains.tar.gz.

If @catalinii updates the container and he put the toolchain (this or anything else similar) in one alternative path (like /opt/arm-legacy/), then I'll update the workflow to generate the new target: ARM-legacy.

You agree?

[catalinii]

The issue here is that the toolchain contains an old gcc version (4.8.3). I am trying to use newer C features and not really enjoying keeping old gcc supported.

Why not update openatv: http://images.mynonpublic.com/openatv/7.2/index.php?open=jdhdduo

[lars18th]

Why not update openatv: http://images.mynonpublic.com/openatv/7.2/index.php?open=jdhdduo

Because this version has some troubles with this device. I've tested 7.x and 6.x, and the most reliable version (for running minisatip) is the 6.4 (LTS). I do the same with my other Enigma boxes (Edision and ZGemma). So please, continue supporting old compilers. With embedded devices is quite common to it have only ancient toolchains.

So, direct question: You agree to include inside a separate subdirectory (i.e. /opt/legacy-toolchain/) the older toolchain in the docker container? If you agree then I'll prepare a PR for it.

Regards.

[Jalle19]

What kind of troubles?

[catalinii]

IMHO if we have to use a older version of glibc that could be ok, but we still need a new compiler. Was wondering how that would work with clang.

Which glibc version is openatv running?

[lars18th]

Which glibc version is openatv running?

$ strings /lib/libc.so.6 | grep LIBC --> GLIBC_2.30

This is true on both ARM (JamesDonkey) and MIPS (Edision).

The errors running the current minisatip binaries in ARM are:

./minisatip: /lib/libc.so.6: version `GLIBC_2.33' not found (required by ./minisatip)
./minisatip: /lib/libc.so.6: version `GLIBC_2.34' not found (required by ./minisatip)

The same version in MIPS is not generating any error. Any idea @catalinii ? I don't know how to link to an specific previous version of glibc with the toolchain.

[lars18th]

Hi @Jalle19 ,

What kind of troubles?

With OpenATV 7.x in the James Donkey HD DUO the system seems to run very slowly. For this reason I prefer to stay running 6.4, because I run only minisatip and nothing else (enigma2 is off). I the rest of boxes I continue with 6.4 because I don't need nothing more... minisatip just runs ok.

[lars18th]

Hi @catalinii ,

FYI, the last "binary release" ARM compatible with GLIBC is v1.1.92 (Apr 28, 2022). And the first with errors is v1.2.1 (May 2, 2022). Checking the docker image seems that this is the change that has broked the compatiblity (Apr 29, 2022): https://github.com/catalinii/minisatip-build-image/commit/6314dff098bbf6671f723e2b489836d7d88a610a

It could be that you compile the OpenSSL for ARM with a different toolchain?

[Jalle19]

@lars18th does it run so slow that minisatip can't be used? Seems odd.

[catalinii]

The driver does not seem to have been updates in the last 5 years: https://github.com/oe-alliance/oe-alliance-core/blame/4.1/meta-brands/meta-dinobot/recipes-linux/linux-dinobot_4.4.35.bb so to run minisatip just libc is probably changed between versions (6.4 -> 7.x)

[lars18th]

Hi @catalinii ,

Regarding the Docker Container to generate the binaries:

• I do this check changing the last version of the Dockerfile: using FROM ubuntu:20.04 insted of FROM ubuntu:lunar.
• I compile manually the ARM target with ./configure --enable-static --host=arm-linux-gnueabihf --disable-dvbca --disable-dvbcsa && make.
• With the resulting binary the command objdump -T minisatip | grep -Eo 'GLIBC_\S+' | sort -u displays this:

GLIBC_2.12
GLIBC_2.14
GLIBC_2.16
GLIBC_2.17
GLIBC_2.28
GLIBC_2.4
GLIBC_2.7

• And the binary will run without errors in the device.

Therefore, I suggest these changes:

1. Continue generating the binaries with the "ubuntu:lunar" base.
2. Add the layer "ubuntu:20.04" to compile after new x64 and ARM "legacy" binaries.
3. Add the minimum target GLIBC version as a text file in the asset zip file (executing the objdump command after the compilation ends).

Futhermore, I see that for MIPS platform you are'nt using any Ubuntu package, but a custom "ancient" toolchain. Doing this you can't use new GCC versions, because you're limited to this ancient toolchain. Why not convert this in a "legacy" asset, and use the "mipsel-unknown-linux-musl" stock packages to compile using the Ubuntu toolchain?

Regards.

[lars18th]

Hi @Jalle19 ,

@lars18th does it run so slow that minisatip can't be used? Seems odd.

The OpenATV firmware is not just the "driver" package and the "enigma2" process. Using the 6.4 firmware (remember LTS, so with all updates and the last driver version) the system seems to be more reliable. And this is true without running any enigma process and only minisatip driving the tuners. Any other daemons are still there. So, I don't know the root cause for the difference in the performance. But if the branch 6.4 is more robust than 7.x with some devices without updated drivers and kernels, then I prefer to continue to use this "legacy branch". Don't you think so?

Hi @catalinii ,

The driver does not seem to have been updates in the last 5 years: https://github.com/oe-alliance/oe-alliance-core/blame/4.1/meta-brands/meta-dinobot/recipes-linux/linux-dinobot_4.4.35.bb so to run minisatip just libc is probably changed between versions (6.4 -> 7.x)

In OpenATV the drivers are updated based on the kernel, and not in the firmware version (6.x or 7.x). Therefore, nothing is erroneous if the device uses a 5 years old driver. This driver (linux-dinobot_4.4.35 version 20210607) is fine except for the limited pid filtering handle (and used on more devices). Futhermore comparing my "Edision OS nino+" with dual tuner (C/T + S2) with this dumb "James Donkey HD DUO", the ARM mono-core CPU of the last is three times faster than the dual core MIPS of the first. So, the box is good to drive two DVB-S2 tuners using minisatip. The problem is only the uggly driver.

Anyway, at time the "official" minisatip binaries for MIPS are using an ancient toolchain. But this is not true for the ARM platform. And these binaries have sense only on systems without a compiler. Therefore IMHO it's preferable to offer binaries for a large "legacy" level of platforms. Don't you think so?

Regards.

[catalinii]

1. My point was that minisatip needs a kernel (which contains the DVB drivers) and glibc. Of course other libc-like projects works for minisatip as well. I've not heard of glibc to cause minisatip perf issues (which uses syscalls which are wrappers for the kernel functionality) so my intuition was that 7.x and 6.4 should be similar from minisatip perspective.

Basically what I am trying to say brings back to @Jalle19's point about supporting axe. The short story is that it becomes a hassle. Maybe now is ok to support openatv 6.4, but I am not sure this is a long term solution.

Does the minisatip build with ubuntu:22.04 works ? Will try to move off that old toolchain for MIPS.

[lars18th]

Hi @catalinii ,

I do more tests related to the container:

• Starting a plain ubuntu:20.04 (not the "22.04" or the "lunar" that is "23.04"), I execute the command to download the toolchains:

DEBIAN_FRONTEND=noninteractive apt-get install -f -y wget git autoconf automake make libtool gcc bzip2 dvb-apps libssl-dev pkg-config g++ vdr-dev zlib1g-dev libxml2-dev curl vim zip libc6-i386 lib32z1 curl libssl-dev
DEBIAN_FRONTEND=noninteractive apt-get install -f -y binutils-arm-linux-gnueabihf gcc-arm-linux-gnueabihf
DEBIAN_FRONTEND=noninteractive apt-get install -f -y binutils-mipsel-linux-gnu gcc-mipsel-linux-gnu 

• And this installs a GCC 9.4.0 for both platforms: ARM and MIPS.
• Then to compile to the ARM platform is not a problem: ./configure --enable-static --host=arm-linux-gnueabihf and all will works.
• However for the MIPS, I'have this problem: Then configuration generated by ./configure --enable-static --host=mipsel-linux-gnu enables a target default platform for mip32r2 cpu. And regular Enigma2 STBs are Broadcom, so cat /proc/cpuinfo prints isa: mips1 mips2 mips32r1. And you can check with readelf -h minisatip that the binaries created by the legacy toolchain are Flags: 0x50001007, noreorder, pic, cpic, o32, mips32. While the new are Flags: 0x70001007, noreorder, pic, cpic, o32, mips32r2. Therefore the difference is the ABI: "mips32" instead of "mips32r2".

To solve this problem, I've done some more tests: The most simple is to edit the src/Makefile and add to the CFLAGS the parameter -mips32. This is sufficient to disable the mips32r2 target cpu. You can check it with a simple readelf over the *.o builds. But when you will link for the minisatip binary the GCC will use some prebuild libraries (i.e. /usr/lib/gcc-cross/mipsel-linux-gnu/9/../../../../mipsel-linux-gnu/lib/../lib/crtn.o) that as you can imagine are "mips32r2" because is the "default" for the GCC toolchain. Therefore, at the end the final binary will be "mips32r2" too. This binary can run for a simple ./minisatip --help but it generates and illegal instruction when doing others tasks.

So, if you want to upgrade the toolchain, I feel it's possible. And you can use the stock Ubuntu only if you stay in the "20.04" version. But, to support the MIPS platform, we need to reuse the libraries precompiled from another toolchain for mips32r1. Or recompile the same GCC version (9.4.0) as Ubuntu but with -mips32 instead of -mips32r2.

What you think? Regards.

[lars18th]

FYI @catalinii ,

$ cat config.log

[...]
Target: mipsel-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,
go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-i
ncluded-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-
default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libsanitizer --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-
system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch --disable-werror --enable-multilib --with-arch-32=mips32r2 --with-fp-32=xx --with-madd4=n
o --with-lxc1-sxc1=no --enable-targets=all --with-arch-64=mips64r2 --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mipsel-linux-gnu
--program-prefix=mipsel-linux-gnu- --includedir=/usr/mipsel-linux-gnu/include
Thread model: posix

[catalinii]

@lars18th was thinking more like this: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html

If this could be used to generate statically linked binaries against libc it would be really nice.

[lars18th]

Hi @catalinii ,

Doing some initial testing to compile minisatip for mipsel with zig cc results in some small errors:

[...]
/home/user/ziglang/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig cc -target mipsel-linux-gnueabi -Wall -Wno-switch -ggdb -fPIC -fno-common -Warray-bounds -O2 -I../src   -fsanitize=bounds -fsanitize-undefined-trap-on-error -DMAJOR=\"1.2.\" -DMINOR=\"81\" -DREVISION=\"d968ac2\" -DDISABLE_DVBCSA -DDISABLE_DVBCA -DDISABLE_NETCVCLIENT -DDISABLE_DDCI -DDISABLE_T2MI  -c stream.c -o ../build/stream.o
stream.c:610:5: warning: implicit conversion from 'int' to 'char' changes value from 128 to -128 [-Wconstant-conversion]
    copy16(rtp_buf, len + 0, 0x8021);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/minisatip.h:34:27: note: expanded from macro 'copy16'
        a[i] = ((v) >> 8) & 0xFF;                                              \
             ~ ~~~~~~~~~~~^~~~~~
1 warning generated.
[...]
/home/user/ziglang/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig cc -target mipsel-linux-gnueabi -Wall -Wno-switch -ggdb -fPIC -fno-common -Warray-bounds -O2 -I../src   -fsanitize=bounds -fsanitize-undefined-trap-on-error -DMAJOR=\"1.2.\" -DMINOR=\"81\" -DREVISION=\"d968ac2\" -DDISABLE_DVBCSA -DDISABLE_DVBCA -DDISABLE_NETCVCLIENT -DDISABLE_DDCI -DDISABLE_T2MI  -c utils.c -o ../build/utils.o
utils.c:246:12: error: call to undeclared function 'backtrace'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    size = backtrace(array, 10);
           ^
utils.c:860:32: warning: unknown warning group '-Wstringop-truncation', ignored [-Wunknown-warning-option]
#pragma GCC diagnostic ignored "-Wstringop-truncation"
                               ^
1 warning and 1 error generated.
[...]

You can/want to fix them?

[catalinii]

make EMBEDDED=1 should fix this

I've been looking yesterday at this but zig still misses libssl for all platforms (and libdvbcsa). I will try to see if using clang I can het a static binary that has libc inside

[catalinii]

@lars18th can u provide remote access?

[email protected]

[Jalle19]

@catalinii did you find the CAM performance issue with the remote access? 😄 Good job!

[lars18th]

Hi @catalinii ,

Regarding the specific case of the James Donkey DUO HD, I've sucess compiling with "zig cc". Here the guide:

$ /home/user/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig version
0.11.0-dev.3315+4422af8be

$ export CC="/home/user/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig cc -target arm-linux-gnueabihf"

$ cd /home/user/minisatip
$ ./configure --enable-static --host=arm-linux-gnueabihf --disable-dvbca --disable-dvbcsa CC="$CC"
$ make EMBEDDED=1 CC="$CC"

$ readelf -h minisatip
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x32220
  Start of program headers:          52 (bytes into file)
  Start of section headers:          906636 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         12
  Size of section headers:           40 (bytes)
  Number of section headers:         36
  Section header string table index: 34

$ ldd minisatip
        not a dynamic executable

This generates a really static binary that can run without errors. And the CPU will run with less than 20% with UHD FTA channels. Great! (I recommend to add -1 3 in the command line).

However, when compiling for mips (with zig cc -target mipsel-linux-gnueabi) the compilation runs without troubles, but the linking fails with zig cc LLD Link... ld.lld: error: undefined symbol: __mips_syscall5. And this is an error from the ziglang (missing implementation for the mips target host). I feel that we need to wait until the support will be completed.

Regards.

[catalinii]

I think zig cc comes with an old LIBC that’s why it works.

I tried adding -static flag and this creates static binaries and wanted to test those even with a new gcc/clang.

yeah I hit that problem with zig cc on mips too, but I am testing with gcc/clang 16 too

[lars18th]

Hi @catalinii ,

Using zig cc you can select the "target" GLIBC adding the version. For example zig cc -target mipsel-linux-gnueabihf.2.33 compiles for a target glibc 2.33. And in this case the final linking goes right (with <2.33 the error of __mips_syscall5 appears). But in this case, in my mips OpenATV boxes the resulting binary can't run because this:

$ objdump -T minisatip | grep GLIBC_2.33
00000000      DF *UND*  00000000  GLIBC_2.33  stat64
00000000      DF *UND*  00000000  GLIBC_2.33  fstat64

And this feels to me that is related to the FILE_OFFSET_64 support added in this version. Now I don't know how to overcome this. Anyway this is only for MIPS, as ARM compiles without troubles (and I imagine that X86_64 and PowerPC will do too).

In any case I feel the way of using the zig cc compiler to generate the binaries will be useful. That's because then it will very easy to cross-compile for a lot of different platforms.

Regards.

[catalinii]

can you add -static flag (zig cc -target mipsel-linux-gnueabihf.2.33 -static) and see if that works?

[lars18th]

can you add -static flag (zig cc -target mipsel-linux-gnueabihf.2.33 -static) and see if that works?

This is done "automatically". I'm configuring with ./configure --enable-static --host=arm-linux-gnueabihf --disable-dvbca --disable-dvbcsa CC="$CC" and then ldd minisatip shows "not a dynamic executable". And when I copy this binary to the target box the ldd indicates the necessary (minimal) libs:

./minisatip: /lib/libc.so.6: version `GLIBC_2.33' not found (required by ./minisatip)
        linux-vdso.so.1 (0x77962000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x77928000)
        libc.so.6 => /lib/libc.so.6 (0x777c4000)
        /lib/ld.so.1 (0x558ac000)

I feel this the correct, right?

[lars18th]

Hi @catalinii ,

This works!

$ export CC="/home/user/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig cc -target mipsel-linux-musl -static"
$ ./configure --enable-static --host=mipsel-linux-musl CC="$CC"
$ make EMBEDDED=1 CC="$CC"

$ ldd minisatip
        not a dynamic executable

The final binary is fully static, and with a small size. ~~The only problem detected until now (all works as a SAT>IP server) is that the web UI doesn't renders correctly.~~ All works like a charm! 😉

[lars18th]

And it also works for ARM with:

$ export CC="/home/user/zig-linux-x86_64-0.11.0-dev.3315+4422af8be/zig cc -target arm-linux-musleabihf -static"
$ ./configure --enable-static --host=mipsel-linux-musleabihf CC="$CC"
$ make EMBEDDED=1 CC="$CC"

$ ldd minisatip
        not a dynamic executable

[lars18th]

I feel the next step is to make an alternative workflow file to compile binaries with this technique (without disturbing the current "binaries.yml"). You agree?