"terminals database is inaccessible"

[matteoguglielmi]

Hello,

When boxes 2.3.0 is statically compiled with make clean static, nearly all make test fail (169 over 181) with terminals database is inaccessible.

I have compiled it statically on Debian 11 (amd64 CPU), Ubuntu 24.04 (arm64 CPU) and Ubuntu 24.04 (riscv64 CPU).

The only static compilation that works is on the riscv64 CPU.

Could anyone reproduce this error?

Thank you.

EDIT1:

Changing the TERM variable affects the number of successful tests, yet I still need help getting all of them successful since none of the TERM values can achieve that (on the riscv64 CPU, all tests are successful with TERM=xterm-256color).

EDIT2:

It works when statically compiled on Arch Linux with the TERM variable set to xterm or xterm-256color (no difference). Therefore, the problem seems to be, to some extent, distribution dependent.

[tsjensen]

Thanks for bringing this up and also for analyzing it already!
I don't have access to my machine for the next few weeks, so can't contribute atm. Looking forward to seeing what you find out. 😎

[tsjensen]

Alright, back at my machine. Thank you for your patience!

So far, I can't reproduce the problem. make clean static followed by make test works fine (on Debian).

However, I do remember that that same error, terminals database is inaccessible, plagued me on Windows. I even added a workaround of sorts where you can hard-code a fallback location of the terminals database to use (line 51 in the linked code). Maybe that's a way forward for you?

It seems one needs the "new" terminfo format, with hex directories. The linked script checks for that. Eventually, a copy of the terminfo database should be baked into the static binary, imho.

[matteoguglielmi]

Ok, let's consider a single Linux distribution:

Debian 12 (I've created a VM and installed Debian 12 on it)

Then:

apt update

apt upgrade

apt install flex bison libunistring-dev libncurses-dev libpcre2-dev

reboot

Now:

v2.3.0.tar.gz compiles correctly (make clean static) but fails 169 tests (make test)

v2.2.1.tar.gz compiles correctly (make clean static) but fails 0 tests (make test)

Please try to reproduce this.

Thank you!

[tsjensen]

Thanks for providing concrete steps to reproduce. This confirms my idea from 2 days ago. When I just follow the steps, and TERMINFO is not set, I get the following log output:

WARNING: Terminfo database not found or outdated.
ERROR: Fallback unavailable.
Find a terminfo database that uses a hashed structure (hexadecimal numbers as directories, not single letters).
Set its path in the TERMINFO_FALLBACK variable near the top of the root Makefile.

Also, most tests fail.

When I set TERMINFO correctly, or change the terminfo fallback location to something that contains a correct terminfo database, it works. The log changes to

WARNING: Terminfo database not found or outdated.
Using /mnt/c/msys64/mingw32/lib/terminfo as fallback.

And all tests pass.

infocmp $TERM worked in both cases! But it says Reconstructed via infocmp from file: /lib/terminfo/x/xterm-256color, which indicates it's an old terminfo database (single letter structure (x), not hex numbers). For the static binary, we use ncurses 6.4, which requires the new format. I don't currently know how to automatically find a terminfo database in the correct format. If we knew that, we could modify the script linked in my previous post to automatically fix this.

[tsjensen]

Some more googling gave me infocmp -D, but on my test Debian, all returned folders are wrong or in outdated format.

Maybe we just need to document that when compiling a static binary, one should first make sure that TERMINFO points to a terminfo database in hashed ("new") format.

[matteoguglielmi]

Is it possible to rewrite boxes in GO language to make 100% portable static binaries?

[tsjensen]

Jokes aside, did my answers make sense to you? Could you get your static build to work by setting TERMINFO correctly?

[matteoguglielmi]

Yes, it makes sense, but what should be done when the same static binary for a given architecture (amd64/arm64/riscv64) is executed on different OSes? Detect the OS and change the TERMINFO variable accordingly before using boxes? If so, that doesn't scale (for me). I need (it's my problem, obviously) three static binaries of boxes (one per architecture) capable of running on as many operating systems as possible (without presetting any OS-dependent variable).

[tsjensen]

I believe that the TERMINFO database is copied into the static binary, so if it's found at build time, you should be good.

[matteoguglielmi]

Why is make ; make test instead of make clean static; make test working (no failed tests) on Debian 12?

[tsjensen]

That's because what we've been discussing in this issue has very little to do with boxes itself. Instead, all this business of setting TERMINFO correctly is for building a proper ncurses static binary.

When you're just invoking make, you make use of the dynamic libraries supplied to you by the Debian folks, who did correctly configure everything. As part of make static, we must first create the ncurses static library.

[matteoguglielmi]

I see that when ncurses 6.4 is manually installed using the command in boxes's Makefile:

./configure --enable-static --enable-termcap --prefix=/tmp/ncurses-64 ; make ; make install

It creates the old format /tmp/ncurses-64/share/terminfo with single-letter subfolders.

Is there no way to tell ncurses to create the new format (hashed) terminfo folder and then use it to make the static binary of boxes?

Looking at the configuration options, I found --with-hashed-db:

./configure --enable-static --enable-termcap --with-hashed-db --prefix=/tmp/ncurses-64 ; make ; make install

which in turn creates this file:

/tmp/ncurses-64/share/terminfo.db

instead of /tmp/ncurses-64/share/terminfo with two-letter subfolders (this is what I was hoping for).

Is this of any meaning to you?

[tsjensen]

Sounds complicated. Just set TERMINFO correctly and the build will work as is.

[matteoguglielmi]

What's the correct value for TERMINFO on Debian 12?

All folders returned by infocmp -D:

/etc/terminfo
/lib/terminfo
/usr/share/terminfo

are either empty or have single-letter subfolders.

[tsjensen]

I used one I found in my local MinGW installation.

On pure Debian, it seems you actually need to compile your own tic using --with-hashed-db, and then use that to create the hashed database. But I don't know these things, I'm just passing on what ChatGPT says about this.
It says, to get the terminfo database source: curl -LO http://invisible-island.net/datafiles/current/terminfo.src.gz, unzip it: gunzip terminfo.src.gz,
then compile like you described, and use /tmp/ncurses-64/bin/tic -x -o terminfo-hashed terminfo.src.

[matteoguglielmi]

I just tried the /tmp/ncurses-64/bin/tic -x -o terminfo-hashed terminfo.src command, and the result is again the same terminfo.db file, which gets automatically generated with the --with-hashed-db ncurses's configure option.

To solve the static compilation problem of boxes, it might be worth understanding if terminfo.db holds the same terminfo data in the new hashed format.

[tsjensen]

Ok, just to get you going, here is a hashed terminfo database: terminfo-hashed.zip

Just unzip it somewhere and give that location as a "fallback location" as described above.

It's probably not a correct terminfo db for your system, but that doesn't matter because boxes only uses it to determine if the current terminal is color-capable.

[matteoguglielmi]

Could you provide me with a patch so that boxes doesn't look for the terminfo folder and always assumes a color-capable terminal? Or does boxes need to fetch something from the terminfo folder? Thx.

[tsjensen]

Yes, that should be possible. We to this on Windows anyway. It will take a while, of course.

Until then, you can just modify boxes.c to always return 1 from terminal_has_colors().

[matteoguglielmi]

It works, all test passed!

[tsjensen]

Good! Which solution did you use? Always assume color-capable terminal, or the terminfo-hashed.zip?

[matteoguglielmi]

Always assume a color-capable terminal.