Invisible text with ncurses 6, not 5

[RJVB]

When I build htop against ncurses 6 I'm getting only a few headers that are visible:

while builds against ncurses 5 are functional normally:

Both running from the same terminal (under Linux) with TERM=xterm and using the colour profile for a "light terminal".

The snapshots were taken with the current git master/HEAD (3.2.2.166) but I've known about the issue since 2.0.2 .

Am I doing something wrong?

[RJVB]

FWIW, the text remains invisible when I switch the terminal to a dark colour profile.

[BenBE]

Can you provide a config.log for both the working and non-working versions? What is the output for ldd for both of these? Are proper terminal definitions installed for both versions of ncurses? What does selecting this invisible text look like: When you copy it, does it contain part of the screen output with proper text?

Given that some of the text is rendered, there seems to be something off with the terminal definitions. Especially since there have been several changes to the color rendering since 2.0.2.

NB: The HEAD branch has been renamed to main a while back.

[RJVB]

Can you provide a config.log for both the working and non-working versions? What is the output for ldd for both of these? Are proper terminal definitions installed for both versions of ncurses?

config-ncurses6.log config-ncurses5.log

The ldd output is appended to the 2 config.log files. You'll notice that my ncurses6 library is installed in a parallel prefix; self-built with --prefix=/opt/local and installed from the complete v6.4 sources so including the entire set of included terminfo files.

What does selecting this invisible text look like: When you copy it, does it contain part of the screen output with proper text?

Selecting seems to be disabled.

Given that some of the text is rendered, there seems to be something off with the terminal definitions. Especially since there have been several changes to the color rendering since 2.0.2.

Moving my /opt/local/share/terminfo/x/xterm out of the way causes htop to fail with error opening terminal: xterm, so I'm confident that the upstream terminfo file is used (when in place).

FWIW, I tested the exact same build approach on my Mac (same versions too); there htop works just fine with ncurses6.

NB: The HEAD branch has been renamed to main a while back.

HEAD is usually meant to refer to the latest commit, not the branch, and that's how I use it ;)

[fasterit]

Cannot reproduce.

Debian Bookworm

$ ldd htop linux-vdso.so.1 (0x00007fffc6d66000) libncursesw.so.6 => /lib/x86_64-linux-gnu/libncursesw.so.6 (0x00007f0bac267000) libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f0bac234000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0bac155000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0babf74000) /lib64/ld-linux-x86-64.so.2 (0x00007f0bac316000)

Works as intended.

[BenBE]

@RJVB Can you run both your htop versions under strace and attach those logs?

Also, when building other applications (like e.g. mc) in the same way, does this work?

For me that's BTW a CNR on Lubuntu 24.04 with htop from main too:

$ ldd htop
        linux-vdso.so.1 (0x00007ffe5ab8b000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fc2ef40b000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007fc2ef3ef000)
        libncursesw.so.6 => /lib/x86_64-linux-gnu/libncursesw.so.6 (0x00007fc2ef3b4000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc2ef380000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc2ef295000)
        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fc2ef270000)
        libnl-genl-3.so.200 => /lib/x86_64-linux-gnu/libnl-genl-3.so.200 (0x00007fc2ef268000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc2ef000000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc2ef4a1000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc2ef236000)

[RJVB]

Also, when building other applications (like e.g. mc) in the same way, does this work?

Forgot to mention that: cgdisk works just fine, but that doesn't use colours AFAICT.

Annoyingly I don't see a way to make htop monochromatic (= just use the current terminal colours). There's a monochromatic setting which isn't.

But assuming that that setting only uses black and darkred (i.e. 2( foreground "colours" and not multiple shades of those 2, then I think we're not really dealing with a colour related problem here. The ncurses6 build continues to show me only the text labels from the meter display plus the footer. They're supposedly black (0,0,0) but all the other text that's also supposed to be black isn't rendered. It's more like it isn't rendered at all.

BTW, I realise I couldn't select text because my builds have mouse support. I launched htop --no-mouse --no-color and this is what I could copy:

That's just a bunch of newlines.

I do see one big difference now that I can select text: the selection is dropped/cancelled at each refresh in the ncurses5 build, probably as you'd expect. It isn't in the ncurses6 build - maybe there is actually never any text output? The process does take 20-35% CPU so it is doing something.

[RJVB]

@RJVB Can you run both your htop versions under strace and attach those logs?

Good luck finding something in those 2 haystacks ;) (Should I have used some filtering options?) htop-nc6.strace.log htop-nc5.strace.log

[RJVB]

Also, when building other applications (like e.g. mc) in the same way, does this work?

I built midnight commander (had to look up what mc was!) and as far as I can tell it works fine.

[BenBE]

Okay, I had a first look at things and from what I saw, I noticed the following:

• Your ncurses6 version links libncurses.so.6, but should link libncursesw.so.6 (compare with my ldd output)
• Your ncurses6 version lacks linkage to libtinfo.so.6 (TermInfo)
• When compiling your ncurses6 version, linking with LTO fails
• Your configure for ncurses5 fails to find /opt/local/lb/libncurses.so, while the same test for the ncurses6 version fails with an LTO error -> This looks much like your build directory is not properly cleaned in between builds; either use git clean -fdx or make distclean.

 configure:7277: checking for waddwstr in -lncursesw6
 configure:7304: ccache /usr/bin/cc -o conftest -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -I/opt/local/include -Os -m64 -isystem/opt/local/include -Wl,-rpath=/opt/local/lib -L/
opt/local/lib -Wl,--enable-new-dtags -Wl,-rpath,/opt/local/lib conftest.c -lncursesw6
          -L/opt/local/lib -Wl,--enable-new-dtags -Wl,-rpath,/opt/local/lib -O3 -g -flto=thin -march=westmere -msse4.1 -msse4.2 -msse3 -mssse3 -msse2 -msse -mmmx -mpclmul -
Wl,-rpath,/opt/local/lib /opt/local/lib/libncurses.so
        -ldl -lm  >&5
-cc: error: /opt/local/lib/libncurses.so: No such file or directory
+cc1: error: unrecognized argument to '-flto=' option: 'thin'

(- = ncurses5 build, +=ncurses6 build, blank=common line)

• Although LIBS='-lcap -lunwind -lncursesw -ltinfo -ldl -lm -L/usr/lib/x86_64-linux-gnu -lnl-3 -L/usr/lib/x86_64-linux-gnu -lnl-genl-3 -lnl-3', ldd reports libncurses.so.6 => /opt/local/lib/libncurses.so.6 for ncurses6 to be used.

Please re-check your build to actually link libncursesw.so.6 like determined by ./configure.

[RJVB]

On Tuesday November 14 2023 08:41:47 BenBE wrote:

• Your ncurses6 version links libncurses.so.6, but should link libncursesw.so.6 (compare with my ldd output)

ncursesw is a symlink to ncurses on my system, not because of something I did either - or because I configure with --enable-widec --disable-lib-suffixes?

• Your ncurses6 version lacks linkage to libtinfo.so.6 (TermInfo)

Maybe because I don't configure ncurses with --with-termlib?

• When compiling your ncurses6 version, linking with LTO fails

What do you mean that? I didn't upload a build log so what step are you referring to exactly?

• Your configure for ncurses5 fails to find /opt/local/lb/libncurses.so, while the same test for the ncurses6 version fails with an LTO error -> This looks much like your build directory is not properly cleaned in between builds; either use git clean -fdx or make distclean.

It was, with make distclean IIRC. The build for ncurses5 fails to find /opt/local/lib/ncurses.so for the very reason I deactivate (temporarily uninstall) the ncurses-dev package so my ncurses6 install isn't visible to configure. The LTO error for the ncurses6 version is probably because I built ncurses6 with LTO but didn't use that option when doing the current test build of htop. I can show you a log where this will work (using clang and -flto=thin like I used for ncurses6) but that has no incidence on the issue. Rather, I thought I try a very standard htop build with gcc for this test.

• Although LIBS='-lcap -lunwind -lncursesw -ltinfo -ldl -lm -L/usr/lib/x86_64-linux-gnu -lnl-3 -L/usr/lib/x86_64-linux-gnu -lnl-genl-3 -lnl-3', ldd reports libncurses.so.6 => /opt/local/lib/libncurses.so.6 for ncurses6 to be used.

Please re-check your build to actually link libncursesw.so.6 like determined by ./configure.

Yes, as I said, libncursesw.so.6 -> libncurses.6.so .

Thanks for the time you spend looking into this despite not being able to reproduce it! In case it helps, here is how I configure ncurses 6:

> env CPP="/opt/local/bin/clang-cpp-mp-8.0" \
CC="/opt/local/bin/clang-mp-8.0" \
CXX="/opt/local/bin/clang++-mp-8.0" \
CFLAGS="-O3 -g -flto=thin -march=westmere -msse4.1 -msse4.2 -msse3 -mssse3 -msse2 -msse -mmmx -mpclmul" \
CXXFLAGS="-O3 -g -flto=thin -march=westmere -msse4.1 -msse4.2 -msse3 -mssse3 -msse2 -msse -mmmx -mpclmul" \
LDFLAGS="-Wl,--enable-new-dtags -Wl,-rpath,/opt/local/lib -O3 -g -flto=thin -march=westmere -msse4.1 -msse4.2 -msse3 -mssse3 -msse2 -mss \
configure  --prefix=/opt/local --enable-widec --disable-lib-suffixes --enable-overwrite --with-shared --with-cxx-shared --without-debug --without-ada --with-manpage-format=normal --without-pkg-config --enable-pc-files --with-pkg-config-libdir="/opt/local/lib/pkgconfig" --enable-rpath
#...
* Configuration summary for NCURSES 6.4 20221231:

       extended funcs: yes
       xterm terminfo: xterm-new

        bin directory: /opt/local/bin
        lib directory: /opt/local/lib
    include directory: /opt/local/include
        man directory: /opt/local/share/man
   terminfo directory: /opt/local/share/terminfo
 pkg-config directory: /opt/local/lib/pkgconfig

config-ncurses6.log

[RJVB]

Also: ncurses5 is a system library, not one I build myself, in case that wasn't clear.

[BenBE]

Thank you for that additional input. In all my remarks I was referring to the differences I saw from the configure logs (did a diff -U13 nc5.log nc6.log). Doing so I expected quite a few more differences, as the different paths where things were found should have been reflected in those logs.

As terminfo is used at some parts of htop, could you check if building your nc6 --with-termlib makes a difference? The setting for --disable-lib-suffixes shouldn't have an impact, as htop's ./configure looks for both versions (and ultimately finds a suitable one) and prefers the one with suffix if found (if the one without suffix provides Unicode support that's fine too; we check for that).

Does building htop without Unicode support (--disable-unicode) bring an improvement? Also, ./configure for htop tries to find libncurses using ncurses5-config first (later, a second try using AC_CHECK_LIB is performed; libncursesw is tried with ncursesw6-config). Including without library suffixes in your ncurses build may confuse ./configure (@natoscott: Can you take a look?).

Will later on take a look at the runtime strace logs. And no, without filters is fine (I expected the log to be massive); I need to filter the actual file accesses myself, because I need some context (to correlate htop source to the strace).

[RJVB]

On Tuesday November 14 2023 10:57:45 BenBE wrote:

As terminfo is used at some parts of htop, could you check if building your nc6 --with-termlib mackes a difference?

Will try to find some time to do that, but it's my understanding that this just causes libtinfo to be split out. I was going to check for a few symbols found in my system libtinfo5 to see if they existed in my own libncurses6 but libtinfo was installed without symbols and then I got side-tracked.

Does building htop without Unicode support (--disable-unicode) bring an improvement? Also, ./configure for htop tries to find libncurses using ncurses5-config first (later, a second try using AC_CHECK_LIB is performed; libncursesw is tried with ncursesw6-config). Including without library suffixes in your ncurses build may confuse ./configure @.***: Can you take a look?).

Will also try the unicode hypothesis, and see what happens if I move ncurses5-config out of the way. The preference for nc5 could explain why htop does what it does for me, but not e.g. mc (if that one doesn't have the same preference of course).

[RJVB]

I'm going to see if I can configure my ncurses6 to put its headerfiles into /opt/local/include/ncurses6 and if that fixes the confusion (aliasing?) during the build.

Sadly that alone is not enough: despite configuring nc6 with --includedir=/opt/local/include/ncurses6 the -I/opt/local/include/ncurses6 directive added through ncurses6-config and/or the pkgconfig file, htop's configure still picks up ncursesw/ncurses.h.

And that's the system one in /usr/include/ncurses because my nc6 headers are all directly under /opt/local/include/ncurses6.

[RJVB]

OK, I think I finally understand.

Putting the headers into $prefix/include/ncursesw is maybe common enough that htop or mc check the location but it is not the default, at least not for nc6. I didn't notice until just now, and it explains why I didn't get an nursesw subdirectory under /opt/local/include/ncurses6 above.

Configuring with --includedir=/opt/local/include/ncursesw creates almost the same header organisaton as in the system /usr/include (and a warning that I configured a non-standard header location). And with that layout htop's configure command does pick up the nc6 headers and I get a fully functional build.

I still would be nice if htop "would just use the correct headers". I double-checked: the nc5 config utility and .pc files do all correctly return -I/usr/include/ncursesw . IOW, I have a hunch that it shouldn't be necessary (anymore?) to find the nc headers through a trial-and-error process. It is quite common for configure scripts to provide ways to specify header or library locations that are known to be auto-detected unreliably so there's always that option.

[natoscott]

| It is quite common for configure scripts to provide ways to specify header or library locations that are known to be auto-detected unreliably [...]

@RJVB, @BenBE and I were chatting about this yesterday and this seems like a good approach to us, if someone wants to send through a PR? Thanks.

[RJVB]

@RJVB, @BenBE and I were chatting about this yesterday and this seems like a good approach to us, if someone wants to send through a PR? Thanks.

I have absolutely no experience coding configure scripts (properly, that is ;) ) so I'll give someone else a chance first :)