DF description/missing color is unreadable on dark themes

[fbruetting]

trafficstars

In the picture below you see a DataFrame in combination with following themes:

• Solarized Dark
• Solarized Dark (with color “alternate black” manually changed to grey)
• Solarized Light

Solarized Dark is one of the most popular dark themes, because of its very good readability as its contrast being neither too low nor too high. Without modifications, missings and column types are not readable at all.

I found out that this is due to DataFrames using the “alternate black” color – which on dark themes usually is equal to the background color, or at least very near. So it’s not a problem at all on light themes, but all dark themes have problems with this (not just Solarized). See the following examples with the themes:

• Adwaita Dark
• Builder Dark (GNOME Builder)
• Classic Dark
• Cobalt
• Kate Dark
• Peninsula Dark
• Solarized Dark
• Oblivion
• Arctic Dark

I haven’t encountered these issues anywhere else, neither in my IDE (GNOME Builder), nor in general Julia, so I guess the problem’s root is this package’s color selection. I guess it’s necessary to chose another color here, as “black” just works well one bright themes.

[bkamins]

Thank you for the comment. Do you have a recommendation then?

CC @ronisbr

[fbruetting]

Thank for the quick reply!

I think one of the colours should be chosen, as neither background/foreground nor black/dark grey/light gray/white seem to work well for both light an dark themes.

This leaves red, green, orange, blue, magenta and cyan as options. The alternate colours originally were intended as brighter tones (light red, light green, yellow, light blue, light purple and teal), but some themes – for whatever reason – abuse them for completely different colours. Solarized Dark for example uses these places in the colour array to store some dark tones. I don’t know why this is the case, but it seems these cannot be used reliably in every case. So if you chose for example yellow, then in some themes that text gets flashy, while in Solarized Dark it gets very dark.

Julia seems to mostly use cyan as accent colour, so maybe this is a good choice? Otherwise orange and blue also seem to be a good fit. I’d reserve red, green and magenta for signaling purposes.

This webpage might be helpful.

Here are some theme examples:

[ronisbr]

Hi!

Fixing this issue would be extremely simple since we just need to change the color here:

https://github.com/JuliaData/DataFrames.jl/blob/8c32d537b1d5a2ae9b1cdd72928def18aeae0bf9/src/abstractdataframe/prettytables.jl#L28C48-L28C79

Personally, I have no idea what would be the best option.

[fbruetting]

The best option, I guess, would be to select light gray for dark themes and dark gray for light themes. But that seems complicated. 😶️

[bkamins]

In summary: we are OK to change this given we know what color to pick. The restriction is that:

1. It has to be one color (as we cannot detect the theme the user uses).
2. The intention is to dim missing a bit (it should not cry-out, rather it should be visually a bit less standing out, as this is the point of having a different color for it)

[ronisbr]

Hum, there is no universal way to check if the theme is dark or light AFAIK :(

[ronisbr]

I agree 100% with @bkamins 's proposal here.

[fbruetting]

In summary: we are OK to change this given we know what color to pick. The restriction is that:

1. It has to be one color (as we cannot detect the theme the user uses).

2. The intention is to dim `missing` a bit (it should not cry-out, rather it should be visually a bit less standing out, as this is the point of having a different color for it)

I searched for a bit, but it doesn’t seem possible to tick both boxes. Maybe then hardcoding a medium gray, which is suitable for both theme variants?

[ronisbr]

I searched for a bit, but it doesn’t seem possible to tick both boxes. Maybe then hardcoding a medium gray, which is suitable for both theme variants?

Do you mean, setting a RGB color?

[bkamins]

I think it would make sense to hardcode RGB medium gray (I guess it is possible - right?)

[fbruetting]

In the shell it sure is, I just don’t know how Julia interfaces with that. Shells use strange color codes like \033[31m, as you can see in my link above.

[ronisbr]

The problem is for terminals that do not support 24-bit colors. I need to test to check what will happen.

[ronisbr]

In the shell it sure is, I just don’t know how Julia interfaces with that. Shells use strange color codes like \033[31m, as you can see in my link above.

This code says the terminal must change the foreground color to red. In PrettyTables, we use Crayons.jl that is a conversor between a color and those escape sequences.

Maybe we can assume everyone will be using at least a terminal that supports 256 colors. In this case, we have access to a wider range of colors. IMHO, assume 24-bit color is too much restrictive.

In the case of 256 colors, there are those options:

(from https://www.lihaoyi.com/post/BuildyourownCommandLinewithANSIescapecodes.html)

Thus, we can select a color between 232 and 255.

[fbruetting]

You can also substitute :dark_gray by (127,127,127) in your referenced code. I just tested this.

[ronisbr]

But AFAIK it is converted to a 24-bit color code, which should not work in terminals that do not support it.

[fbruetting]

While we’re at this: Is it possible to specify this color as an environment variable or by a function call? I guess that would be nice for circumventing corner cases due to hardcoded values, for example in the rare case anyone has a grey background.

[ronisbr]

Yes, it should be fine to check if a ENV variable is defined and use this value instead. Maybe it will add an overhead, but should not be noticeable. What do you think @bkamins ? Any suggestions about the variable name?

[bkamins]

In the past we have had a policy not to introduce behavior depending on global state.

@nalimilan - in this case do you think it would be acceptable?

[giordano]

Isn't this the same problem as https://github.com/JuliaLang/julia/issues/38730 caused by the fact the solarized theme uses same colours for background and foreground?

[fbruetting]

Isn't this the same problem as JuliaLang/julia#38730 caused by the fact the solarized theme uses same colours for background and foreground?

Yup, seems so. Thanks for the tip, now I see that my stack traces contain more information than what is visible. 😅️

[nalimilan]

In the past we have had a policy not to introduce behavior depending on global state.

@nalimilan - in this case do you think it would be acceptable?

I would find it kind of weird to have a mechanism to set the color used when printing a particular object (missing). Isn't there a more general theming system that we could use?

Anyway, regarding this particular issue, requiring users to set an environment variable seems suboptimal. Most of them won't even know they can do that. Is there a way to check whether light_grey gives the same color as the background, and switch to dark_grey in that case?

[ronisbr]

Unfortunately no. There is no universal way to check what is the color the terminal will use given a ANSI escape sequence.

[fbruetting]

In the past we have had a policy not to introduce behavior depending on global state. @nalimilan - in this case do you think it would be acceptable?

I would find it kind of weird to have a mechanism to set the color used when printing a particular object (missing). Isn't there a more general theming system that we could use?

Anyway, regarding this particular issue, requiring users to set an environment variable seems suboptimal. Most of them won't even know they can do that. Is there a way to check whether light_grey gives the same color as the background, and switch to dark_grey in that case?

My proposal was to use medium gray by default and only if that collides with themes (which then are neither light nor dark but rather something in the middle, which I assue to be pretty rare) users are required to manually chose a suitable color for de-highlighting. The problem is, that terminals just define highlighting-colors but no de-highlighting-colors.

[ronisbr]

My proposal was to use medium gray by default and only if that collides with themes (which then are neither light nor dark but rather something in the middle, which I assue to be pretty rare) users are required to manually chose a suitable color for de-highlighting. The problem is, that terminals just define highlighting-colors but no de-highlighting-colors.

Notice that this is only possible if we assume a terminal capable of outputting 256 colors since the default 8 colors only contains bright black as gray. IMHO, I think the number of people using terminals that cannot output 256 colors should be minimal. Thus, it should be fine to select one of those colors:

Assuming 24-bit and selecting the RGB values is way more restrictive.

@fbruetting can you please test in your terminal what would be a good gray selection for those themes? To do so, open a terminal with your theme paste: echo "\e[38;5;239mmissing", changing 239 to the color you are testing.

[fbruetting]

Of course, here it is for Solarized Dark and Light: (btw: echo -e was necessary)

I think 243 provides the best readability for both variants.

[ronisbr]

In the "default" themes, we will have (before on top, after on bottom):

IMHO, the highlight will lose its purpose because it almost indistinguishable from the normal text (which usually is light gray instead of pure white).

[bkamins]

Side note: the same issue applies to the column type subheader.

[fbruetting]

Not having a good contrast is much less of a problem than not seeing those values at all, so I’d still prefer hardcoding medium gray.

Here’s a comparison of common themes for the colors 240-247 (all of these are shipped with Tilix):

Solarized Light / Linux / Tango

Solarized Dark / Material / Monokai Dark / Orchis / Yaru / Base16 Twilight dark

[bkamins]

By the way - can you please confirm how the following renders under the color schemes we discuss? Thank you! I am asking, because maybe we can use the gray that uses standard Julia in this printout.

[fbruetting]

That would lead to invisible text in Solarized Dark, like in the Julia-issue referenced above.

Same order as above: