Add terminal and color escape sequences

[awvwgk]

• implement low-level API for color support
• style, foreground and background color enumerators
• ~~true color (24-bit) types~~
• generation of escape strings via to_string function

Questions:

• how to test this?
• aim for compile time conversion?

Related

• https://github.com/fortran-lang/stdlib/issues/229
• https://github.com/awvwgk/fortty/blob/main/src/escape.f90
• https://github.com/urbanjost/M_attr/blob/main/src/M_attr.f90
• https://github.com/jupyter-xeus/cpp-terminal/blob/master/cpp-terminal/base.hpp

[14NGiestas]

Looking good. Testing if they produce the expected code escapes should be enough since this is a low-level API, users should be aware that no checks are done to ensure that the terminal will support colors.

[certik]

I am personally not a fan using derived types. The ideal for this would be to use enumerations (@fortranfan would probably approve:), in fact that's what we ended up using in C++ that you linked above.

We also have to initialize the terminal (especially on Windows).

I don't know if there is a way to simplify this somehow. If not, then the present approach is fine.

[certik]

One question: this module talks about "colors", but these are specifically terminal colors, and in fact only "ansi escape sequences", but it seems all terminals now converges to support those, including Windows. Should this be perhaps named "terminal colors"?

[certik]

And since we require to setup a terminal (on Windows) even to just print colors, we might as well use this machinery for other terminal things, such as the goal of the cpp-terminal library. Then we can write good terminal applications.

I think this is within scope of stdlib (although I always struggle if we should create a separate package or add it to stdlib).

[ivan-pi]

I am not a user of terminal colors (yet), however upon reviewing the module I was wondering if abbreviating the components as r, g, b would be just as clear but more concise. This might be just a personal preference, but I think it gives a nice clean look when initializing a color using the structure constructor:

type(fg_color24) :: my_color

my_color = fg_color24(r=34, g=12, b=10)   ! vs
my_color = fg_color24(red=34, green=12, blue=10)

[awvwgk]

Thanks for the feedback. Building something like cpp-terminal in stdlib for Fortran would be awesome, eventually we could just use stdlib_terminal for this purpose. The color escape codes would be the first step (maybe as stdlib_terminal_colors) to define our low-level API for handling escape sequences.

I usually start building this kind of projects separately and than notice that simple things like to_string for integer output are something I'm already missing. Usually, I'm just copying the snippet (at least for to_string), but I figured it would be better to start doing things properly and just contribute it directly to stdlib.

[ivan-pi]

I am personally not a fan using derived types. The ideal for this would be to use enumerations (@FortranFan would probably approve:), in fact that's what we ended up using in C++ that you linked above.

I don't think the fact Fortran doesn't support proper enums should be a show stopper. Here is a small quote from the Lua community:

Our motto in the design of Lua has always been "mechanisms instead of policies." By policy, we mean a methodical way of using existing mechanisms to build a new abstraction. Encapsulation in the C language provides a good example of a policy. The ISO C specification offers no mechanism for modules or interfaces. Nevertheless, C programmers leverage existing mechanisms (such as file inclusion and external declarations) to achieve those abstractions.

Derived types offer a nice and type-safe way to organize code, even if they might lead to reduced performance in some cases. I don't think this usage case is so performance-critical, that derived types would represent an issue. Moreover, when the derived types are default-constructed and used as parameters, an optimizing compiler can just copy the value and insert it in the right place of the client code (at least the NAG compiler is known to do this).

For ANSI Escape sequences specifically, the other two options I see available are 1) using a character string, like the M_attr module does, or 2) use an array of 3 integers. (In principle you could encode all 24 bits in a single integer, but I guess it makes code less clear.)

Would it be feasible to overload the // operator for the foreground and background color types?

[awvwgk]

Would it be feasible to overload the // operator for the foreground and background color types?

I actually thought about overloading operator(+) to create combinations from style, fg_color and bg_color. This will create a bit of combinatorics blowup if not designed properly, therefore I just went with the simple variant first to get the discussion going.

[ivan-pi]

Here's the example @certik posted in #229:

 std::string text = "Some text with "
            + color(fg::red) + color(bg::green) + "red on green"
            + color(bg::reset) + color(fg::reset) + " and some "
            + color(style::bold) + "bold text" + color(style::reset) + ".";
        std::cout << text << std::endl;

The same can be achieved in Fortran. I guess using operator(+) between the colors and styles makes more sense, however I'd still go for // to attach the escape sequence to a string. I'd also leave it unsymmetric, i.e. the escape sequence is always the left operand:

type(fg_color) :: green, red

print *, green // "Sentence in green"   ! valid
print *, "Red sentence" // red          ! invalid 

Edit: on second thought, overloading // hides the fact these are not two strings... Perhaps that's not desirable and a function or custom operator (or even +) would be better.

[certik]

I don't think the fact Fortran doesn't support proper enums should be a show stopper.

I agree. All I am advocating for is to consider and search for the simplest solution that involves the least amount of "layers" (such as derived types or even more OO approach). Btw, I am not convinced the C++ I approach I took is the best either.

I guess the only approach I can think of is to have functions like color_fg_bright_3bit and it would accept an integer and we define integer constants like color_red, but it's easy to pass the wrong integer in. It's less safe than the current approach with derived types I think.

That's why I CCed @fortranfan, because this use case is far from unique, I think this is actually quite common.

[urbanjost]

Since Fortran now supports (some more than others) everything from procedural to functional to OOP programming there are a lot of options and all of us have different approaches (some of mine and others discussed in M_esc and to a lesser extent M_attr) but if going with OOP and user defined types I might want a string that can have attributes (bg color, fg color, blink, underline, ...) that I can set that I would otherwise use like the STRING type in stdlib but that on output I could optionally print with all the attributes applied; albeit if putting out a lot of short strings with similar attributes it would take some work to not produce redundant output when all the strings have a common attribute.

I think some example programs solving some common use cases would help.

Looks good given the approach taken, but unless you want to expand this to allow building panels and./or ASCII graphics it seems a little overkill; but since it works and something is needed and it has the attribute of being expandable to include positioning and clearing I am good with it.

Having been using an ncurses interface and more recently M_attr I am having a hard time getting through the "not what I am used to" roadblock while trying it, but everything I have tried so far has worked.

I personally need something I can easily read and write from an external file and turn off so I need more of an abstraction of the attributes I can easily embed in text files; but I think a lot of people just want an easy way to color some text and this works for that so I am good with this.

[FortranFan]

@certik wrote Nov. 28, 2021 11:52 AM EDT:

I guess the only approach I can think of is to have functions like color_fg_bright_3bit and it would accept an integer and we define integer constants like color_red, but it's easy to pass the wrong integer in. It's less safe than the current approach with derived types I think.

That's why I CCed @FortranFan, because this use case is far from unique, I think this is actually quite common.

Thanks @certik.

I agree the use case here is hardly unique: scientific and technical computing is replete with the need to work with named constants with particular scopes in type-safe manner even in compute-intensive sections of codebases.

The need here to work with color codes for terminal escape sequences is but one instance that some might consider is outside of hard-code number crunching but it should not trivialized.

Nonetheless proper ENUMs are a bridge way too far for Fortran, the solution in the next revision Fortran 202X is far from optimal.

Thus moving ahead with this Fortran stdlib item in sync with all your consensus is as good as it can get.

[jvdp1]

@awvwgk is this ready for review?

[awvwgk]

I think we have agreed that terminal escape sequences are generally in-scope for stdlib. The question now is how make best use of them in an actual application and I don't really have a definite answer, this touches points like:

1. how to represent a color (string, enum, integer, derived type, ...)
2. applying color to strings (function, binary/unary operator, ...)
3. ways to enable/disable color support at runtime (atty detection, ...)

I haven't touched point 3 here in this PR, but I think this is one of the most important points to cover. However, I don't think there is a best answer.

The most practical solution I was able to come up with is here, using a derived type to hold all escape sequences and initialize them with empty string if color support is disabled, while deferring the decision of color usage to the user of the library. But I doubt this is the best strategy, since it requires explicit initialization.

[14NGiestas]

Point 3 is not a blocking issue for this PR to go IMO, it produces all the code escapes needed to color a text. We can just let the user of the module deal with it and provide helpers later on (E.g: a portable isatty function). Some other options that do not involve a higher-level abstraction:

• creating a private global variable (module-wise) and a function to turn on / off the colors.
• using an environment variable (I don't know if that is portable, however it worth doing it)

[jvdp1]

@awvwgk Thank you for the answers. I can't answer your questions. However, I would suggest to finish this PR as is, and ask users some feedbacks regarding these 3 questions. With fpm, people could test it quite easily IMO.

[milancurcic]

@awvwgk Do you think there is sufficient interest and feedback to wrap up and merge this PR? If yes, I'd be happy to help resolve the merge conflicts.

[awvwgk]

Didn't have time to look into this PR for a while now.

Working with color support for a bit I was somewhat unhappy with the API defined here for my projects, as I usually need a more high-level interface to turn off the color printout. The overall structure defined here might be just right to build such a low level interface.

I'm fine with finishing this PR and merging it as is, still there is some way to go.

[awvwgk]

I just revisited the API for color escape sequences for a different project and came up with a more robust scheme in https://github.com/tblite/tblite/pull/37. The implementation in question can be found at terminal.f90, if anyone is interested, feel free to use the linked file under MIT license for stdlib.

The linked PR shows beside the escape code implementation, an integration all the way up to the callback logger in the Python API which should allow to see colorful output when driving the Fortran library from Python in a cell of a notebook.

Colorful Python/Fortran interop

[awvwgk]

I removed the true color support because I didn't use it for my applications so far. Regarding the naming of the type, I'm open for suggestions, the currently chosen in this PR might not be the best choice.

[certik]

I reread my comments, I still agree with them. Essentially:

• I am not a big fan of overloading operators and derived types, it feels like an API that one has to learn, but others like it, so it's fine with me.
• Windows require initialization, but that is out of scope for colors
• It's hard for me to decide if we should just merge it? I feel it's important to move stdlib forward, even if we don't have perfect agreement, and then iterate on it, rather than stall the development trying to find the perfect solution, however I am also worried about stdlib having half-baked solutions that don't end up being used by users much. Related: should this be an fpm package first to get some real usage by people first?

The last point is the most important, so I opened up a meta issue #658 to discuss it more.

[awvwgk]

Fpm project is up at https://github.com/awvwgk/fortty

[milancurcic]

It seems like we have a few approvals and no objects, so let's give it a few more days and merge on Monday, May 23, if there are no objections.

Thanks Sebastian and all reviewers!

[certik]

As discussed in #658, it seems we all agree to go with the route of "merge much quicker to experimental, and reserve the right to completely change or even remove later". So I am fine to merge this as is and go from there. And for other PRs, let's do this in a matter of days or weeks, not months.

(And my objections above were not meant to derail the merging of this. In light of the above, it now seems to me it is much more important to merge quickly and iterate, rather than wait.)

[jvdp1]

As discussed in #658, it seems we all agree to go with the route of "merge much quicker to experimental, and reserve the right to completely change or even remove later".

I don't think that we all agree on that. However, as I mentioned it in this comment, I think this PR could be merged as is, such that people can try it through fpm.

[awvwgk]

The only thing a little off is that the derived type is called ansi_color but it represents not only a color but an entire ansi_style or ansi_escape.

Just rebased and renamed the derived type to ansi_code to better reflect that it can contain a style while keeping the name short (and avoiding British / American English variants).

[ivan-pi]

ansi_code is very reasonable given that "ANSI code" redirects to ANSI Escape Codes on Wikipedia.

[awvwgk]

I'm not 100% convinced by the naming of the module and constants. The derived type name ansi_code is good, but I wonder if stdlib_ansi or stdlib_ansi_codes would be better for the module name? For the colors, I feel like the "color" word is just extra characters. There is the actual color name in the variable name anyway, and it could be just stated in the documentation that parameters prepended with fg_ and bg_ are colors. A second consideration I had was to prepend ansi_ to all named parameters for extra clarity, but I'm not fully decided if I like it or not.

Naming is always hard, stdlib_ansi might work, I will rename those modules.

NO_COLOR could be easily supported in the future by changing the behavior of to_string_ansi_code.

This module is developed with support for easily disabling colors in mind, if you do not initialize your escape sequence with a color you get the NO_COLOR behavior, however you can still safely use the escape sequences as if there were color.

Perhaps I missed the discussion, but what was the reason for making the structure constructor private? Is it to force users to initialize it explicitly after the declaration section? This ties back to comment 2).

What should happen if the user initializes the escape sequence with bg=huge(1_i1). To avoid the headache of dealing with faulty user input the constructor is private.

Should string type be supported to? Can also be deferred to a new PR.

Can be supported as well, should only require two new procedures.

We can offer a functional API in addition to the operator one. This would reset the sequence by default. There could also be a second function with a logical flag for resetting.

Isn't code // str // style_reset already functional style?

[jvdp1]

@awvwgk If you are satisfied with this PR, I suggest that you merge it, such that users can test it, e.g., through fpm.

[awvwgk]

I'm already using the module in a project and will soon also adopt it in TOML Fortran, the design works nicely so far for my usage.

Now that we have the tree-shaking in fpm I could adopt it TOML Fortran via stdlib, but I'm still hesitant to limit myself in the compiler choice (I have to support GCC 5 and NAG 7 which are both not supported with stdlib but currenly work with TOML Fortran).