ANSI Colors support

[14NGiestas]

Colors Module

In order to support colors and styling (bold, underline and etc) I propose a small utility to handle colored output, namely stdlib_colors. The use cases can vary, but the main application would be the richer logger and errors which helps a lot to visually identify what kind of message is right away.

Figure 1 - coloredlogs python library showing a use case

Few remarks from the community in #193

@mobius-eng commented on 20 May
(...)
Julia (and Python and R) are meant to be used interactively. Thus, colors play an important role with the output to the terminal. By contrast, Fortran programs are meant to be run in the "batch mode" with log going to the file.

@aradi commented on 24 May
As for colors: I am fine, as long as they are optional. I think most people typically run Fortran programs in batch systems with redirected output. Seeing a lot of color control sequences when opening the output files is rather disturbing...

@jvdp1 commented on 23 May
(...)
Colors are not needed for me.

@wclodius2
A problem with using colors is that output_unit can be directed to a file where the "color codes” will be distracting.

Proof of Concept and fixing the issue with redirection

• Well, gnu utils have support to colored output for eons and when it gets redirected it handles correctly, how is that possible? So I did some research in order to figure out how they are made and found this.
• So basically we test if there is a teletypewriter device (A interactive terminal) connected with the isatty function (from <unistd.h>), if isatty returns true then we write the ANSI colors, otherwise no colors in the output.
• A lot of compiler vendors support this function as a extension, however is not a standard so we can't rely on it. That being said I think it should be included in the stdlib_system module as a new function binding the C one (and alternatives to other platforms).
• In order to show that it works I made a small change in a program to output colors from the fortran-wiki and its here

Some other related projects that could inspire the API

• FACE by @szaghi
• Foul, as pointed by @arjenmarkus]

[certik]

I think something along the FACE and Foul libraries would be good for stdlib. I like using colors a lot in a terminal, it helps.

The ANSI escape sequences for colors work out of the box on Linux / macOS, but must be turned on to work on Windows.

So one needs some basic terminal support. I created this library in C++ to do colors on all platforms as well as keyboard input:

https://github.com/certik/terminal

This allows to create things like interactive prompts, get arrows working for cursor movements, and to draw anywhere on the screen. Similar to the ncurses Linux library, but multi-platform and only using the ANSI sequences that are now supported on all platforms (including Windows), which simplifies things.

[urbanjost]

I like the FACE interface. I have a similar library but it is not as completely or nicely developed. For something fancier than colors I generally use ncurses from Fortran http://www.urbanjost.altervista.org/LIBRARY/libscreen/ncurses/pdsrc/ncurses_from_Fortran.html but I think that is a bit beyond the stdlib but would be appropriate for fpm if it supported C wrappers as well as Fortran. Specifically for xterm(1) I have a library that lets you set fonts, window size and position, window title, etc. The esc(1) program in the GPF collection actually combines the two

when called without CLI options. So I use escape sequences quite extensively in one form or another but find using ncurses preferable for something past basic text formatting; so I personally support adding some basic ANSI color interface but think that getting a mostly-Fortran fpm(1) package manager up and getting the authors of packages like those mentioned to add their libraries to the registry more appropriate myself. There are ISO_C_BINDING interfaces for terminfo/termlib out there too which give you extensive terminal escape sequence support as well; although the ability to support different terminals is not nearly as important now that almost everyone uses a terminal emulator instead of a terminal, and those tend to emulate the ANSI terminals like VT102 or do not support escape sequences at all.

[certik]

@urbanjost yes, I agree that going beyond colors should go into a separate fpm package.

I would recommend in general to not depend on platform specific libraries, such as ncurses (which doesn't work on Windows in the native cmd.exe terminal), and rather always implement things in a platform independent (multiplatform) way. I have done that in C++ in the terminal library, and the same thing can be done in pure Fortran as an fpm package also.

I agree that the reason that can be done is that nowadays everybody supports some subset of the ANSI escape sequences, and so one can just use those.

[urbanjost]

I put M_escape
onto github as it uses a different API model than the others referenced so far, although the parsing method used is just a prototype it is functional for basic string coloring.

I found it interesting to see what language have color built-in and how many have a standard module or library for it in Rosetta Code.

[certik]

That seems similar to the approach here: https://python-prompt-toolkit.readthedocs.io/en/master/pages/printing_text.html#html It works. But it's slower, because you have to parse the string. So it depends on the application.

…

On Sat, Sep 12, 2020, at 8:15 PM, urbanjost wrote: I put M_escape <https://github.com/urbanjost/M_escape> image <https://user-images.githubusercontent.com/29845229/93008620-c2065e80-f544-11ea-84d1-0fbba159c769.png> onto github as it uses a different API model than the others referenced so far, although the parsing method used is just a prototype it is functional for basic string coloring. I found it interesting to see what language have color built-in and how many have a standard module or library for it in Rosetta Code. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFAWBXFUPNUIPL4BX5VCLSFQTMBANCNFSM4Q74M77Q>.

[urbanjost]

Speed is not always an issue in that you can process fixed strings once, as in
error=esc("ERROR")
and then just use the error variable; but usually adding color for human consumption at a terminal so I would not imagine using this with large amounts of data; but I agree.

[urbanjost]

The advantage of the approach of replacing in-band escape sequences
with formatting directives contained on each line is that it is easy
to turn off when running batch, but more importantly your program can
be run in "raw" mode and write a file with the directives in it that
can then be read back in by a simple filter program that strips it
back to plain text or displays it to a screen in color or converts it
to HTML or Adobe PDF. By making each line self-contained by default
this can still be done with any selected group of lines from the file.

I need to add some example filters (a loop that reads the "raw" files and displays them as plain text or as color on screen would liternally be a read/write loop that calls the esc(3f) routine, but an HTML and PDF example ala like the M_ncurses does for ncurses dumps (to HTML) or the asa2pdf(1) program does for "ASA carriage control" might make a stronger case); but have not seen a lot of interest in this so far(?) so it is on the "one of these days" lists for now. People can write HTML pretty easily from Fortran, after all. This actually has some advantages because it is simple and assumes fix spaced output and keeps each line independent by default. So that is how I justify the parsing overhead. It compliments traditional batch use but still lets you throw a little sparkle in your output.

[certik]

It's a question what to use for markup to show colors. HTML is certainly one way to do that, used in python-prompt-toolkit. It's a clean way to do that.

To be honest though, I still kind of like the simplicity of just outputting ANSI sequences, as you don't have to worry about any kind of HTML, you just output the sequences directly, here is an example from C++:

        std::string text = "Some text with "
            + color(fg::red) + color(bg::green) + "red on green"
            + color(bg::reset) + color(fg::reset) + " and some "
            + color(style::bold) + "bold text" + color(style::reset) + ".";
        std::cout << text << std::endl;

You can use less -r to see files with ANSI escape sequences. It works great. One can then convert it to HTML, I've used the ansi2html filter in Jinja in the past when converting a terminal output to html:

• https://nbconvert.readthedocs.io/en/4.2.0/architecture.html#templates-and-filters

One way to design color output is to first provide the color function together with the predefined fg, bg and style enumerations (somehow in Fortran) and then build the HTML layer on top. People who prefer HTML can use it, people who prefer a more direct style can use the color function directly.

[urbanjost]

I use the ansi2html(1) bash shell as well. I changed M_escape to have not only the pseudo-XML mode, but a function-based and direct ANSI escape sequence mode and added an fpm(1) config for it and simple filters called plain(1) and light(1) to read the pseudo-XML files and print the output to stdout with and without color as examples for discussion. I was going to do an OOP interface too, but I think the other resources listed here cover that. Comments welcome. I put a list of resources that compliments those mentioned here as well. If anyone tries it I would be interested in a vote as to which interface people prefer. The README files shows an example of each of the three modes.

[certik]

@urbanjost thanks! I like your example as the lowest level API:

   program direct
      use M_escape, only : &
     ! FOREGROUND COLORS
        & fg_red, fg_cyan, fg_magenta, fg_blue, fg_green, fg_yellow, fg_white, fg_ebony, fg_default, &
     ! BACKGROUND COLORS
        & bg_red, bg_cyan, bg_magenta, bg_blue, bg_green, bg_yellow, bg_white, bg_ebony, bg_default, &
     ! ATTRIBUTES
        & bold, italic, inverse, underline,  unbold, unitalic, uninverse, ununderline,  reset, &
     ! DISPLAY
        & clear
      implicit none
	write(*,'(*(g0))')fg_red,bg_green,bold,'Hello!',reset
   end program direct

I think that works great. With this, and your html API, I think that covers most use cases.

[14NGiestas]

I think that works great. With this, and your html API, I think that covers most use cases.

I think this kind of low level API is great, but we should keep in mind the complaint about the redirection issue too. So i think it should have a higher level API with a function that handles all the functionality:

program
    use stdlib_colors only: color & ! other suggestions: colorize, stylize...
                            fg_blue, sgr_bold
    implicit none
    write(*,*) color('Hello... ',      fg='red',  bg='yellow'), &
               color("it's me you're ",fg='blue', bg='yellow'), &
               color('LOOKING FOR?',   fg='red',  bg='yellow', style='bold')
    write(*,*) color('I can see it in your <b>eyes</b>') 
    write(*,*) color('I can see it in your <y>smile</y>') 
    write(*,*) 'Tell me how to win your ', color('<3', 'red')
end program

This would apply the escapes and the reset and handle the file redirection thing.
Using only the XML tags and the low level API would leave behind some kind of procedural colorization use case: imagine parsing a json and colorize it in some way, i think would be easier to just read the color name and style and pass to a function rather than build a XML string to escape or use a select case to apply the right low level escape.

[certik]

complaint about the redirection

Can you explain what you mean by this? Is it to change the output from ANSI sequences to HTML using the same code?

The API you proposed is very good also, and it can be built on top of the low level API we discussed above.

[14NGiestas]

I mean the redirection of such outputs to a file, since they would write the ansi escapes sequences too, polluting it (see "Few remarks from the community" section of this issue).
EDIT: And yes I think the higher API should be built upon the low level one.

[certik]

I see. I feel this must go into the API that is above the low level API.

I see his architecture all over many of the other issues in stdlib. In this case, the API layers as I can see them could be (from lowest to highest):

• lowest level: the goal is to output the correct ANSI string without having to deal with ansi sequences directly, but there should be no over head and it should be as simple as possible. The two proposed solutions would both work:

  • print *, fg_red, bg_green, bold, 'Hello!', reset
  • print *, color('Hello!', fg=fg_red, bg=bg_green, style=style_bold) (notice the fg_red is not a string "red" to avoid string parsing / comparison, and just efficiently output the ANSI sequence)

• higher level: a function color that can work in many ways, output both ANSI or HTML output and allows to turn off colors

• high level: some intermediate representation (XML) that can be parsed into, and then manipulated to any output (HTML, ANSI, latex, ...)

[ghost]

It would be nice to have some sort of pre-alpha version of this module in stdlib. I'd certainly use it.

I believe many programmers don't use colors, because colors are not cheap. He has to find a third party library, figure how to install and use. This process is not always easy.

Writing use stdlib_color and instantly get beautiful output on any platform is a good idea, imho.

[urbanjost]

I was hoping a few people would try it via fpm or even GPF and bounce around some ideas and firm it up before trying to go straight to stdlib. It is very easy to access it from fpm. That has not been happening so perhaps a more direct approach would be better. There are a number of stdlib projects (sort, split, LAPACK interface, ...) that I think are occupying a lot of resources and I did not want things to get even more fragmented and I was hoping some other things that would make this better (some system routines like ISATTY, Regular Expressions, ...) would emerge quickly that would make this simpler. Would you find one interface style preferable? Would other options like cursor positioning ("curses" type functions) be something you want too or should we just look at color? To use ANSI in-line color it is really useful to be able to use the ANSI query strings to detect color support, etc. and that is hard to do without a C interface to raw mode and so far stdlib has been avoiding such things, which is another limiting feature. Maybe just keeping it simple and doing basic text attributes really is the starting point and all that can come later. I have some old (really old) basic color conversion routines that might be useful for allowing HLS color models as a supplement for RGB values (I find HLS much more intuitive myself).

[ghost]

Hi @urbanjost . I think you are right. I've tried M_escape with fpm and it works beautifuly on Windows. Yes, I use only basic color formatting (don't need cursor positioning).
It seems that the only open question is the redirection issue pointed out above.

[urbanjost]

I added an fpm(1) package M_attr that concentrates on polishing up the HTML-like mode explored in M_escape, including an example of using the ISATTY extension from gfortran and ifort as an example of what was discussed above, as an alternative to the solution given, as it eliminates the need for a C interface in trade for using a non-standard but common feature. Made prettier examples; added support for a character array and a way to set a fixed right margin for non-default blocks with a background color set and a few other expansions on the original M_escape submode, including examples of pre-defined strings like "write(,)attr(' error message') for very simple basic but common use-cases.

[ivan-pi]

@zoziha, @arjenmarkus you can contribute to the issue on color support here (I've hidden your comments in #487 to keep the issue focused).

[ivan-pi]

For API inspiration, it might also be worth looking at what other languages do:

• AnsiColor.jl (Julia)
• Crayons.jl (Julia)
• termcolor-c (C)
• Termcolor (C++)
• rust-ansi-term (Rust)

[certik]

@ivan-pi thanks for the links. There is a longer list at the end of the README here: https://github.com/jupyter-xeus/cpp-terminal/

[ivan-pi]

In #580, @urbanjost wrote:

Since Fortran now supports (some more than others) everything from procedural to functional to OOP programming there are a lot of options and all of us have different approaches (some of mine and others discussed in M_esc and to a lesser extent M_attr) but if going with OOP and user defined types I might want a string that can have attributes (bg color, fg color, blink, underline, ...) that I can set that I would otherwise use like the STRING type in stdlib but that on output I could optionally print with all the attributes applied; albeit if putting out a lot of short strings with similar attributes it would take some work to not produce redundant output when all the strings have a common attribute.

I don't think this has been considered so far. so I thought I'd just expand on this idea. The first step would be to re-implement the stdlib string-type adding the necessary color and style fields:

    !> String type holding an arbitrary sequence of characters.
    type :: colored_string_type
        sequence
        private
        integer :: style
        character(len=3) :: color
        character(len=:), allocatable :: raw
    end type string_type

A new type constructor would accept foreground, background, and style arguments. Finally, the method

stdlib/src/stdlib_string_type.fypp

Line 1129 in 3af3259

subroutine write_formatted(string, unit, iotype, v_list, iostat, iomsg)

could be modified to check if the unit is a teletype writer, and if yes print the escape sequences accordingly.

If the interface of string type and the colored string type remained the same, you would in principle need to modify a single line:

use stdlib_string_type, str_t => string_type                               ! replace
use stdlib_colored_string_type, str_t => colored_string_type

A few issues with this approach:

• while it may be easy to go from a regular string to a colored string, going the other way round might be impractical.
• how to define the concatenation operator // for strings of different color? Do you give precedence to one of the operands, or is color reset to the default value?
• how to define equality/inequality? only compare strings or also their colors?
• the derived type I/O formatting strings are not very practical

[urbanjost]

Some interesting questions. I could picture not just color but screen position being an attribute, and so being able to build a panel to display on the screen as another aspect. An object-oriented ncurses library, so to speak. But that does complicate the simpler string operators, so maybe it needs it's own type. Not sure if it would still be useful to just treat it like the current stdlib_string_type for all existing syntax, or if that would be too non-intuitive. Appending a red string and a green string and ending up with an all red string seems non-intuitive, although as you mention operating with a regular string seems like it would be straight-forward. It would be nice to be able to not just color strings but to be able to easily create TLI interfaces with a Fortran-only (mostly) interface. CDC NOS and NOS/VE and VAX/VMS had those years ago and they were very useful at the time, but that was before X11 Windows and such. But having a string object with text attributes and position is appealing to me, as a screen in many ways is a great example of how OOP programming can be used to great advantage; I can picture an entire introduction to Fortran OOP programming using such a type. To truly be an extension and not change current behavior I think new operators like .oeq. would be needed and == would just compare the strings; not sure what // should do; returning a linked list solves some problems and introduces othersl

[urbanjost]

Cannot find anything where anyone took the approach, but I keep thinking it is not only potentially useful but also easily accessible for demonstrating aspects of modern Fortran, as everyone has a terminal and you can see the results "graphically" on a screen,
and even maybe do some coarrays to allow for event processing.

[arjenmarkus]

Currently there is a thread <https://community.intel.com/t5/Intel-Fortran-Compiler/How-to-use-colored-prompt-in-Fortran-application/m-p/1340123/emcs_t/S2h8ZW1haWx8Ym9hcmRfc3Vic2NyaXB0aW9ufEtXTEtZUU1FR00xSUtRfDEzNDAxMjN8U1VCU0NSSVBUSU9OU3xoSw#M158542> on the Intel Fortran forum about coloured output. I suggested they have a look at the foul library <https://sourceforge.net/projects/foul/>, but there are other possibilities as well. An object-oriented interface to ncurses does sound nice, though I have not done anything with ncurses for a few decades ;). Op di 30 nov. 2021 om 15:59 schreef urbanjost ***@***.***>:

…

Cannot find anything were anyone took the approach, but I keep thinking it is not only potentially useful but also easily accessible for demonstrating aspects of modern Fortran, as everyone has a terminal and you can see the results "graphically" on a screen, and even maybe do some coarrays to allow for event processing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR4UPPOMOWTDBKMWVN3UOTRGFANCNFSM4Q74M77Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.