Support XTPUSHSGR / XTPOPSGR

[jazzdelightsme]

The XTPUSHSGR / XTPOPSGR control sequences (from xterm are really handy for composition of text that uses SGR sequences to colorize it.

The way it works is that XTPUSHSGR saves the current text attributes (fg color, bg color, underline, etc.) to an internal stack. Then later, an XTPOPSGR restores them--pops the attributes off the internal stack and sets the current text attributes back.

This capability allows you to use SGR sequences (which change text attributes like color) in a much more modular manner, because you are not required to have global awareness of the current text attributes. I.e. I can do something like:

printf( "The <xtpushsgr><fg blue><bg white>xterm.js<xtpopsgr> project is really cool." )

The phrase "xterm.js" will appear as blue on a white background... and the text after will be the same color as the text before. It's the semantic equivalent of an "end tag", and opens up all sorts of new possibilities for how text attributes are handled by programs. I'd love to see these sequences supported in xterm.js.

The xterm implementation allows supplying parameters to the sequence, to specify that only certain portions of the current text attributes should be saved (like "just save the underline state only"). But even only supporting a no-parameters form of the sequence would support the most common and useful (IMO) scenarios.

[Tyriar]

@jazzdelightsme any idea on what other popular terminals support this?

[egmontkob]

See also https://gitlab.gnome.org/GNOME/vte/issues/23.

[jazzdelightsme]

There is also an open PR for microsoft/terminal.

[jerch]

Hmm should not be hard to support those, still the concerns in the vte issue make them looking weakly specced. Is the attribs stack terminal global or bound to a buffer? Will have to look at xterm source...

Further note that, as handy as this might seem - it looses ground as soon as multiple producers use this interleaved, they will screw up the stack.

Edit: Why is the latter a concern? Isnt terminal state always global? Yes, but here the stack behavior might suggest to app devs to get a reliable way to refer to some older attrib values, which is not true for the reliability part. This is misleading at best, and produces wrong formatting at worst.

I think these sequences are interesting for tools that repeat several attribs often - like loggers or debuggers with certain output formatting. But those often operate as "collectors" - they might grab output from other commands to put it into a joint output. Now they cannot assume anymore to still get their wanted attribs with XTPOPSGR - a sub command might have "screwed up" the stack - and they again have to style explicitly (and to do that to memorize those attribs themselves). So nothing was won in the end.

I somewhat question the usefulness of those sequences. 😸

Edit2 (slightly offtopic): Recently I have the impression, that ppl want to see more and more stuff in terminals which allows a tighter coupling of app side and terminal side (this partly reminds me of the OSC 7 discussion in terminal-wg). This is not a bad thing per se, but simply not possible from within the current terminal interface - it cannot be achieved in clean way with some new sequences, it would need changes on much more basic interfaces (process and pty interface, which is not an easy task at all). I think we should restrain from adding stuff that looks promising in a certain way but will fail badly as soon as a more general use case gets applied. A sequence, that has several "works only if ... and if ..., never when ..." restrictions is a bad one.

[jazzdelightsme]

I have found these sequences in practice to be more than what I would describe as merely useful; they enable a completely new model for text composition.

It is certainly true that interleaved output would produce nonsense (assuming the two components are not deliberately working together; in other words the definition of "separate components" here captures this fact).

To be clear, I believe this is the sequence of events we are referring to:

1. component A: write some output, including a push, and a color change
2. component B: write some output, including a push, and a color change
3. component A: writes some output, including a pop
4. component B: writes some output, including a pop

That this produces nonsense is expected. And, importantly, no different than the state of the world before the push/pop sequences were available--the following sequence is also not likely to produce desired results:

1. component A: write some output, including a push, and a color change
2. component B: write some output, including a push, and a color change
3. component A: writes some output, including a pop
4. component B: writes some output, including a pop

The advantage of the push/pop is that now we have the ability to write code that does this:

1. component A: write some output, including a push, and a color change, more content, and a pop
2. component B: write some output, including a push, and a color change, more content, and a pop
3. component A: write some output, including a push, and a color change, more content, and a pop
4. component B: write some output, including a push, and a color change, more content, and a pop

In each case, each component does not need to know what the "current" text attributes are--they can independently style some (or all) of their output.

So what we derive from this observation is that for these sequences to be useful, an independent component should use a push and a pop (or multiple) together in a single chunk. This is not burdensome... it's the whole point!

the concerns in the vte issue make them looking weakly specced

I felt like the specific questions raised were answered (for instance in Martin's comment). My read of the situation is that there was just disagreement from VTE maintainers about how aspects of it should have been done (especially selective saves) (correct me if I'm wrong). There's not much I can do about that, except to preach about how useful the capabilities bestowed by these sequences are. But even if other terminal implementations decided to pass on supporting selective saves, the full-save feature alone would be very useful.

[jerch]

@jazzdelightsme Yes, if the components restore the stack to the their entry level, thus all pushs done by the component are popped before returning, it should be fine. Its very much like not forgetting closing tags in wellformed XML.
I still see a burden here, app/component devs now have to track that outer state as well, and have to deal with it gracefully to keep the stack aligned. This means they cannot simply exit early anymore (in case of errors), now they have to to cleanup the terminal state as well in an exception or exit handler. In ASM we have a stack pointer to cleanup pretty fast, here we have to track dangling pushes, urgh. This is not failsafe, thus an outer component mostly likely will have to fall back to a more "traditional styling" with a prehand cleanup SGR (see next section).

Not so with the "traditional styling" - here a nested component would just style stuff prehand (e.g. SGR xy + some output; SGR z + other output) and just leave when done. The outer component can deal with it likewise up to the root component, which would have a final SGR 0 as a good citizen before exiting to the shell.

Plz dont get me wrong - I see a benefit in your examples above, mainly for pushs/pops done in close successions, where it can save the need to hold some of the previous intermediate attribs. Still not sure if these savings are worth the headaches introduced by spreaded terminal state manipulations, further illustration:

const SGR_0 = 'SGR 0';
const SGR_ERROR = 'SGR red|bold';
const SGR_WARNING = 'SGR yellow';
const SGR_INFO = 'SGR green';

// traditional way
output(SGR_ERROR, 'something bad happened...');
output(SGR_0);
some_other_call();  // code controlled by project
output(SGR_WARNING, 'just a warning...');
output(SGR_0);
foreign_call();  // output out of control
output(SGR_INFO, 'we are done.');
output(SGR_0);

// with push/pop SGR
PUSH()  // push SGR_0
output(SGR_ERROR, 'something bad happened...');
POP();  // restore SGR_0
some_other_call();  // code controlled by project
output(SGR_0);  // needed if we did not ensure the call returns with SGR_0 on top
PUSH()  // push SGR_0
output(SGR_WARNING, 'just a warning...');
POP();  // restore SGR_0
foreign_call();  // output out of control
output(SGR_0);  // needed since we cannot control the stack in foreign_call
PUSH()  // push SGR_0
output(SGR_INFO, 'we are done.');
POP();  // restore SGR_0

I think this shows the problem - as long as the stack usage happens within comprehensible code (close succession), its fine. But as soon as a complicated call enters the stage, the stack should be re-initialized as no one (beside compilers) can track reliably a mutable global state over complicated control flow. This example feels abit like facing problems again from good old goto days. Not sure if incorporating a wonky outer global state helps in terms of testability, imho the functional programming paradigms are on the better track here.

To sum this up - the benefit I see only holds for very close PUSH+output+POP actions, otherwise it should be discouraged.

[egmontkob]

1. component A: write some output, including a push, and a color change, more content, and a pop
2. component B: write some output, including a push, and a color change, more content, and a pop
3. component A: write some output, including a push, and a color change, more content, and a pop
4. component B: write some output, including a push, and a color change, more content, and a pop

This can just as well be written as

1. component A: write some output, including color changes, and finally an sgr0
2. component B: write some output, including color changes, and finally an sgr0
3. component A: write some output, including color changes, and finally an sgr0
4. component B: write some output, including color changes, and finally an sgr0

can't it? Maybe it takes a more complex setup to showcase your point.

Apart from the great observations from @jerch, and the ones pointed out in the VTE bug, I also cannot see where and how this feature wants to go. I don't see it reasonable for this feature to get implemented in the vast majority of terminals in the next 10 years.

An independent story: look at truecolor delimiter, even though the standard says to use colons which would be technically superior (more compatible with future extensions), using semicolons instead have become so popular that terminals have about zero incentive to support colons, too. It's easy to say that all of them should support colons, too, the world would become a better place. But it's just not happening.

Another independent story: a popular terminal emulator (I don't want to blame them so I'm not providing a link, you can sure easily find it if you want to) didn't support ST (string terminator for OSC). Took them 7 years to implement it (counting from the day it was reported). It was buggy. They claimed to fix that bug quickly afterwards. It was still buggy. Took another 1 year to discover that it was still buggy, and yet another 1 year to really fix it. That's 9.5 years from the initial bugreport to a working solution, for recognizing a freaking two-byte closing sequence. Based on this: Would they be interested in implementing a new escape sequence with close to zero use? Maybe, maybe not, I don't know.

We already have a pretty good solution for changing colors and attributes. Push+pop is not part of that. Why would any software author write a piece of software that runs only in a small fragment of terminals, rather than one that (perhaps with a tiny little bit more complicated code) runs on all of them? No one who aims to share their work with others would do so. Which, in turn, means that terminals don't have much incentive to implement support either because there aren't apps that would need it. It's a chicken and egg problem, and is very hard to break out of the current situation because it's developer effort not well spent, you'd need dozens of parties to agree and work in the same direction for some tiny benefit.

In the terminal ecosystem, every application currently assumes that the terminal is at the default colors+attributes when it's started up, and accordingly it's a fair game to return to the default (sgr0). E.g. you switch to yellow background from your shell prompt, executing a utility that colors its output (ls, gcc, grep, you name it...) won't revert to yellow but to the default color right after the first colored piece of text. It might be a great goal to fix it, but that requires changing all these utilities. Which, due to lack of generic support in terminals, means a temporary (for a decade or two) solution of sgr0+pop to restore, so that it pops correctly in terminals that support it, and falls back to the old sgr0 in terminals that don't. And then maybe in 20 years from now these utilities will be able to go with popping only.

The other use case is when your application can theoretically know the previous color, because it was this very application setting it, it's just cumbersome for your code to carry it and you'd rather push+pop. Actually, you could also relatively easily implement this stack itself in your application, or carrying and knowing the state, whichever you prefer. The difference: while for XT{PUSH,POP}SGR my estimate for becoming widely usable is 10–20 years if you push for it hard in all the terminals that matter, the other can be implemented in your app in one rainy evening.

In the VTE bug, for a side issue (the exact choice of escape sequence) you showed a semantically incorrect piece of code, and you convinced xterm's author to modify the escape sequence to make that incorrect code work. Over here you showed a theoretical example, a piece of code that could just as easily go with sgr0. I'm yet to see an actual example where push+pop would offer a notable advantage, let alone one that is worth the price of pushing for this feature across all terminals for many-many years, rather than just going a less convenient solution in your app.

That being said, I don't have concerns with push+pop per se, and by no means do I want to discourage other terminals from implementing it. I have concerns with the exact details how this escape sequence looks like, and I believe that the entire effort of getting this feature universally available is doomed to fail. Would be glad to see being proven wrong over the next few years.

[jerch]

Stressing the testability aspect one more time - as @egmontkob already noted, you can always build such a stack on your own and carry it around. Now you have full control over the stack, it is much easier to test. No need for uncertain outer state here. I am pretty sure haskell devs would avoid these sequences for very much the same reason - no need to write an ugly monad if you can actually control the full state reliably.

Going one step back from the sequences at hand - it seems we have a need for addressing certain attribs sets during runtime. I am not a big fan of putting to much app related stuff into the terminal state, still I wonder if "SGR/attrib slots" on terminal side would be a better way to deal with this:

• they are reentrant - an interrupt handler cannot screw up values anymore (the stack variant will go bonkers here)
• can be easily set up, e.g. by loading current attribs into a slot, maybe even make an explicit store sequence
• can be requested independently by slot numbers
• still needs discipline on app dev side, but its much easier to comprehend as an app global or subcomponent-local contract

Just a wild idea...

[jazzdelightsme]

First, you are absolutely right that there is a "chicken and egg" problem: who will use these sequences if they are not widely supported, and who will support them, if few people use them? I agree with your assessment that it will take a long time for these to be adopted. That said, I do still hope to be writing code in 10-20 years. :D

Why sprinkling SGR0 (reset to default colors) is not enough

Resetting current colors to default works in many cases. This would be equivalent to a color stack that never has more than one item on it. But if your stack ever grows beyond a single item, SGR0 no longer does the correct thing.

For example, one place I have used this in my own code is when dynamically "highlighting" a particular line of output. Some layer of my program generates objects which, when printed, contain color sequences to help make information more clear. A higher layer that is consuming and writing out these objects may decide, for one reason or another, to "highlight" a particular object / line of output, which it does by effectively doing printf( "<push><bgDarkRed; fgBlack> %s <pop>", objWhatever.ToString() ). Because the color sequences used by objWhatever.ToString() use push/pop when changing the color, instead of SGR0, this all works dandy. If it used SGR0 instead, it would not work (the "highlight" would be "forgotten" after the first bit of color in the %s insert).

Alternate ways to handle this: certainly, my app could maintain its own internal stack of SGR state. Functionally, it would work exactly the same. However, there would be a couple of MAJOR drawbacks to doing it that way:

Problem 1: solving this myself is likely NOT simple

For a "simple" implementation, all output would have to be generated "in order": if we can ensure that objWhatever.ToString() is always called (and it dynamically formats the object), then all code that deals with pushing/popping can just reach out into some global g_colorStack to determine what the "current" color should be, and all is well. As you say, this is a "one rainy evening" problem. In fact, I spent one such rainy evening implementing this for VTE. (Side note: actually, now that I remember it, it was two rainy evenings: the first 1.5 were just getting a proper build environment set up. Have y'all ever considered a docker build container solution?)

However, I have no code which is so simple. Many objects may not actually dynamically generate a new string when .ToString() is called; they may want to be able to cache a previous representation. Furthermore (even without cached strings), there are many other reasons why output may be dynamically composited, such that stuff is outputted in a different order than it was generated in, which simply will not work with a "simple" global stack implementation.

The only solution to that, then, is to be able to embed conceptual push/pop sequences into your output, and then dynamically interpret them on their way out the door to the terminal. This is a considerably more complicated (and annoying) piece of code to write. This is semantically equivalent to writing your own VT parser (though you only have to handle "your own" sequences, and let the others flow out to the terminal). In addition to being complicated, it's simply wasteful.

Problem 2: solving it myself is not reusable

The second problem is that no matter how you "do it yourself" (whether the "simple" implementation, or the more complicated, "streaming" implementation), your code is no longer reusable. Any code that wants in on this scheme has to be baked together, either knowing to use the same g_colorStack or the same embedded codes and intermediate output streaming layer. Even among code that I completely write myself, I would (and do) find this to be an almost insurmountable burden, and it simply means that such bodies of code are just not shareable. No shared lib can do this stuff--no shared lib can take colorized output from any other shared lib, and add "highlight".

I'll go ahead and reiterate that last statement, because I allege it also applies to the current state of the world:

Without a standard solution, no shared lib can do this stuff--no shared lib can take colorized output from any other shared lib, and add "highlight".

In my opinion, the way to solve this, and thereby enable entirely new scenarios for textual output composition, is to bake that "standard solution" into the terminal itself: the code becomes "one rainy evening" simple, and it's usable by everything (using that terminal). We've got the ball rolling in xterm, and I know everybody (e.g. you, @egmontkob :D) is not happy with every aspect of that, but I intend to continue advocating for it elsewhere.

[egmontkob]

That said, I do still hope to be writing code in 10-20 years. :D

First, as they say: hope is not a strategy. I wrote down why I believe the attempt to address this issue is likely to fail. Maybe you'll prove me wrong.

Second: I bet you a beer that in 15 years from now you'll be working on something completely different and no longer care about this issue. Deal? :)

Without a standard solution, no shared lib can do this stuff--no shared lib can take colorized output from any other shared lib, and add "highlight".

This is correct – assuming you're not willing to enter the territory of parsing the SGR sequences. Again, you have two choices: spend a few days implementing a library that parses SGR and you have every chance to release it this year; or try to convince dozens of developers who would all need to agree and all do some work so that it's maybe widely usable in 10 years from now. If I were you, I would choose the former. Or maybe you can do these two in parallel, and in 10 years from now you can clean up your code and remove the ugly SGR-parsing hack.

I intend to continue advocating for it elsewhere.

I sincerely wish you good luck! (Which does not imply that I can help you by quickly adding this feature to VTE.)

[jazzdelightsme]

Or maybe you can do these two in parallel, and in 10 years from now you can clean up your code and remove the ugly SGR-parsing hack.

Yes, this is exactly what is happening. In fact I think it's been close to ten years since I wrote my implementation. Which is why I am very confident that a) this feature is very useful, and b) the "do it yourself" approach does not scale, because it only works if all code running in a process agrees to send output through the hack, which is a pretty egregious requirement.

[jerch]

@jazzdelightsme This pretty much sounds like you want to daisy chain the SGR modifications with different stack levels across components or even separate processes. There is one big flaw in this idea - at no point in this chain the terminal is isolated, it is always a global thing any process with enough rights can hook in and screw up your stack and idea of SGRs. Its not even needed to be done by outer processes, even an interrupt handler triggered by one component could do this (that you might have forgotten in your big color control flow diagram).

What you describe is a big stack machine for SGR handling across several processes/components, while the held state is not secured or self-contained by any means, nope its OS global. This is doomed to fail. At least in POSIX terminals cannot be bound/restricted to certain processes. It works the way around, the controlling terminal is a notion of a process (group).

[jazzdelightsme]

Perhaps I don't understand the concern. Yes, multiple processes can have access to the terminal... Any uncoordinated access from multiple processes is bound to result in garbage on the screen (and a poor user experience), no matter whether the output contains anything SGR-related or not. This is not a problem unique to an SGR stack...