Term encoder still has problems with nested tokens

[kyrylo]

Hey. I'm developing a Ruby terminal application on top of CodeRay and I'm having troubles with understanding of string, regexp and shell attributes.

What is regexp function? Why does self sub-attribute set a color value for \d, \w, \b and so on?

regexp:
  self
  content
  delimiter
  modifier
  function

Why does self sub-attribute set a color value for escape sequences like \n, \t and so on?

shell:
  self
  content
  delimiter

Why does self sub-attribute set a color value for escape sequences like \n, \t and so on? What is a string modifier? Why does escape sub-attribute not set a color for escape sequences?

string:
  self
  content
  modifier
  escape 
  delimiter

[korny]

a Ruby terminal application on top of CodeRay

Awesome :-)

From your examples, I guess that you looked at the Terminal encoder. It is not a good reference of how token kinds are used. Instead, I suggest you look at styles/alpha.rb which should contain all important token kind nestings.

regexp.function is obsolete and not used anywere, to my knowledge.

(in :string) Why does escape sub-attribute not set a color for escape sequences?

That's because escape sequences like \t are marked as :char, not :escape. :escape is used by the Ruby scanner to mark the # character in inline variables like "#$0".

The Terminal encode should highlight :char inside of strings. I change this, thank you for spotting it!

What is a string :modifier?

PHP, Python, C, and even SQL use modifiers for strings to mark them as binary, Unicode, wchar_t or something else.

Why does self sub-attribute set a color value for …

Can you explain more? What output do you get, and which one did you expect?

[kyrylo]

From your examples, I guess that you looked at the Terminal encoder. It is not a good reference of how token kinds are used. Instead, I suggest you look at styles/alpha.rb which should contain all important token kind nestings.

I did look at the Terminal encoder. Although styles/alpha.rb contains not only Ruby tokens, thanks for the tip.

That's because escape sequences like \t are marked as :char, not :escape. :escape is used by the Ruby scanner to mark the # character in inline variables like "#$0".

:escape colors an octothorpe sign only in the following situations: "#@ivar", "#$gvar". But it doesn't color the sign in interpolations: "age: #{age}". For that case we use :self token. Am I understanding you correctly?

Can you explain more? What output do you get, and which one did you expect?

For example, I have the following string: "#{n} times\n". In order to color #{} and \n I use :self token, while the rest of the string is colored via :content token. I didn't know about :char, so it looks like I misuse these tokens a bit.

However, when you remove a color value of :content, the times section (in string #{n} times\n") gets colored to a color value of :self. That is a bit confusing, so I don't really understand what :self token does.

[korny]

Code interpolations inside of strings should be an :inline group, and the #{ and } parts would be :inline_delimiter tokens inside that group.

For that case we use :self token. Am I understanding you correctly?

There is no :self token. It's just the way the Terminal encoder specifies the color of the un-nested token (eg. :string). Maybe this is causing the problem you describe. Do you need to use the Terminal encoder? If not, I would suggest taking the :self method as an anti-pattern for now. Token nestings should instead be understood like CSS classes on nested SPANs, since HTML is the intended output type of CodeRay.

I'm sorry that there is no description of the token kind semantic right now ^_^ Maybe I can find the time to write an article about that. Feel free to ask if you run into more unexpected behavior.

[kyrylo]

Do you need to use the Terminal encoder?

Yes, the "connection" between my program and CodeRay is established via CodeRay::Encoders::Terminal::TOKEN_COLORS constant.

I'm sorry that there is no description of the token kind semantic right now ^_^ Maybe I can find the time to write an article about that. Feel free to ask if you run into more unexpected behavior.

It would be great. Thank you for the answers. Currently, I have no questions left. Thanks!

[kyrylo]

It looks like inline has a precedence over the inline_delimiter. When I set inline and inline_delimiter both, the latter merely isn't taken into account.

Example:

Given string: "#{foo} bar"

inline : red
inline_delimiter : blue

So instead of having a blue color for inline_delimiter (#{ and }) and a red color for foo, I have #{foo} colored to red. So a blue color of inline_delimiter is overwritten by the value of inline. When I set inline_delimiter only, I get #{ and } colored to blue and foo colored to default color of a terminal.

My question: how to separate apples from oranges? I want to get inline_delimiter working with inline.

[korny]

The Terminal encoder is not able to work with tokens nested deeper than 2 levels, as far as I know (it's not my implementation, but Rob Aldred's = @raldred, I believe).

If you want to have this functionality, you have to fix/enhance the Terminal encoder. I recommend branching off or copying it.

The CodeRay library contains no code to interpret all possible token nestings; rather, that's left to the browser in the form of HTML+CSS. Since the terminal has no concept of colors applied to nested spans of text, you would need to recreate quite a bit of browser layout logic.

Or you find something that converts HTML+CSS to ANSI colored text.

[kyrylo]

The Terminal encoder is not able to work with tokens nested deeper than 2 levels, as far as I know

But they are not nested: https://github.com/rubychan/coderay/blob/master/lib/coderay/styles/alpha.rb#L85-86

[korny]

They are, in the HTML code, and also in the token structure. The Debug encoder is helpful here:

puts CodeRay.scan('"#{foo} bar"', :ruby).debug
#=> string<delimiter(")inline<inline_delimiter(#{)ident(foo)inline_delimiter(})>content( bar)delimiter(")>

string<…> is a token group of the kind :string (with its content between the angle brackets), and delimiter(…) is a token of kind :delimiter (with the text between the parentheses). So, the structure of the token output is:

string<
  delimiter(")
  inline<
    inline_delimiter(#{)
    ident(foo)
    inline_delimiter(})
  >
  content( bar)
  delimiter(")
>

[kyrylo]

OK, thank you :)

I'll keep this issue open, if you don't mind (maybe someone wants to fix the nesting problems).

[nathany]

The current implementation doesn't handle more than one nesting of token groups. We need to immitate some CSS behavior.

From Redmine: http://odd-eyed-code.org/issues/145

[korny]

Changed the title subtly.