Add support for expressions and nested strings in templates

[lddubeau]

[Opening an issue for a problem discovered while discussing #20 .]

Template strings allow expressions in string and strings to nest. For instance:

`a${1+2}b`

equals "a3b". And since expressions can be strings, then this is valid:

`a${`b`}c`

and equals "abc"

Problematic areas:

1. The highlighter fails to handle anything in ${} as an expression. (Trying the same in plain JS code: js-mode is just as broken as typescript-mode. js2-mode, however, handles it fine.)

2. typescript--re-search-backward-inner completely ignores expressions in strings or the fact that strings can nest. Same for typescript--re-search-forward-inner. These are used for movement across syntactic features and for indentation.

3. syntax-ppss, which is also used for movement across syntactic features and for indentation, ignores string nesting. js2-mode appears to have solved the issue though because I get meaningful results from checking the values of syntax-ppss in nested strings in a JS file with js2-mode.

There may be yet other problematic areas I've not identified.

[josteink]

The highlighter fails to handle anything in ${} as an expression

I'm going to just assume that however we decide to handle this, it ultimately comes down to syntax-tables and other relatively low-level cruft.

To my knowledge syntax-tables have no understanding of interrupting string-sections with intermediate code-blocks by a string-delimiter-escape-character, before the subsequent string-section continues until the original string-delimiter is met (and whatever evil cyclic conditions one can manufacture using this technique).

We can try hack our way around this, but if this functionality is needed in js-mode too anyway, I think it would be best (yes, I know my previous statements about these matters), to first investigate with emacs-devel if they:

1. are aware of the problem/use-case
2. are planning any changes to syntax-table related functionality to accommodate syntax like this.

There's no need to reinvent wheels emacs-devel is making for us (if they are making it for us).

Agreed? Disagreed?

[lddubeau]

I agree. It may turn out that typescript-mode will remain its own independent thing rather than aggressively derive from js-mode but even if that's the case, we can benefit from any fix made to js-mode.

I'll note again that js2-mode appears to have fixes for the highlighting and syntax-ppss so we can look at that if js-mode remains broken.

[josteink]

I've written a fairly detailed email about the issue to emacs-devel and put you on CC.

Feel free to chime in on the issue and thread if you feel like I missed something important.

[danielpza]

Any news about this ?

[root0x]

Any update on this? Its been years.

[josteink]

I think the only realistic way this is getting solved is if somebody is willing to do what @theothornhill did for csharp-mode: start a "from scratch" rewrite based on the tree-sitter package and library.

IMO that rewrite worked out really well. So well in fact I tried to do the same for typescript-mode, but I quickly backed off once I realized how far I was from being able to complete it ;)

If somebody else however wants to take a stab at that, I would fully support such an initiative. 😄

[theothornhill]

Hi! I'm actually looking to do this, since I'm a little bothered by the indentation troubles with typescript. In particular inside jsx. I've been holding off a little because I was waiting for the native implementation by Yuan Fu to land in emacs proper. I could do the same for typescript mode if you want, but I'm a little unsure if it is best to wait for emacs proper. I saw you added me to this repo, so I can start working on a branch anyways if you'd like. I write a lot of ts these days, so my motivation is improving, hehe.

I think it should be pretty fast to do - the most problematic things would probably be setting the indentation properly, but most people use prettier anyways these days don't they?

Thinking about it I think most of the work should be in the indentation engine.

[theothornhill]

@josteink how about I take over your branch and turn the pr into a working draft for now? Then we'll see how far I get :)

[josteink]

Of all bad luck, I think I only had that as a local branch... On a laptop I had Ubuntu on, but literally yesterday reformatted and installed Debian on instead. Ooops 😁

[theothornhill]

No I found it. No problem! I've started the work. Some indentation is working already, so I think this will be quick. How about we cooperate on this? I'll push what I have and we can just use it and extend it? Right now all you need to do is to C-M-x the tree-sitter-indent-typescript-tree-sitter-scopes inside of typescript-tree-sitter.el and revert the buffer, and the new indentation can be used. tree-sitter-debug-mode is a must to make this work. So when you are in a meeting you can do something useful :) I'll just push what I have now and start using it myself. Sounds good?

[theothornhill]

pushed