Some feedback on the Org parsing

[tecosaur]

Hello,

This is a bit of a drive-by issue as I'm not really a neovim user but I thought I'd check out the tree-sitter parsing since I think it looks cool. I've only had a little look, but a few rough spots have become apparent in the three files I checked out.

System setup

• Linux, OpenSUSE Tumbleweed
• neovim version v0.5.1
• orgmode.nvim commit c30b645
• tree sitter parser commit emiasims/tree-sitter-org@39a377f
• nvim config http://ix.io/3FM3 (basically bach's defaults.nvim + recommended setup from readme + conceal)

config.org

This is the first file I looked at, it's my Emacs config file, and it's massive.

Experience

File took a few seconds to open, then I got hit with

Error detected while processing CursorHold Autocommands for "*.org":
E5108: Error executing lua ...ack/packer/start/orgmode.nvim/lua/orgmode/utils/init.lua:297: too many results to unpack

The header keywords are all grey and underlined, and the rest of the document is red and underlined.

Thesis.org (file available on request)

This is the second Org file I tried, it's a ~60 page Thesis I wrote this year. A fair bit is going on, but it's smaller and less complicated than config.org.

Experience

Upon opening the file I immediately see the warning gutentags: ctags job failed, returned: 1.

While there's no error being blasted in my face like with config.org, unfortunately, the results don't look great here either.
All of the header keywords are underlined grey, except for the commented out subtitle which is red for some reason.
The first header, * LaTeX setup :ignore: is also red for some reason.
A few lines later we have the first source code block, and from then on the rest of the document is underlined red except for headings.

2021-04-26-Welcome.org

This is the first file that wasn't mostly underlined red text! 🎉 It's a fairly short and simple file, the first blog post I published.

Experience

Links

I notice that concel can cause some funny visual results with links

For reference, this is how that looks in Emacs,

Quite a few links also seem to be incorrectly parsed, such as

I tried changing the link target to see what was happening and this threw another error

Error detected while processing TextChangedI Autocommands for "*"
E5108: Error while executing lua ....e/pack/packer/start/cmp_luasnip/lua/cmp_luasnip/init.lua:65: attempted to call field 'get_snippet_filetypes' (a nil value)

Ignoring this and trying a few different link targets, it becomes apparent that this issue occurs when the link target spreads over multiple lines.

Inline source blocks

These don't seem to be handled well at the moment.

Meanwhile in Emacs,

It's worth noting that even if src_*{} constructs weren't special, this still shouldn't be handled as an underline. Say I replace src with hey in the Org file. This is what I then see:

Final comments

There are a lot of elements that aren't parsed at all, but I thought it would be best to ignore those for now while there prominent issues with the current parsing.

Lastly, apologies if this isn't as helpful as it could be. As mentioned at the start, I am not at all familiar with neovim (or vim for that matter) and so have just tried to highlight the roughest spots I've noticed. All in all I think it's fantastic to see a genuine effort being made with using tree-sitter based parsing for Org and wish you all the best going forwards. If there's anything I can do to help with a next to no neovim knowledge but a good knowledge of Org and some big/complicated files I'd be happy to assist where I can 🙂.

[kristijanhusak]

Hi,

thanks for the detailed report. I'm a bit sorry that you didn't do this earlier, while tree-sitter support was not on the master branch. It apparently has a lot of issues. Now when I see this, it seems that I maybe introduced tree-sitter support too early. I'll give a short explanation for each section, since it's caused by different things:

config.org

This is a big file, but I tested with up to 30k lines and didn't had these issues. My test didn't include big "content" chunks, just bunch of headlines. This is failing on a place where tree sitter parses node text for some purposes. This should be fixable, and it is very valuable feedback!

Thesis.org (file available on request)

Underline issue comes from bad markup parsing, which I'm hoping it will be solved in the tree-sitter parser at some point. Currently tree-sitter has some limitations for it, and I'm using the Neovim one, which is also not too smart. I would appreciate if you could send me this file via email so I can look into it.

Error that you got is not from the orgmode, it's from the vim-gutentags plugin which generates ctags file. I'm not sure why it errors out here though.

2021-04-26-Welcome.org
Links

Unfortunately, Neovim's conceal is much less powerful then Emacs. It hides the concealed text, but it still uses all the place as a content. I already got a report for that here.

Error that you got is not from the orgmode, but instead from cmp-luasnip plugin which provides autocompletion for snippets. I'm assuming you are using some prepared config like LunarVim or something like that (Emacs equivalents of doom emacs and spacesmacs), and it's not loading some things properly here.

Inline source blocks

I wasn't even aware that this exists in orgmode. Seems like I just didn't set up proper highlighting for these, since tree-sitter parser seems to parse it as a subscript.

Underline issue is similar to the one I mentioned in the Thesis.org section.

As I said above, this feedback is very helpful, and thank you very much for this. I'm hoping to solve some of these (which I can). I'll test all of these "the-month-in-org" files to see if I run into some other issues.

[mjlbach]

Error that you got is not from the orgmode, but instead from cmp-luasnip plugin which provides autocompletion for snippets. I'm assuming you are using some prepared config like LunarVim or something like that (Emacs equivalents of doom emacs and spacesmacs), and it's not loading some things properly here.

Teco is using https://github.com/nvim-lua/kickstart.nvim which is the source of the gutentags/luasnip issues (although I don't think there is any misconfiguration :)) luasnip isn't configured to use any custom snippets, only those provided by a language server (which presumably isn't running). Ofc if there is an issue LMK.

[yantar92]

I wasn't even aware that this exists in orgmode. Seems like I just didn't set up proper highlighting for these, since tree-sitter parser seems to parse it as a subscript.

Have you seen https://orgmode.org/worg/dev/org-syntax-edited.html ?

[bronzehedwick]

Not sure if this is entirely related, but recently, as of the last update as of this writing, my heading TODO and URLs lost their highlighting. DONE in headings are still highlighted for some reason.

[kristijanhusak]

@bronzehedwick what colorscheme are you using? There were some changes recently to default highlight groups, so maybe that's the issue. Try doing this to see if links get highlights: :hi @org.hyperlink guifg=red.

[bronzehedwick]

@kristijanhusak Yep, that was the issue, thanks!

For the record, I am using vim-colors-xcode.

[gerazov]

This might be a treesitter thing so not sure if this is a good place to post. Large files (>3k lines) are really laggy and hard to edit.

Here's an example of a lorem ipsum test file. As I type no text is shown and only at the end when I press ESC I get the full line I've written.

The problem persists when marking the file as Markdown (uploading it as md since GitHub doesn't allow org file upload).

The problem is gone when renaming the file to txt.

test.md