docs/automation: include luv meta definitions files

[altermo]

Problem

Currently, there's no luv _meta file for LSP annotations, and the most used lua LSP is planning to drop the builtin luv meta file, which would require users to download a separate luv meta library.

Expected behavior

Have the luv meta-file library built in.

Downside

Adding a large meta library such as luv to the meta files would increase LSP load times.

[justinmk]

Thanks for raising this.

With the VS Code addon manager releasing soon, we have decided to remove the built-in libraries that ship with the language server

Sigh.

I am currently investigating how we can make using and sharing definitions files even easier. Ideally, the language server will be able to install and manage addons defined in a setting by simply providing the git clone URLs. This would hopefully be cross-platform and super easy for all users.

Hopefully this will be implemented, then users (or nvim-lspconfig) can point to the metafile URL. Unless this requires vscode?

[clason]

Unless this requires vscode?

Yes.

[clason]

I'm vaguely in favor of this, since luv is our main dependency and core infrastructure. It would also tie in to autogenerating luvref.txt -- maybe luvit themselves would be open to maintaining their own _meta file? I certainly don't think it's reasonable for us to maintain our own _meta_ file.

(Other directly exposed dependencies like lpeg and msgpack have diminishing returns.)

[justinmk]

Even if the "VS Code addon manager" doesn't help us, it seems like the metafiles will be maintained by LuaLS and available via some URL, which we could then consume via bump_deps.lua (automated vendoring) or a user command (so we don't need to vendor).

[lewis6991]

This is still an unfinished thread in my lua type annotations crusade: luvit/luv#632

[clason]

https://github.com/Bilal2453/luvit-meta

(in particular https://github.com/Bilal2453/luvit-meta/blob/main/library/uv.lua)

[wookayin]

First of all, this has less thing to do with lsp but is all about documentation and automation, so the title could be changed as docs: instead of LSP:.

I am investigating how we can include luv (vim.uv) meta files, and also generate or maintain vimdoc luvref.txt. As already discussed earlier in this issue, this is quite a complex one until luvit/luv#632 is resolved. I raise several discussion points as follows.

What would be the gold (upstream) "source" for libuv documentation that neovim can pull?

We have two choices.

(1) https://github.com/luvit/luv/blob/master/docs.md is the docs maintained and kept up-to-date with the actual libuv implementation.

• Pros: This doc is supposed to be always up-to-date with the actual libuv implementation, so the docs and the library will remain synchronized and compatible with each other.
• Problem: it's not machine-readable (see Maintain documentation in a machine readable format luvit/luv#632).
• Possible solution: Manual processing/edit is needed. Essentially the same (duplicate) efforts as in (2) done on the neovim side.
• Possible solution: Or we can write a scripted generator, but this would be of too much work and maintenance burden (lots of corner cases will bother us).

(2) https://github.com/Bilal2453/luvit-meta (maintained by @Bilal2453)

• Pros: Already working solution for Lua-LS. @folke's neodev also bundles exactly the same file uv.lua from this repository. If neovim pulls this, neovim doesn't need to own or maintain this.
• Problem (maintenance): This is maintained by "manually editing" the meta file, following the changes time to time.
• Problem (maintenance): It's not clearly tracked or mentioned that the current HEAD version is compatible with or catching up to which luv version. Therefore, we cannot guarantee that the meta file exactly matches the libuv version we include as the CMakeFile dependency. This could be improved in coordination/collaboration with @Bilal2453, but since this is basically a 3rd-party repository we don't have much control.
• Problem (technical): neovim's vimdoc generator (lua2dox + gen_vimdoc) does not like the format. This is mainly because of two problems (1) overload functions and (2) alias types, see below.

(3) What lua-ls currently ships as meta/3rd

• https://github.com/LuaLS/lua-language-server/tree/master/meta/3rd -> https://github.com/LuaCATS/luv/blob/3615eb12c94a7cfa7184b8488cf908abb5e94c9c/library/uv.lua
• Pretty outdated, 2 years old; this is going to be removed from lua-ls as @altermo already pointed out, so we don't need to consider this one.

Technical Problems on doc generator on the uv.lua from Bilal2453/luvit-meta

Note that the current meta file uv.lua works just well for Lua LSP. But the problem is when neovim wants to generate vimdoc (luvref.txt) from this meta file.

(1) Overload functions with different params/returns signature, e.g., uv.fs_open().

SOLUTION: This can be addressed on the neovim side without requiring the meta file to change its style: I managed to improve gen_vimdoc and lua2dox to handle overloaded functions.

Example of generated *uv.fs_open()* vimdoc

uv.fs_open({path}, {flags}, {mode}, {callback})                 *uv.fs_open()*
uv.fs_open({path}, {flags}, {mode})                                           
    Equivalent to `open(2)`. See below for available access `flags`.

    Note: On Windows, libuv uses `CreateFileW` and thus the file is always
    opened in binary mode. Because of this, the `O_BINARY` and `O_TEXT` flags
    are not supported.

    Parameters: ~
      • {path}      (`string`)
      • {flags}     (`uv.aliases.fs_access_flags|integer`)
      • {mode}      (`integer`)
      • {callback}  (`fun(err: string, fd: integer?)?`)

    Return: ~
        (`uv_fs_t`)

    Parameters: ~
      • {path}   (`string`)
      • {flags}  (`uv.aliases.fs_access_flags|integer`)
      • {mode}   (`integer`)

    Return: ~
        (`integer? fd, string? err_name, string? err_msg`)

Admittedly it's a bit ugly to show [Parameters & Returns] twice. Merging them into one is non-trivial because parameter/return types are not necessarily the same.

(2) Parameters for complex table options are translated into alias lua types, e.g.

• uv.aliases.getaddrinfo_hint for uv.getaddrinfo
• and their human-readable explanation on the parameters are intentionally and manually removed from the docstring.

While this might be useful for Lua_LS, this causes some problems when generating vimdocs (luvref.txt) because our doc generator does not unroll the alias types. Personally I think what is written in the markdown doc is more useful for vimdoc; but as we need to generate vimdoc automatically from this meta file, the docs included in the current version of the meta file won't be enough.

• OPTION 1: Update the meta file uv.lua to include this back (the simplest solution, but requires a major re-work on the meta file).
• OPTION 2: Just don't show them. This would be a regression in the luvref.txt (Example below)

Example for OPTION 2

-uv.getaddrinfo({host}, {service} [, {hints} [, {callback}]])  *uv.getaddrinfo()*
-
-                Parameters:
-                - `host`: `string` or `nil`
-                - `service`: `string` or `nil`
-                - `hints`: `table` or `nil`
-                  - `family`: `string` or `integer` or `nil`
-                  - `socktype`: `string` or `integer` or `nil`
-                  - `protocol`: `string` or `integer` or `nil`
-                  - `addrconfig`: `boolean` or `nil`
-                  - `v4mapped`: `boolean` or `nil`
-                  - `all`: `boolean` or `nil`
-                  - `numerichost`: `boolean` or `nil`
-                  - `passive`: `boolean` or `nil`
-                  - `numericserv`: `boolean` or `nil`
-                  - `canonname`: `boolean` or `nil`
-                - `callback`: `callable` (async version) or `nil` (sync
-                  version)
-                  - `err`: `nil` or `string`
-                  - `addresses`: `table` or `nil` (see below)
+
+                                                           *uv.getaddrinfo()*
+uv.getaddrinfo({host}, {service}, {hints}, {callback})
+uv.getaddrinfo({host}, {service}, {hints})                                    

     Equivalent to `getaddrinfo(3)`. Either `host` or `service` may be `nil`
     but not both.

-                Valid hint strings for the keys that take a string:
-                - `family`: `"unix"`, `"inet"`, `"inet6"`, `"ipx"`,
-                  `"netlink"`, `"x25"`, `"ax25"`, `"atmpvc"`, `"appletalk"`,
-                  or `"packet"`
-                - `socktype`: `"stream"`, `"dgram"`, `"raw"`, `"rdm"`, or
-                  `"seqpacket"`
-                - `protocol`: will be looked up using the `getprotobyname(3)`
-                  function (examples: `"ip"`, `"icmp"`, `"tcp"`, `"udp"`, etc)-
+
+    Parameters: ~
+      • {host}      (`string?`)
+      • {service}   (`string?`)
+      • {hints}     (`uv.aliases.getaddrinfo_hint?`)
+      • {callback}  (`fun(err?: string, addresses?:
+                    uv.aliases.getaddrinfo_rtn)`)

-                Returns (sync version): `table` or `fail`
-                - `[1, 2, 3, ..., n]` : `table`
-                  - `addr` : `string`
-                  - `family` : `string`
-                  - `port` : `integer` or `nil`
-                  - `socktype` : `string`
-                  - `protocol` : `string`
-                  - `canonname` : `string` or `nil`
- 
-               Returns (async version): `uv_getaddrinfo_t userdata` or `fail`
+   Return: ~
+       (`uv_getaddrinfo_t`)

Users will have no idea about uv.aliases.getaddrinfo_hint, uv_getaddrinfo_t, etc.

• OPTION 3: Auto-generate vimdoc from the meta file, but resolve the alias type (uv.aliases.getaddrinfo_hint?) to generate a sub-list (family, socktype, etc.). This will require a non-trivial work on our documentation generator.

I can open a draft PR to show the preliminary work after some clean-up.

(3) Generating sections for vimdoc is also difficult because of the different doc structure (@section v.s. @defgroup) being assumed.

Decision to make:

• Should neovim "own" a meta file, or pull from Bilal2453/luvit-meta or directly from docs.md?

• Do we generate luvref.txt or maintain them manually (in addition to _meta/uv.lua)? How much difference between luvref.txt and docs.md we will allow?

• Or should we wait until Maintain documentation in a machine readable format luvit/luv#632 is done? (it seems that it won't be done in the very near future)

• An alternative possible approach is the opposite way; to generate (manually maintained) the meta file for vim.uv from luvref.txt, instead of generating luvref.txt from the meta files. This is the former approach used by @folke in neodev, but currently it was superseded by just fetching the uv.lua meta file. In neovim we don't generate meta files from doc files, so I don't think this is a very compelling approach.

Also, we should make sure that luv meta files and docs are up-to-date and properly synchronized (either manually or automatically) whenever libuv dependency is bumped. #22186

If we are fine with a "minimal viable" vimdoc that might be somewhat different from the current `luvref.txt or https://github.com/luvit/luv/blob/master/docs.md, we can choose the option to use @Bilal2453's one (with some patches contributed) and generate vimdoc from it.

[clason]

For the record, there's a variant of these options: We provide the upstream meta file (starting from one of the existing files, edited to suit us), in the hope that once it's there, upstream will maintain it and we can just pull it on any luv bump.

[lewis6991]

That might be the easiest thing tbh.