Workflow for generating meta definition files

[SwissalpS]

I'm looking for pointers on how to proceed to generate meta definition files for a somewhat sizable lua project.
https://luals.github.io/wiki/export-docs/ hints that documentation generation is built in and that the output could have different formats.
So I'm wondering if this is the way to go, to make some custom template that formats for meta definition files.

It seems others have generated meta files for their libraries and I doubt they did it all manually.
I'm aware that there will be some manual effort needed, especially since this project hasn't been using a strict standard for commenting the code.

Thanks in advance for any pointers.

[SwissalpS]

Looking at tools/build-3rd-meta.lua and 3rd/love-api/ it seems they scrape their wiki pages to build tables that can be used with the build-3rd-meta.lua tool.

I was thinking, since lua-language-server already scrapes tokens from source files, it could be possible to export those in some way. Especially the ones that matter. Tokens from inside functions can be skipped for the most part.

[tomlau10]

Looking at tools/build-3rd-meta.lua and 3rd/love-api/ it seems they scrape their wiki pages to build tables that can be used with the build-3rd-meta.lua tool.

I don't know much about love-api or the generation tool, but I guess the tool is tailor made to read the wiki page:

• extract the api functions as well as the types
• and then generate a meta definition file
• this is a library / framework which will be used in other projects
  so other projects can then include this definition file to enjoy love-api's type hint / completion

---

The export doc feature is for exporting documentation (currently support markdown and json format AFAIK), which requires your codebase to be annotated first if you want the exported doc to be meaningful.

What's your aim by saying generate meta definition files for a somewhat sizable lua project ?
Without annotating your lua project codebase beforehand, I don't think you can generate meaningful meta definition files from the codebase itself. 😅 (the garbage in garbage out principle)

maybe you want annotations?

If what you want is to make luals smarter when doing type inferencing, meta definition might be a wrong approach.
You are going to need annotations in your codebase: https://luals.github.io/wiki/annotations/
Without annotations, luals can only do it's best when inferencing type, and most of the time it will infer a variable as any type which means it cannot guess its type.

Suppose you have a util function

function util.strcat(a, b)
    return a .. b
end
--[[ inferred type:
function util.strcat(a: any, b: any)
  -> unknown
]]

• if you want better inferencing, you need inline annotation in your codebase

---@param a string
---@param b string
function util.strcat(a, b)
    return a .. b
end
--[[ inferred type:
function util.strcat(a: string, b: string)
  -> string
]]

I think it is impossible for you to generate proper annotations from your unannotated codebase directly 😕 .
On the other hand, if luals already did the correct type infer for some functions, then you may not need to annotate them.

By just studying LuaLS (this github repo) project, you can find that inline annotation are all over the place. And the meta definition files (file with @meta) are for some native libraries only. As you cannot add lua annotation to the native library itself.

[SwissalpS]

The export doc feature is for exporting documentation (currently support markdown and json format AFAIK), which requires your codebase to be annotated first if you want the exported doc to be meaningful.

json sounds fine, that can then be parsed by various tools. The first step is to have such a db of all globals and their functions. I still haven't figured out how to actually export anything. The wiki only
says something about adding "--doc" flag. Running the server with that didn't seem to work at all.

Yes, I'm aware that annotations would be great to have. The problem is that every project follows a different style or none at all.
(Some argue that it makes code bulky and hard to maintain, in some cases I agree with that group. Working with simple editors that don't auto-hide anotations it can be really hard to scroll through pages of them. There are also cases where file size does matter, so separating production code and annotated meta files makes sense.)

The most important part for me is to have a db per version to be able to validate used functions against versions. That this db would also improve autocomplete, validate arguments and types etc. is just a bonus for now.

maybe you want annotations?

My idea is that once I have meta files, I could add annotations in there. What can be scrapped from code could already be pre-populated of course.
Old versions won't change, so this would only have to be done once and can then be used to export into various other formats for various end-goals.

If what you want is to make luals smarter when doing type inferencing,......, and most of the time it will infer a variable as any type which means it cannot guess its type.

It is not my first priority, having type any is already a lot better than not having any reference to its existance :D

... On the other hand, if luals already did the correct type infer for some functions, then you may not need to annotate them.

Exactly, anything that can already be done with automation helps reduce the workload.

By just studying LuaLS (this github repo) project, you can find that inline annotation are all over the place. And the meta definition files (file with @meta) are for some native libraries only. As you cannot add lua annotation to the native library itself.

Yes I've seen that and the style is written for LuaLS. I can't expect a big segment of lua developers to switch to LuaLS compat just as you cannot annotate libraries you don't have control over.

I'm aware that my end-goal is not synced with the purpouse of this repo. It's just that this repo could provide an intermediary tool (as it deals with similar processes) and as thanks I'd make sure my tools would provide a compatible output also for LuaLS to use.

[tomlau10]

says something about adding "--doc" flag. Running the server with that didn't seem to work at all.

Yeah, the doc export function is a little bit bugged after a recent major refactor, as reported in these issues:

• --doc CLI argument does not work #2954
• Doc export file paths all start with ile:// #2971
• LuaLS autodoc is including global table in output md/json files #2977

for some reasons, you have to also specify the --doc_out_path in the mean time:

lua-language-server --doc=. --doc_out_path=.

then it should be able to generate a doc.json and doc.md

Yes, I'm aware that annotations would be great to have. The problem is that every project follows a different style or none at all.
(Some argue that it makes code bulky and hard to maintain,
...
It is not my first priority, having type any is already a lot better than not having any reference to its existance :D

I totally agree with you. In my team's project, we use annotation as minimal as we can.

In the old days, my old team leader loves doxygen, probably because documentation can be generated with doxygen annotations.
That's not my cup of tea and I think it makes the codebase very bulky, and it's not much of use when writing code since doxygen itself has no type hinting or validation.
=> it's like a useless decoration 🤦‍♂️

‼️ But with LuaLS, the world is entirely different 👀 .
Annotation have its use, because LuaLS diagnostics are based on LuaCAT (the annotation syntax used by LuaLS).

• For example you can mark a table as ---@enum

---@enum MyEnum
MyEnum = {
  a = 1,
  b = 2,
  c = 3,
}

print(MyEnum.d) --> warning: Undefined field `d`
-- if without the `@enum`, then no undefined check is performed on the `MyEnum` table
-- I caught quite a number of bugs in my team's project when I first started to annotate the codebase :D

• by using inline annotation, you just need 1 line @enum at the location of MyEnum
• however if you want to do this in a separate meta definition file, then you have to copy the whole table
  otherwise luals won't know the content of your global table ☹️
  moreover the values in your separate meta definition file might be out of sync very easily
  it would be much worse than just using inline annotation in this case

---

My idea is that once I have meta files, I could add annotations in there. What can be scrapped from code could already be pre-populated of course.

I have no idea how to automate the process, but IIUC you will want to achieve something like this finally?
Imagine the whole project just have a util.lua module, and a main.lua

• util.lua

local util = {}

function util.add(a, b)
    return a + b
end

function util.sub(a, b)
    return a - b
end

return util

• meta/util.d.lua

---@meta util # this mean whenever `require "util"`, luals looks up definition from this file, instead of original file

local util = {}

---@param a integer
---@param b integer
---@return integer
function util.add(a, b) end

---@param a integer
---@param b integer
---@return integer
function util.sub(a, b) end

return util

• main.lua

local util = require "util" --> luals will know to use `meta/util.d.lua` instead of `util.lua`

local c = util.add(1, 2)) --> integer
local d = util.sub(3, 4)) --> integer

• In this way, you can separate the meta definition and actual implementation
• but be reminded that with this usage pattern, you will lose the goto definition function from luals.
  Because it will jump to the meta/util.d.lua instead of the actual implementation.
• moreover, type hint will not work for the functions inside util.lua, because luals cannot link the definition to your implementation.
  => it can only link the returned value of a require() call

[SwissalpS]

Thanks for the elaborations. Good to know the caviats.
If the autogenerated meta files contain links to source of definition (file and line is sufficient to generate a URL to webpages), that would already be very helpful middle ground.
The sync issue isn't so important as this is mostly for published versions that aren't subject to change. The changes will appear in the next versions reference.

In the past I've worked with LDoc, LuaDoc, Doxygen, Sphinx, Breathe, etc. too and some of them are still on the table for after there is a json file.