build, doc: use new api doc tooling

[flakey5]

Switches over to using the new doc generation tooling. For more background on this, please see #52343

Currently a draft just to get feedback on the approach to this integration.

cc @nodejs/web-infra

[nodejs-github-bot]

Review requested:

• @nodejs/nodejs-website
• @nodejs/web-infra

[ovflowd]

@flakey5 I guess you also need to check our GitHub Action Workflows, and also other mentions of these files within the source.

Like within the Contributor Docs, there is a file that describes how the legacy API doc tooling works, and I believe there are other references also.

[flakey5]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

[flakey5]

Also not sure what's going on with lint-addon-docs? cc @araujogui

[ovflowd]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

REPLACEME shouldn't error, imo, just give a warning. Our linter should have warn and error levels.

And yes introduced_in must be top level!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.76%. Comparing base (dfee0b1) to head (c4c479a).
⚠️ Report is 76 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #57343      +/-   ##
==========================================
- Coverage   89.89%   89.76%   -0.14%     
==========================================
  Files         656      654       -2     
  Lines      193141   192706     -435     
  Branches    37886    37744     -142     
==========================================
- Hits       173623   172974     -649     
- Misses      12051    12316     +265     
+ Partials     7467     7416      -51     

see 49 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[araujogui]

lint-js-and-md is failing because of the linting errors since it exits with a non-zero status code if there's anything wrong with the docs. I think we should skip the REPLACEME checks for normal ci runs.

It also looks like synopsis.md fails the introduced_in check because it's not under the top-level header, should it be enforced that introduced_in goes under the top-level header or should we change it to just check that it exists somewhere in the file (like it was doing previously)?

Actually, the linter only returns 1 if there's an error-level issue, and I don't think that's the case here.

[araujogui]

Also not sure what's going on with lint-addon-docs? cc @araujogui

I’m not sure either, but I’ll check it out.

[ovflowd]

@flakey5:

3:1-3:9   warning Use "the Node.js" instead of "Node.js'" prohibited-strings remark-lint
4:46-4:50 warning Use "Node.js" instead of "Node"         prohibited-strings remark-lint

On the README.md file you updated (tools/doc/README.md) after updating those can you run make format-md (?)

[ovflowd]

This is the result of many months of arduous work between many awesome folks, including @flakey5 @AugustinMauroy @araujogui @ovflowd @avivkeller and others.

I'm so proud of what we are achieving here and this is a huge step towards a modern tooling and a revamped API docs within Node.js

Approving, as I believe this is ready!

[ovflowd]

cc @nodejs/collaborators can we have another approval here? 🙏

[lpinca]

RSLGTM because it is hard to review and outside of my comfort zone.

[ovflowd]

cc @aduh95 are we good to merge?

[ovflowd]

(or is something still failing?)

[flakey5]

cc @aduh95 are we good to merge?

bump 😄

[aduh95]

I'm getting the following error when I try to generate several files:

Error: EEXIST: file already exists, mkdir '/Users/duhamean/Documents/node/out/doc/api/assets'
    at async mkdir (node:internal/fs/promises:857:10)
    at async mkDirAndCopy (node:internal/fs/cp/cp:308:3)
    at async Object.generate (file:///Users/duhamean/Documents/node/tools/doc/node_modules/@node-core/api-docs-tooling/src/generators/legacy-html/index.mjs:180:7)
Makefile:827: warning: pattern recipe did not update peer target 'out/doc/api/http2.json'.
make[1]: *** [Makefile:827: out/doc/api/embedding.html] Error 1

[ovflowd]

I'm getting the following error when I try to generate several files:

Error: EEXIST: file already exists, mkdir '/Users/duhamean/Documents/node/out/doc/api/assets'
    at async mkdir (node:internal/fs/promises:857:10)
    at async mkDirAndCopy (node:internal/fs/cp/cp:308:3)
    at async Object.generate (file:///Users/duhamean/Documents/node/tools/doc/node_modules/@node-core/api-docs-tooling/src/generators/legacy-html/index.mjs:180:7)
Makefile:827: warning: pattern recipe did not update peer target 'out/doc/api/http2.json'.
make[1]: *** [Makefile:827: out/doc/api/embedding.html] Error 1

@flakey5 any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

[flakey5]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

[ovflowd]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

☝️ @aduh95

[aduh95]

any idea here? Is this trying to generate all the files at the same time? Individual file generation should still work, no?

What's the command being used?

make docclean out/doc/api/addons.html out/doc/api/assert.html out/doc/api/async_context.html out/doc/api/async_hooks.html out/doc/api/buffer.html out/doc/api/child_process.html out/doc/api/cli.html out/doc/api/cluster.html out/doc/api/console.html out/doc/api/corepack.html out/doc/api/crypto.html out/doc/api/debugger.html out/doc/api/deprecations.html out/doc/api/dgram.html out/doc/api/diagnostics_channel.html out/doc/api/dns.html out/doc/api/documentation.html out/doc/api/domain.html out/doc/api/embedding.html out/doc/api/errors.html out/doc/api/esm.html out/doc/api/events.html out/doc/api/fs.html out/doc/api/globals.html out/doc/api/http.html out/doc/api/http2.html out/doc/api/https.html out/doc/api/index.html out/doc/api/inspector.html out/doc/api/intl.html out/doc/api/module.html out/doc/api/modules.html out/doc/api/n-api.html out/doc/api/net.html out/doc/api/os.html out/doc/api/packages.html out/doc/api/path.html out/doc/api/perf_hooks.html out/doc/api/permissions.html out/doc/api/process.html out/doc/api/punycode.html out/doc/api/querystring.html out/doc/api/quic.html out/doc/api/readline.html out/doc/api/repl.html out/doc/api/report.html out/doc/api/single-executable-applications.html out/doc/api/sqlite.html out/doc/api/stream.html out/doc/api/string_decoder.html out/doc/api/synopsis.html out/doc/api/test.html out/doc/api/timers.html out/doc/api/tls.html out/doc/api/tracing.html out/doc/api/tty.html out/doc/api/typescript.html out/doc/api/url.html out/doc/api/util.html out/doc/api/v8.html out/doc/api/vm.html out/doc/api/wasi.html out/doc/api/webcrypto.html out/doc/api/webstreams.html out/doc/api/worker_threads.html out/doc/api/zlib.html out/doc/api/addons.json out/doc/api/assert.json out/doc/api/async_context.json out/doc/api/async_hooks.json out/doc/api/buffer.json out/doc/api/child_process.json out/doc/api/cli.json out/doc/api/cluster.json out/doc/api/console.json out/doc/api/corepack.json out/doc/api/crypto.json out/doc/api/debugger.json out/doc/api/deprecations.json out/doc/api/dgram.json out/doc/api/diagnostics_channel.json out/doc/api/dns.json out/doc/api/documentation.json out/doc/api/domain.json out/doc/api/embedding.json out/doc/api/errors.json out/doc/api/esm.json out/doc/api/events.json out/doc/api/fs.json out/doc/api/globals.json out/doc/api/http.json out/doc/api/http2.json out/doc/api/https.json out/doc/api/index.json out/doc/api/inspector.json out/doc/api/intl.json out/doc/api/module.json out/doc/api/modules.json out/doc/api/n-api.json out/doc/api/net.json out/doc/api/os.json out/doc/api/packages.json out/doc/api/path.json out/doc/api/perf_hooks.json out/doc/api/permissions.json out/doc/api/process.json out/doc/api/punycode.json out/doc/api/querystring.json out/doc/api/quic.json out/doc/api/readline.json out/doc/api/repl.json out/doc/api/report.json out/doc/api/single-executable-applications.json out/doc/api/sqlite.json out/doc/api/stream.json out/doc/api/string_decoder.json out/doc/api/synopsis.json out/doc/api/test.json out/doc/api/timers.json out/doc/api/tls.json out/doc/api/tracing.json out/doc/api/tty.json out/doc/api/typescript.json out/doc/api/url.json out/doc/api/util.json out/doc/api/v8.json out/doc/api/vm.json out/doc/api/wasi.json out/doc/api/webcrypto.json out/doc/api/webstreams.json out/doc/api/worker_threads.json out/doc/api/zlib.json -j12

[avivkeller]

Shouldn't opts be passed here, which'll include the force used in the cp call in api-docs-tooling?

[aduh95]

It looks like CI's failing on a missing fi

[avivkeller]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

[ovflowd]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

Shouldn't we update the version being used here on the package.json?

[aduh95]

@aduh95 We’ve drastically improved performance, so I hope we’ve resolved all your concerns 😃

Shouldn't we update the version being used here on the package.json?

I think that's indeed needed as I'm still getting Error: EEXIST: file already exists, mkdir '…/out/doc/api/assets' on 0194b0e when running make docclean docserve -j.
Also there are conflicts to solve.

[ovflowd]

cc @flakey5 could you give an eye here?

[ovflowd]

Tests are failing, cc @flakey5 are these related or just flacky?

[github-actions]

Failed to start CI

   ⚠  Commits were pushed since the last approving review:
   ⚠  - build, doc: use new api doc tooling
   ✘  Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/17270174219

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/68926/