Bin exported without extension, breaks node's native loader picker and ts-node/esm

[FossPrime]

Description

custom loaders like ts-node have a really hard time figuring out the mocha "binary"

The binary lacks.js or cjs extensions. Changing the node_modules/.bin/mocha extension isn't enough.

Steps to Reproduce

1. Sandbox: https://codesandbox.io/s/mocha-ts-node-16-ctt5k?file=/package.json
2. Open Terminal
3. Run npm run broken-test-npx or run source env.sh; mocha test/test.js

OR

1. npm install ts-node typescript @types/mocha mocha on a type: "modules" project.json
2. Set export NODE_OPTIONS="--loader ts-node/esm/transpile-only"
3. Run npx mocha test/any-test-at-all.js

Expected behavior:

No errors, tests run.

Actual behavior:

Setting a loader prevents node from running the mocha binary, in any way

$ npm run test # Copies mocha to mocha.js as workaround
3 passing (3ms)
$ npm run test-cjs # Copies mocha to mocha.cjs as workaround
1 passing (2ms)
$ npm run broken-test-npx # Node cant tell what loader to use and errors out
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha
$ npm run broken-test-direct # same behavior as before
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /sandbox/node_modules/mocha/bin/mocha

Reproduces how often: 100% of the time

Versions

• The output of mocha: 8.4.0
• The output of node --version: 14 (Sandbox) and 16 as well
• Your operating system
  • name and version: yes
  • architecture (32 or 64-bit): 64
• Your shell (e.g., bash, zsh, PowerShell, cmd):
• Your browser and version (if running browser tests):
• Any third-party Mocha-related modules (and their versions):
• Any code transpiler being used (and its version): ts-node loader

Additional Information

Transpile only mode does not make a difference. Extension of tests do not make a difference. As long as the loader boots up, when node proceeds to guess which loader to use, node errors out.

[juergba]

This is an working example with Mocha, ESM and ts-node/esm.
Why do you think this issue is a Mocha issue? Not a loader or configuration problem?

[FossPrime]

@juergba The error actually comes from node... internal/modules/esm/get_format.js:71:15

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

TypeStrong/ts-node#116

Most symlinks in node_modules/.bin use extensions... I have the loader set to transpile only... it doesn't have much configuration to do.

lrwxrwxrwx   1 mind ya   21 Jun  2 13:15 mocha.js -> ../mocha/bin/mocha
lrwxrwxrwx   1 mind ya   22 Jun  2 11:45 ts-node -> ../ts-node/dist/bin.js
lrwxrwxrwx   1 mind ya   26 Jun  2 11:45 ts-node-cwd -> ../ts-node/dist/bin-cwd.js
lrwxrwxrwx   1 mind ya   29 Jun  2 11:45 ts-node-script -> ../ts-node/dist/bin-script.js
lrwxrwxrwx   1 mind ya   32 Jun  2 11:45 ts-node-transpile-only -> ../ts-node/dist/bin-transpile.js
lrwxrwxrwx   1 mind ya   40 Jun  2 11:45 ts-script -> ../ts-node/dist/bin-script-deprecated.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 eslint -> ../eslint/bin/eslint.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 esparse -> ../esprima/bin/esparse.js
lrwxrwxrwx   1 mind ya   28 May 12 09:45 esvalidate -> ../esprima/bin/esvalidate.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 flat -> ../flat/cli.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 json5 -> ../json5/lib/cli.js
lrwxrwxrwx   1 mind ya   25 May 12 09:45 js-yaml -> ../js-yaml/bin/js-yaml.js
lrwxrwxrwx   1 mind ya   15 May 12 09:45 katex -> ../katex/cli.js
lrwxrwxrwx   1 mind ya   33 May 12 09:45 markdown-it -> ../markdown-it/bin/markdown-it.js
lrwxrwxrwx   1 mind ya   14 May 12 09:45 mime -> ../mime/cli.js
lrwxrwxrwx   1 mind ya   20 May 12 09:45 mkdirp -> ../mkdirp/bin/cmd.js
lrwxrwxrwx   1 mind ya   24 May 12 09:45 nanoid -> ../nanoid/bin/nanoid.cjs
lrwxrwxrwx   1 mind ya   29 May 12 09:45 replace-in-file -> ../replace-in-file/bin/cli.js
lrwxrwxrwx   1 mind ya   16 May 12 09:45 rimraf -> ../rimraf/bin.js
lrwxrwxrwx   1 mind ya   23 May 12 09:45 semver -> ../semver/bin/semver.js
lrwxrwxrwx   1 mind ya   19 May 12 09:45 shjs -> ../shelljs/bin/shjs

nandoid goes as far as using cjs

[juergba]

Again, this is just how node works - it'll use the extension loader defined or fallback on the .js loader. We can't make it use ts-node as a result.

I haven't understood yet, or something is wrong in your argumentation.

• without extension the above fallback should be triggered. Which would be correct with .js loader. Why isn't this the case?
• why do you pass Mocha, a CJS module, to your ESM loader? For transpilation?
• why does my sample mentioned above work? What is the difference to your case?

[juergba]

If you remove --loader ... from NODE_OPTIONS and add it to .mocharc.cjs instead , does it work then?
I guess so, because in this case Mocha starts first, then the loader is called in a subprocess.. The entry point is lib/cli/cli.js.

[juergba]

Yes, above sample does not work anymore, when I remove loader: 'ts-node/esm' from .mocharc.cjs and use NODE_OPTIONS="--loader=ts-node/esm" instead.

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for D:\temp\ts-node-repros\node_modules\mocha\bin\mocha
    at defaultGetFormat (internal/modules/esm/get_format.js:71:15)
    at defer (D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:81:7)
    at D:\temp\ts-node-repros\node_modules\ts-node\src\esm.ts:103:12
    at Generator.next (<anonymous>)
    at D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:4:12)
    at getFormat (D:\temp\ts-node-repros\node_modules\ts-node\dist\esm.js:54:16)
    at Loader.getFormat (internal/modules/esm/loader.js:102:42)
    at Loader.getModuleJob (internal/modules/esm/loader.js:231:31)

@cspotcode can you have a look at this, please?

[cspotcode]

This is a confirmed bug in nodejs. Please remind them that they need to fix it! I have cited mocha as one of the libraries being broken by it, but it will help if mocha's users and maintainers also remind them (politely) that they need to fix this!
nodejs/node#33226

To reproduce the bug, use a completely empty --loader. This will prove that node is responsible for the error, not ts-node nor any other loader.

$ node -v
v16.2.0

$ touch empty-loader.mjs
$ touch extensionless-entrypoint
$ node --loader empty-loader.mjs ./extensionless-entrypoint
(node:10077) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
node:internal/process/esm_loader:74
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /c/Users/cspot/extensionless-entrypoint
    at new NodeError (node:internal/errors:363:5)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:71:15)
    at Loader.getFormat (node:internal/modules/esm/loader:105:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:243:31)
    at async Loader.import (node:internal/modules/esm/loader:177:17)
    at async Object.loadESM (node:internal/process/esm_loader:68:5) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

EDIT Fixed a typo above; the node --loader invocation was missing the ./extensionless-entrypoint

[juergba]

@cspotcode thank you for your explanation, I will remind nodejs tomorrow.

@rayfoss you have the workaround with .mocharc.cjs. I don't know yet what to do with our extensionless binary.

[cspotcode]

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

[FossPrime]

I think if mocha wants to implement a simple fix, they can add a new bin entrypoint file with a file extension (make sure to keep the old one, too, for backwards-compatibility) and then point package.json "bin" field to the new entrypoint with the file extension.

I believe that update can be published as a non-breaking change.

... My PR is not a breaking change in any way. mocha/bin/_mocha.js is a new entry point. And NPM makes a symlink called .bin/_mocha. People who use mocha/bin/_mocha directly or via the symlink .bin/_mocha would be unaffected.

There is an argument to be made that it should be mocha/bin/mocha.js, not mocha/bin/_mocha.js, which is then symlinked to .bin/_mocha by NPM, I was just preserving the convention established by mocha/bin/_mocha.

[cspotcode]

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

[FossPrime]

You mention a drawback in your PR; is that non-breaking?

We might be talking about the same thing. I'm imagining that mocha ships 4x different files: mocha, mocha.js, _mocha, and _mocha.js. package.json "bin" points to the ones with file extensions.

My PR only addresses the issue at hand, I couldn't see how you guys output bundled code... So the PR has

• _mocha (naked vanilla)
• _mocha.js (clothed vanilla)
• mocha (naked bundled)

Symbolic Links made by NPM:

• .bin/_mocha (& to clothed vanilla _mocha.js)
• .bin/mocha (& to naked bundled mocha)

We can make a mocha/bin/mocha.js that uses the naked bundled mocha/bin/mocha without duplicating the big file by using a funky, but legal, side-effect only, require or import call to the naked bundle.

Update: I corrected the description of the PR to reflect it's contents. I updated the PR code a few hours after publishing to duplicate naked vanilla mocha/bin/_mocha once I stumped on documentation and projects that directly referenced mocha/.bin/_mocha... And remembered Deno and custom package managers exist, and they may not read our package.json for the location of our stable endpoints.