Roadmap (preliminary draft)

[evanplaice]

Status: On hold, feel free to ignore this topic until further notice.

Below is a draft that attempts to structure some of the issues that have been addressed in @ceejbot's proposal as well as some of the other discussions that have taken place so far. This can be further broken up into separate issues once general consensus is reached on a starting point.

Roadmap (preliminary draft)

Milestone 0 - target [current] - CommonJS in Node

Goal: Ensure backward compatibility

• leave all existing built-ins as is (don't modify existing globals in Node)

Details

It's unrealistic (at this time) to change existing built-ins without breaking the current node module ecosystem. It's not productive to even beginning to discuss deprecating CJS until ESM has been implemented and used extensively in the Node ecosystem.

This could be left unsaid but it's important to establish intent. This group doesn't exist to obsolete CJS without consideration of the greater Node community.

---

Milestone 1 - target [v10.0] - ESM Support in Node

Goal: ESM modules work in Node without plugins, loaders

• Specification
• mode:esm can be explicitly set by a CLI flag
• mode:esm can be explicitly defined in package.json
• mode:esm throws error if CJS is imported as ESM

Details:

This is only the first step. It does not include ESM<->CJS interop. It focuses on providing ESM support in Node. Any attempt to mix ESM with CJS or vice-versa will throw an error.

10.0 is a notable release number and provides an opportunity to mark a big transition point. For example, OS9 vs OSX. It can be used to mark a transition. For example, there is a huge body of documented questions on StackOverflow addressing existing CJS workflows. If the introduction of ESM happens with 10.0, it should be easy to tag all Node questions on StackOverflow with pre-10 vs post-10 context tags.

This will provide an opportunity for early-adopters to experiment with ESM in Node.js. It may also provide an opportunity for eager devs to produce alternative workarounds that address dealing with ESM<->CJS interop.

---

Milestone 2 - target [undefined] - ESM in CJS

Goal: Make ESM work in CJS

Out of scope. This is already addressed in userland by @std/esm and/or transpilation. No changes are required in Node.

---

Milestone 3 - target [undefined] - CJS in ESM

Goal: Make CJS work in ESM

• Specification
• add import.meta for ESM-compatible CJS built-ins

Details:

What needs to take place to make CJS mimic ESM in an ESM loaded module?

• can import.meta.require be made async?
• does making it async require top-level await?

CJS is the better 'known quantity' in the Node ecosystem. That's great! Now, what does it require to make require() work like ESM.

---

Extension A - target [undefined] - Conversion paths for CJS -> ESM

Goal: What does it take to convert CJS to ESM

Out of scope. This would be better addressed by userland.

---

Extension B - target [undefined] - Loader Hooks

Goal: Address how loader hooks can be implemented/used within the current scope

---

Extension C - target [undefined] - Loader Extensions

Goal: Addressing interop with existing loader extension workflows (ex APM)

---

Extension D - target [undefined] - npm asset interop

Out of scope

---

Updates:

• removed 'no expermental flags' from Milestone 1
• added Specification as a requirement to each milesone
• simplified M1
• Note about transpiling removed, out of scope
• Extension A marked out of scope
• Extension D marked out of scope
• Marked M2 as out of scope, userland already provides a solution

[devsnek]

ESM can be automatically detected and loaded

last two years of discussion seem to say this isn't possible

It does not include ESM<->CJS interop

since we already have interop and 99.9% of all the node code is not esm i see no reason to not make this part of the goal

transpile ESM to CJS

can't we just leave this in userland

[MylesBorins]

I think we would benefit from making the Roadmap extremely terse for right now, potentially starting with the current implementation and working towards the goal of removing the flag

[evanplaice]

@devsnek

ESM can be automatically detected and loaded

Following, @ceejbot's proposal. Her recommended approach involved detecting the first instance of require|import and setting a global mode flag so all subsequent modules are handled the same way.

This could be overridden from a CLI entry point by specifying a --mode flag. @guybedford also mentioned adding a similar field to package.json for module loading.

It does not include ESM<->CJS interop

With or without a .mjs extension? Last I checked importing ESM as .js does not work.

transpile ESM to CJS

Yes, and that could be documented as a recommendation or left out altogether.

@MylesBorins Feel free to 'trim the fat' or rearrange as you see fit.

This is more of a suggestion to deal with planning. Right now, we're blocking on 'everything that addresses everything should all happen at once'. IMO, it would be more realistic to identify a concrete starting point and flesh out the unknowns over time.

[weswigham]

Milestone 1, by nature, needs to include a definitive specification of how spec-unspecified behaviors will resolve in esm, with future extensions in mind. Specifically, how module specifiers will resolve by default. Also it must be stated what the goals of consuming ESM (at all) are. Why ESM? What should drive a package or application author to choose ESM over CJS? At milestone 1, is it expected that all esm we load would also be directly loadable in a browser or other ESM compatable runtime, completely unmodified? It should also be noted what the impact of actually shipping with only those features listed in M1 available would be. Doesn't shipping without interop in either direction start to fracture the package ecosystem along module system bounds? Should that be avoided? If so, can M1 actually be shipped? or is it just an internal milestone? (If so, can you really say it's scheduled for 10.0?) Also, why the assumption about what package and cli flags will set goals? AFAIK, I've only seen these in proposals that exist to compete with the disambiguate-via-extension one - your milestone seems to be a bit presumptuous, unless I've missed something.

Milestone 2's a bit confusing. Is this just about loading ESM in a CJS context? I was under the assumption that top-level dynamic import would just support this, either exclusively (only loading esm), or by disambiguating load kind via side channel info (package json or extension). I suppose those decisions must be finalized?

Milestone 3's where I've seen (and participated in) a lot of contention. I honestly believe consensus and implemention on these topics is what's required to ship unflagged and not do harm to the ecosystem, and not before. Moreover, as written it precludes module hooks or loaders (builtin ones, even) from being part of this plan (since they're left in "extension" categories). Based on some discussions, depending on how other things pan out, the most viable routes for interop may involve custom loaders. It's pretty obvious that interop is required for a successful es module deployment, but what's done here directly impacts browser compat/portability conerns (which were hopefully decided on back during M1?). Even just allowing path traversal for looking up modules is enough to break untooled portability. For that reason, making interop opt-in via a package-scoped loader that declares interop behaviors seems reasonable.

To boot, this kinda ignores the current implementation. Node can already parse/load esm (though not via the same specific mechanisms specified in M1) - right now I'm pretty sure the goals are syncing knowledge and coming to consensus on is a short list of concerns around module system interop and portability.

[WebReflection]

following up on what @MylesBorins said, I think a flag that is not experimental but covers ESM in a way we like could be already considered a great success and make migration also easier to opt in.

Stats based on npm modules would be easier to read so that in the future such flag could be dropped once ESM is the default (similar to a use strict migration approach).

Here how I think milestone 1 as MVP could work:

Milestone 1 - target [v10.0] - ESM Support in Node

Goal: ESM modules work in Node without plugins, loaders

• mode:esm can be explicitly set by a CLI flag
• mode:esm can be explicitly defined in package.json
• mode:esm provides a way (import.meta.require or another) to import CJS too
• mode:esm throws error if CJS is imported as ESM

The last point is basically a consequence of the fact ESM module won't contain module or exports or this and neither require, so that basically it will inevitably throw when imported as ESM.

Those points should be hopefully enough to grant ESM in Node, where omitting the flag would result in regular CJS behavior, where requiring untranspiled ESM would fail same way imported CJS would in mode:esm.

[evanplaice]

@weswigham See @ceejbot's proposal. I'm just attempting to establish a starting point based on what has already been said. None of this is concrete, none of this has consensus.

Also it must be stated what the goals of consuming ESM (at all) are. Why ESM? What should drive a package or application author to choose ESM over CJS?

IMO, this is out-of-scope. The community will decide whether ESM is a path they wish to pursue. We're just here to provide that as option.

Yes, M2 addresses loading ESM in a CJS context. AFAIK, no it doesn't just work. If you set context via flag it will prefer one-or-the-other not both. CJS and ESM are fundamentally different, the former loads dynamic/sync, the latter static/async.

I added Specification as a requirement for each of the milestones.

I agree M3 is 'here be dragons' territory. Maybe the specifics will become more clear once M1 is reached. Maybe, further experimentation in the wider ecosystem can provide more context.

Based on some discussions, depending on how other things pan out, the most viable routes for interop may involve custom loaders.

I agree. The order of extension is neither implied nor assumed. It could come before or after, or even be required by one of the milestones. I'm just posting them here to establish that they likely will fall within the scope of this project.

[evanplaice]

@WebReflection I agree except for one point

• mode:esm provides a way (import.meta.require or another) to import CJS too

I added this as M3 because require() works synchronously. Will this cause issues when loaded with ESM? Is it necessary to implement import.meta.require() an async variant of require()? If so, would this change require the addition of top-level await?

That's also the reason why M3 comes later. M1/M2 should be possible with the the existing ES spec.

[MylesBorins]

@evanplaice while the npm proposal and implementation are a great place to draw inspiration from I think we would be better served by starting from the implementation that currently exists in node. The road should start with where we are and reach where we want to go

re: import.meta.require it is run in the body which is executed synchronously in ESM as it is... there is no conflict or need to play with anything... it is also something we could ship today (PR was opened but scaled back)

[evanplaice]

@MylesBorins OK, just to clarify would it best to swap M2 <-> M3 and drop any mention of async loading?

[MylesBorins]

TBH I think that this current approach of a roadmap is not starting from the right place and likely needs to be reworked / envisioned. Unfortunately I am at the end of my work day so I can't take the time to respond with an alternative, but what I think we need to do before starting to map things in time is define a couple things

First thing we need to do is define our current implementation

• what are the current features of our implementation
  • this is where we start

Once we have this we should then start to collect things that need to be done, these things can be in conflict:

• what still needs to be implemented
  • for example modes
• what behavior may have to change
  • for example transparent interop (this is editorialized)

this should be done independent of timelines at first

We should then define shipping goals which we've begun to do in #4

The combination of physical timelines and shipping goals can help us start to place the proposed changes that we have identified. Conflicting changes can then be debated at the point that they block work.

[evanplaice]

Cool. I'll step back for a bit and wait to see how some of the other discussions pan out.

[MylesBorins]

No prob... Thanks for taking a first stab at it! Feel free to keep iterating.

…

On Wed, Feb 7, 2018, 7:06 PM Evan Plaice ***@***.***> wrote: Cool. I'll step back for a bit and wait to see how some of the other discussions pan out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV3InTiINCno3ollf0m9Xv-cntjc6ks5tSjpqgaJpZM4R9eHb> .

[weswigham]

Also it must be stated what the goals of consuming ESM (at all) are. Why ESM? What should drive a package or application author to choose ESM over CJS?

IMO, this is out-of-scope. The community will decide whether ESM is a path they wish to pursue. We're just here to provide that as option.

@evanplaice I'm pretty sure that the feature is being designed for someone, or at least with certain developers and workflows in mind. It's best to be explicit about who these groups are and what their priorities are so you can properly evaluate what tradeoffs to make during design, and how those choices affect each of those groups.

[bmeck]

mode:esm can be explicitly set by a CLI flag

We should probably split this and/or be more descriptive about this as --mode differs from the idea of --format as described in a writeup. mode essentially is loading a DB of format heuristics (in the example using MIME). format is declaring the format of an individual resource.

1. We need to specify what happens when conflicts occur
   i. Which parts of the CLI are affected, --require comes to mind.
2. We need to specify what scope is affected when the flag is set (process cwd package boundary, resolved file package boundary)
3. Shebang interop considerations.

This is only the first step. It does not include ESM<->CJS interop. It focuses on providing ESM support in Node. Any attempt to mix ESM with CJS or vice-versa will throw an error.

I am opposed to this idea. Some form of interoperability must exist, even if it is purely through an exposed require function somewhere. In particular import() will function in Script from day one by JS spec, so I am unsure what this means since you must support CJS->ESM.

My design goals include resource be unambiguous in nature and anything we ship should behave that way from the start.

Goal: Make ESM work in CJS

transpile ESM to CJS (ex babel, webpack, etc.)

I am entirely opposed to having a transpiler be in core if that is the intention. I think this is solved already and doesn't need to be done in core. Transpilers have bugs and evolve over time. If we land it in core we have to preserve lots of compatibility even if it has bugs.

Extension A - target [undefined] - Conversion paths for CJS -> ESM

I think we should leave this outside scope, just like above with transpilers.

Extension B - target [undefined] - Loader Hooks

These already are designed, but we can look at what needs to change.

Extension C - target [undefined] - Loader Extensions

I think APM/Mocks are requirements to some people /cc @mcollina . Existing designs have ways to do this behavior https://github.com/bmeck/node-apm-loader-example .

Extension D - target [undefined] - npm asset interop

I think we should leave this outside scope, just like above with transpilers.

[evanplaice]

@bmeck Following on from what @MylesBorins it's probably premature to address a roadmap for now.

Specifically, how does the current implementation work, what will be the agreed-upon approach going forward, and what changes need to take place to make it work with minimal changes to Node.

mode:esm can be explicitly set by a CLI flag

The details of this approach haven't been fleshed out and require further discussion.

The current approach, using mimetypes to specify module format breaks browser interop. On the Node side .js implies CJS as the default. In the browser .js implies ESM. I'd go so far as to say this is the primary conflict that led to the formation of this working group. The mode flag is one possible approach.

1. Good point --require will need to be addressed
2. Mode can be detected (ie first require/import statement) in the main entry point

• modes can be explicitly defined as an efficient path to override detectiton
• CLI works the same
• pkg.mode only affects how pkg.main (ex /index.js) is loaded
• package or cwd relative paths are ambiguous because libs may provide both CJS and ESM compatible imports

3. Shebang also needs to be addressed, could it be used with the CLI flag to specify mode

Keep in mind the fundamental differences between CJS and ESM. CJS by default encourages deep linking directly to source files because that's the only way to specify imports.

With ESM, the usage of facades (ie index.js files containing re-exports) to expose a public API make directory structure irrelevant. For example, RXJS is currently (ie 6.0) undergoing a massive restructuring that uses facades because users deep linking to internals has caused a lot of support pain for the maintainers.

This is only the first step. It does not include ESM<->CJS interop. It focuses on providing ESM support in Node. Any attempt to mix ESM with CJS or vice-versa will throw an error.

ESM as it's currently used in the browser is incompatible with how it's used in Node. IMO, addressing this issue is a good starting point; even if the functionality is hidden behind an experimental flag until interop is addressed.

Either way, using require() to import an ESM should throw an error. Using import to import CJS should throw an error.

transpile ESM to CJS (ex babel, webpack, etc.)

Of course this shouldn't be added in core. I just brought this up because lib authors may find it preferable to provide both. In the FE, ecosystem isn't common for lib devs to provide alternative formats (ex /dist/[name].umd.js, /dist/[name].esm.js) and tools already exist to accomplish this.

Extension A - target [undefined] - Conversion paths for CJS -> ESM

I agree, the community should be able to provide a solution. I'll mark this as out of scope.

Extension B - target [undefined] - Loader Hooks

Can't say much about this as I haven't used it. What I can say is, if loader hooks are based on directory boundaries, they will not work with emerging ESM based workflows.

Facades will become a common pattern for libs to expose public APIs. The FE ecosystem is already beginning to shift in this direction. In addition, expose * as syntax will likely land in a future spec providing even greater flexibility for defining facades.

Not to mention the likely introduction of decorators in the future. Decorators will allow functions to be wrapped with additional functionality to address specific concerns and/or define alternative forms of execution for different environments.

I'm not saying loaders aren't useful, just that the current approach to using loaders in CJS isn't compatible with ESM.

Extension D - target [undefined] - npm asset interop

I agree. I'll mark this as out of scope.

[MylesBorins]

Could this be closed in lieu of #23?

[evanplaice]

@MylesBorins I'm good with closing it. I'm really liking the direction that 'goals' is going. If it seems beneficial, another draft can be created to map implementation specifics once we have a better idea of how that'll look.