Terminology: Modules and Resources

[jkrems]

As part of the chat between @bmeck and myself around how the main module map, hooks, and vm.SourceTextModule should interact, we agreed that it would make sense to split the concept of "loading" and "loaders" into generic resource loading and the module system as a user of resource loading. The result were three separate concerns which are split between the module- and the resource system:

1. locate: Turning a relative reference and a context into an absolute location.
2. retrieve: Retrieving a resource (content and associated meta data) from an absolute location.
3. Interpretation of a resource as a module.

Only the last item would be the job of the module system. The advantage is that modules could use the resource system to load additional data and reuse all the infrastructure. It would also allow writing web-compatible code that includes data files.

@bmeck did a write-up of a potential design for a resource API for node, including suggested hooks: https://docs.google.com/document/d/12-xJcMHH6r0QQG4fprQvHG1OO-OpZ-fSdxDIT8eKqKk/edit

The idea is to end up with a system where customizations (hooks) could be written in a way that can be fully isolated from application code, be it in a separate thread in node or in a service worker in the browser. And in most cases it should be possible to design it to have the same plugins/hooks in both environments.

[GeoffreyBooth]

Would “retrieve” include finding and returning the nearest package.json if we implement the file specifier resolution proposal?

[jkrems]

It might if meta data like Content-Type depends on the package boundary for file URLs.

[bmeck]

I'd note that a variety of things could be cached or even be done lazily in most scenarios regarding meta-data. Also, a lot of meta-data is not trustworthy to begin with in certain scenarios.

[SMotaal]

@jkrems I was trying to work out this divide in my head while editing the opening statement of the file specifier proposal and felt uneasy with the particular case where retrieving the package.json is potentially a locating chore.

This is I think why I was inclined to add what I later referred to as the "constancy expectation" which I am further refining here to see if I could reach clarity for making it a PR.

So maybe what we needed is to clarify which (if any) package.json related operations fall under locating. I feel inclined to assume that locating import … from 'the-package' will always read the-package/package.json in order to provide "an absolute location". Other than this case, clarity here maybe important for others for different reasons too.

[jkrems]

retrieving the package.json is potentially a locating chore.

Just to clarify: You use "retrieving" here as a generic verb, not in the sense of retrieve defined above I assume..?

If that's the case: In my proposed algorithm, nothing would locate quenchmark/package.json. FIND_PACKAGE_BOUNDARY(packageSpecifier, parentURL) uses absolute URLs, not package names: https://github.com/nodejs/ecmascript-modules/pull/14/files. So the only thing being located is "./package.json" relative to various absolute URLs (the package boundary candidates). It's definitely on my todo to rewrite that resolution algorithm change from a "resource loader" perspective but I think it would mostly stay the same.

[SMotaal]

Sorry for struggling to catch up with the updates, but I will try to clarify where I feel there could be concerns.

When FIND_PACKAGE_BOUNDARY hits the package.json for that boundary, subsequent operations even if independent until all the modules in the current graph are resolved (and loaded) are designed around the premise that this boundary relates to the existence of this instance of the package.json.

I am not sure how we are planning on handling out-of-sync links mid-graph, but decoupling the functions may still be in need of a shared state to address concerns like this (not necessarily this one).

Separately, can you please elaborate in a comment in the gist a little more about quenchmarks/package.json (maybe I got it wrong) to help me hash out the details if we decide it’s worth it to PR vs delete that concern altogether.

[jkrems]

I am not sure how we are planning on handling out-of-sync links mid-graph

Is the concern about a signifier file being removed or modified during the algorithm? I'd be happy to call the behavior "undefined" with the assumption that most implementations would cache package.json data per URL on first attempt and most users won't run in a situation where the contents fundamentally change during resolution. This seems in line with most existing implementations. Calling it "undefined" is to still allow for a system like a REPL to chose to do some hacky thing to break the cache.

Separately, can you please elaborate in a comment in the gist a little more about

I'm not sure what kind of clarification you're looking for. Also, as a general note: Commenting on gists is always pretty awkward because of the lack of notifications so I'm unlikely to respond to follow-ups. But happy to leave a comment once I understand what kind of information you're looking for! :)

[SMotaal]

@jkrems sorry for the delay but I am trying to find a way to better frame my underlying concern which is more challenging for me than expected.

[MylesBorins]

Can this be closed?

[MylesBorins]

Closing, please reopen if necessary.