Improve code loading and modules section

[lukewagner]

This change seeks to clarify the wording in the 'code loading and imports' section. Remember, this is for plain v.1 imports (which are ES6-symmetric) not v>1 dynamic loading, which is still under discussion.

[kripken]

Is this saying that builtin functions are only called via function pointers?

[lukewagner]

No, the "builtin" referenced above is the builtin you call to load a module (i.e., eval). It is the functions imported from the dynamically-loaded module that are only callable via func-ptr.

[kripken]

Got it, thanks.

Probably just my own miscomprehension, but it could be avoided by changing "A builtin function (implemented by the host environment) is provided to load modules at runtime" to e.g. "Modules can be loaded at runtime by a provided builtin function (implemented by the host environment)."

(The current text of "A builtin function is" appeared to mean "Let's talk about builtin functions: A builtin function is something that is ...".)

[kripken]

Ah, I think I see why I misunderstood things that way: the previous bullet at that level is "A module is loaded from a sequence of bytes", which is indeed elaborating about modules. So I thought that "A builtin function is" was doing the same.

[lukewagner]

Good suggestion, will do

[jfbastien]

I'd really like a description of how this is different from dynamic linking, and what dynamic linking will do that module load can't. The way I think about it is that module load is ~ forking a new process, and communicating with it through indirect calls (postMessage under the hood?) is ~ IPC. Is that a fair description? I find the current one somewhat confusing.

[lukewagner]

Not quite: ES6 module imports allow synchronous calls between modules, no postMessage/IPC necessary (that would involve workers, which ES6 modules do not). The fundamental differences are sharing heap, globals and function pointers (which is what I mentioned in the last bullet). I think the right mental model here is: ES6 modules, except the code isn't JS.

[kripken]

"can be loaded at runtime" - by whom? Is this talking about the host environment (JS on the web) loading modules, or modules loading other modules?

[jfbastien]

@lukewagner "ES6 modules, except the code isn't JS" doesn't really help me understand what we would do for a non-browser wasm shell. Multiple heaps in the same process? We can implement sandboxing, or rely on implementation-defined behavior and let cross-heap accesses work. I'd really like this section to be understandable without knowing about ES6 modules.

[lukewagner]

Ok, updated to clarify @kripken's and @jfbastien's questions.

@jfbastien: outside a Web context, yes, v.1 imports imply separate heaps. This makes sense in the general module system of the browser/node.js (heap is an internal detail of a module, along with JS vs. WebAssembly), but I expect that a minimal shell wouldn't have any use for imports until we added dynamic linking support.

[kripken]

I am puzzled by this. What is the use case for wasm loading another module dynamically, as opposed to statically? And, since this is a v1 feature, how would we polyfill it?

[lukewagner]

The impl scheme I was imagining was that, if this feature was used, all indirect calls would be translated into:
(ptr > K ? ffi(ptr, args) : tbl[ptr&M](args))
that is, to dynamically create a function pointer, you'd use an out-of-range high value that would then call out of asm.js into an FFI where you could lookup the right callee in a table.

You're right that there's not a strong use case and initially removed this. What caused me to add it back was I couldn't think of a good alternative for how else one would implement the dynamic feature-test-based switch described at the end of this section which would allow a dev to run the same .wasm on a browser and non-browser. But explaining this now, it does seem like it's better just to say "wait for true dynamic linking, use #ifdef for now and ship separate builds". I'll remove.

Anyhow, answering you now I'm thinking that for v.1, #ifdef would be good enough

[lukewagner]

(Updated)

[kripken]

Not an objection, but I'm curious: what is the use case for modules loading other modules (by URL) in v.1? This is a significant boost over what asm.js modules can do, I'm wondering what use you had in mind for this capability.

[lukewagner]

The "by URL" part just falls out from how modules are defined: you import dependent modules by URL, not by passing them in by value. Other than the URL bit, this is just the asm.js stdlib/ffi feature.

[kripken]

I see. And is the URL bit because of consistency with ES6 modules? Or something else? I feel like I still don't understand the new element here.

[lukewagner]

Yeah, in ES6, you write import foo from "bar" (http://jsmodules.io/). The module loader pipeline has a lot of control for how "bar" gets interpreted, but by default it just does a normal relative fetch. Also, modules are, by default, loaded once and shared (for a given canonicalized URL), so if you want to "prepare" module A before it is imported by module B, you can import A directly, mess with it, and then load module B (which will then import the same instance of A).

[kripken]

Got it, thanks.

[jfbastien]

I don't agree that we should go forward with module loading in V1 as I understand it right now. It sounds like in-process processes, but not quite. The differences seem to come from how things would work in-browser and how ES6 modules work, which are details I strongly think we want to avoid when implementing wasm. It doesn't sound like module loading would work at all in a non-browser context, which IMO makes this a no-go.

Another metric I'd go by is: once we have dynamic linking will module loading be redundant? It sounds like many heaps in the same process (and sync IPC) are the main advantage?

@lukewagner am I misunderstanding what you intend with this section? At the minimum my disagreement means it needs clarification!

I'd like @titzer / @davidsehr / @dannoc to chime in on this.

[lukewagner]

Ah, I think the source of disagreement here is that I put too much in the general bullet points that instead belongs under the "in a Web environment" (which could be generalized to include node.js to say "in an ES6 module environment").

So at a basic level, WebAssembly modules need to be able to call "out", and "module importing" provides a single unified way to do that: you name what module you want to import, what functions of that module you want to call and with what signatures. Past that, though, what happens when you call an imported function should be a black box, from a spec POV. Maybe it's running JS, maybe shell, maybe C++, maybe other WebAssembly. What is on the other side of that import, what may be imported, how names or mapped to what gets imported -- none of this is part of the spec. Thus, the spec wouldn't mandate WebAssembly modules importing other WebAssembly modules -- that would just be something that falls out of the ES6 module loader design -- and, e.g., a minimal WebAssembly embedding could choose to only support importing from a fixed set of builtin modules.

Does that sound reasonable @jfbastien?

[jfbastien]

Ah I see what you mean, @lukewagner. That makes more sense to me: this is only discussing how extern and __attribute__ ((visibility ("default"))) get exported by a wasm module. What an embedder of wasm does with this isn't specified.

Agreed that makes sense, as otherwise a wasm module would be able to do anything.

It does still overlap with dynamic linking, but only in a fairly limited sense. Any interactions with the embedder would originally be ad-hoc through this approach, and once we have dynamic linking we can export regular functionality through a proper dynamic library.

Do I have this right?

I think there's some rephrasing needed!

[lukewagner]

@jfbastien Yes, exactly right, and thanks for helping to push towards clarity. Will rephrase.

[lukewagner]

Basically rewrote it to better capture this discussion. I also moved it to the top of V1.md b/c it logically precedes the following sections which go into more details about what module is.

[jfbastien]

I like PNaCl's approach of only defining _start as an export: it's up to the developer's code (usually provided by the toolchain) to go through the .init_array (which PNaCl merges with the C++ global static ctors). The approach is very extensible-web-y in that it's the minimum feature set and lets developers to everything.

[lukewagner]

Are you specifically talking about initialization or saying that there only be a single export. If the former, I'm missing how that's different than what the bullet says (s/init/_start/). If the latter, then we hurt the use case where a wasm module is written to be used as an ES6 module from JS.

[lukewagner]

Doh, forgot to merge this.