Cross-language module sharing via distributed web assembly package management registry and tool

[runvnc]

Basically the idea is take something like a package registry/management system like npm or CPAN. To me the semantic versioning and functionality of npm is in the lead. These systems are more used during development, not run-time distribution, although it is worthwhile at least briefly considering the possibility of intersection of run-time/production/web page distribution via a similar mechanism, if that does not confuse the issue too much.

But the advantage would be the ability of web assembly modules to work across different programming languages rather than the normal situation with many programming modules available from the community but limited to the community of the specific language of current interest.

My thought is rather than being server-based, or based on some individual company maintaining a CDN as in the case of npm, skip directly to a distributed peer-to-peer system with a content-/data- oriented network architecture in mind from the start.

It may be that some aspect of web assembly makes this seem impractical. If so I am sorry I am not familiar enough to address that specifically ahead of time, but my guess is that those issues may be addressed.

[jfbastien]

I wonder if the details in this file address your concerns:
https://github.com/WebAssembly/design/blob/master/DynamicLinking.md

We certainly don't want to impose a distribution model, but we do want to make what you say possible. It's not an explicit goal of WebAssembly to implement this, at least for now, merely to make it possible and secure.

[runvnc]

That does address some of it. My use case is not necessarily dynamic linking, but just using an external module from a registry/other language during development, like somehow loading it as an 'import' in my normal programming flow.

So from this it says one ABI per language. There is no way to have a common ABI across languages?

Also, you don't want to prematurely lock people into a distribution model, but ideally for some type of package registry, there would be one adequate protocol that people can build clients against, rather than many (we are lucky not to have 5 different CPANs and npms).

Also, does this allow for anything like 'semantic' versioning? In that there is any differentiation at all between versioning for patch, minor, or major breaking changes in terms of compatibility?

[dschuff]

wrt ABIs, I imagine you could have something akin to the C sysv ABI, where there is effectively a common ABI which is also used by C, and then other languages either use a superset (e.g. C++) or use their own ABIs but support the common one for FFI-style use.
Versioning is also probably something that's like ABIs in that it's outside the scope of what wasm will define, but could be done by users in more than one way.

[runvnc]

Perhaps there is an opportunity to design this in such as way as to be multi-language from the start rather than defaulting to support only either C or C++. And maybe it would be good to be able to differentiate between versions that necessarily will break many things versus those that are bug fixes and will not necessarily break many things. (Practically, you cannot be sure, but it is useful to have at least two categories).

[sunfishcode]

ABIs are largely defined in terms of type systems, and because there are vast and fundamental differences between the type systems of popular languages, a single ABI for all languages would be either too limited or too disjointed.

The suggestion to create a new package registry is unfortunately not specific enough to be actionable here. In part, this is because package registries tend to exist at the source language level, such as Rust's cargo or Node's npm or similar, rather than at the compiled code level. The differences between ABIs of different languages is part of this, and differences between tools for different languages is another part.

Also, a "distributed peer-to-peer system with a content-/data- oriented network architecture" is too vague for us to work with. What we'd need here to make something like this happen is for someone to step forward with a clear plan for how this should work, how it can scale, and how it can be secure. With such a plan, we could evaluate whether it makes sense to pursue it within the official WebAssembly group here or whether it should be an independent project.

[runvnc]

Obviously its a very different and difficult idea for a number of reasons.
I don't expect most people to really be able to appreciate it -- web
assembly is a big enough idea on its own, and new and challenging ideas
don't succeed on the basis of merit. Ideas succeed because they are
similar to previous ideas (as web assembly is similar to previous ideas
like CLR etc.) and because people follow authority (Brendan Eich etc.
pushing it).

But I think that whether it is actionable for you now or easy to envision
or not, it is very much something that aligns with the web assembly
project. If you look at npm, there are a number of languages compiled to
JavaScript that have modules in there which one can access with any
language which compiles to JavaScript.

https://github.com/jashkenas/coffeescript/wiki/list-of-languages-that-compile-to-JS

And the idea of a peer-to-peer distribution for packages is also a
no-brainer, especially for anyone who followed the scaling issues npm had.

Package management, versioning, and how all of that works is absolutely
related to web assembly and something that should be planned for, even if
it (obviously) isn't a core part of the project now. If you want web
assembly to succeed then these ideas will be part of your planning.

But I would say to make it easy just leave it closed and wait until one of
your authority figures says it is an idea, and then it can be an idea for
you.

On Mon, Sep 14, 2015 at 7:33 PM, Dan Gohman [email protected]
wrote:

Closed #320 #320.

—
Reply to this email directly or view it on GitHub
#320 (comment).

[jfbastien]

At this point in the project we want to build the base technology that will enable others to build what you propose. Dynamic linking will be key to this, and so is security. I suggest looking at #53 for a few ideas that should align well with yours.

It's not that what you're saying isn't important! Rather, we don't pretend that we know The Right Way for everything, we instead want to build the basic capabilities, and let others take the lead. At some point standards will emerge, and we can adopt them without forcing everyone to do distributed peer-to-peer.

As @sunfishcode points out MVP is very far away from this right now.

[lostdj]

@runvnc take a look at NixOS / Nix package manager. It is essentially distributed (the package tree is in git); works in many environments; provides multiversioned, build reproducible, customizable applications and dev libs for variety of languages; does not depend on a single server to download prebuilt packages.

It's not a rocket science.

What's harder is lang interop, like it was already said here. And I see two ways how this can be accomplished: simple C ABI, or complex CLI-like ABI, with objects, generics and abstract classes. Neither will be perfect and satisfy everyone. I think we should wait for implementation of DOM API and GC first and see how this is going to work.