Support WASIX

[syrusakbary]

Zig currently doesn't support Wasm/WASIX applications.

WASIX is a new standard emerging (as a non-breaking addition) over WASI Preview 1. WASIX adds full support for things like: pthreads, sockets, fork, process spawning and many more things.
See more detailed blogpost for the announcement here: https://wasmer.io/posts/announcing-wasix

And WASIX website: https://wasix.org/

Expected Behavior

Zig supporting WASIX would be huge for the Wasm and Zig communities.

Why?

• It would let using Zig in many platforms (including the browser)
• It would allow having a generic - C compiler that compiles C files (using LLVM under the hood) to any architecture, and runnable from any architecture that supports running Wasm
• It would allow creating http servers (or TCP/UDP ones) to run Zig or C files in the Edge and even the browser

[rohlem]

I'll be taking a look at this.
Anyone interested in collaborating feel free to let me know.

[git-bruh]

Is it correct to assume that the first step should be to integrate https://github.com/wasix-org/wasix-libc into wasi_libc.zig (Translating the wasix-libc Makefile steps), then get C programs (via zig cc) working, and then work on porting the wasm-related zig stdlib code to wasix-libc?

Working on it here https://github.com/git-bruh/zig

[syrusakbary]

Yup, that seems like a reasonable approach @git-bruh .

[rohlem]

@git-bruh That sounds like a reasonable approach.
I've been trying the opposite, to start with manually specifying wasix-libc using the --libc CLI argument with a finished build, however I'm a bit stuck on runtime errors in the startup code of the resulting wasm module.
It's possible that some flags or known workarounds in the wasi_libc.zig logic resolve those issues (or that my wasix build got messed up for other reasons).

[Dispersia]

Question for your last part, wasix can run UDP in the browser? Or is that just worded in a way I am confusing. I thought even with wasm it wasn't possible to use UDP from the browser

[git-bruh]

Got a super hacky PoC working -- had to stub out a bunch of files that don't compile (related to pthreads and processes)

[rohlem]

Alright, I've got something working, but it might not be worthwhile progress yet.

I'm still hoping that wasi_libc.zig solves some of these issues, but it's probably not the only missing piece.

---

zig cc <C source file> --sysroot=...: Fails with undeclared function errors, I might be doing it wrong.

This should be the easiest, because the wasix readme basically says clang is supposed to work / the only supported compiler.
The --sysroot argument is recognized (in that it doesn't lead to the error that unrecognized arguments do), but I keep getting undefined function errors regardless, so I assume Zig keeps choosing the packaged libc instead.

---

zig build-exe <C source file> --libc=....: Works with tweaks / if code doesn't use TLS.

• The current Makefile allows choosing THREAD_MODEL as posix or single,
  however it also contains wildcards and find invocations which always end up including the pthread-dependent files anyway.
  FWICT this Makefile logic needs to be fixed to re-enable single-threaded builds of wasix-libc; seems like that hasn't been tested in a while.
• When compiling with THREAD_MODEL=posix, the resulting executables trigger Wasmer to emit error: RuntimeError: out of bounds memory access:
  • any time they access __wasilibc_pthread_self, which is in ____wasi_init_tp during startup and throughout the library, f.e. in getrandom.
    I assume this might be a linker bug related to TLS, but not sure.
  • also once in __wasilibc_init_ssp. There's no TLS usage here, so I'm not sure what the cause might be.
    Maybe global initialization / CRT constructor calls happen too soon for some reason?

Without much prior knowledge of linkers and WASM I might poke around with binaryen's wasm-dis and wasm-as to see what the issue is.

---

zig build-exe <Zig source file> --libc=...: Fails due to strange lookup failure of symbol main.
libc.a somehow depends on main but doesn't find it. This might have to do with exporting behavior, not sure.
Should probably look for differences between wasix-libc and the bundled wasi-libc, since the latter works.

---

@Dispersia How I understand it is that WASIX is a standardization effort of features beyond WASI, like UDP sockets.
For now (only?) Wasmer supports the standard, but the goal would be for other environments (like Browsers) to also support it in the future.

[git-bruh]

@rohlem Some of the errors at runtime when using threads/processes are probably an upstream thing, they happen for me even when compiling everything using the official wasi-sdk toolchain

[rohlem]

@git-bruh Great progress!
Interesting, from looking at the Makefile it seemed like at some point a single-threaded build without TLS should have worked by just excluding those files.

[git-bruh]

Yeah many parts of the Makefile are broken, like the globs unconditionally include threading-related code even if it's disabled

[ifreund]

Before people put large amounts of effort into implementing this, let's make clear that this is a feature request/proposal which has not (yet?) been accepted by the Zig leadership.

As for my personal take, I see WASIX as an effort to port legacy POSIX APIs (including many with questionable design such as fork()) to WASM. This of course makes it technically easier to port existing programs relying on these legacy POSIX APIs to WASM but I worry about how runtimes could possibly implement this API surface securely while maintaining reasonable performance. It frankly seems quite like "containers, but with extra steps"

Furthermore, It appears to be driven by a single company which is also responsible for the only implementation. I don't any information on the standardization process and can only assume that it is not open. Without any 3rd party implementations including support in browsers, I think adding support to the Zig standard library would be quite premature.

Overall, I do not see how supporting WASIX in upstream Zig aligns with the goals and values of the Zig project.

[rohlem]

@ifreund The way I see it, if WASIX is a functional libc, and Zig allows users to provide a custom libc, then there shouldn't be harm in ironing out bugs (in both projects) to enable that support.

Zig is of course free to decide not to ship with it, and not to mention it as supported target, nor in the official std module.
But a third-party package should ultimately be able to offer all of the same functionality via the Bring-Your-Own-OS mechanism, right?
(Only difference being users have to add the corresponding entry to their projects' build.zon.)

[syrusakbary]

Furthermore, It appears to be driven by a single company which is also responsible for the only implementation. I don't any information on the standardization process and can only assume that it is not open

You may have missed this, posting here in case is useful: https://wasix.org/docs/community-and-governance

Note: WASIX governance is completely open up for any developer and/or company that wants to contribute or collaborate (including yourself, if you feel that being part of WASIX could be useful for you!) :)

Without any 3rd party implementations including support in browsers

WASIX programs can fully run in the browser (as reflected in the wasix.org homepage).
If you want to see a very cool demo, please visit: https://wasmer.io/ and type wasmer run python! 🚀

I do not see how supporting WASIX in upstream Zig aligns with the goals and values of the Zig project

I hope you will eventually be able to see the long-term value that WASIX could provide to Zig, and hopefully, the work done here by some of the contributors may help to showcase the value of WASIX to the Zig community.

Here are a few examples that would be enabled: being able to run Zig compiler anywhere (without restrictions, including Zig+LLVM zig cc running on the browser), or using Zig a package in other ecosystems (including JS!), allowing any Zig program runnable anywhere and so on...

As previously commented, the reward will be awarded to the developer who fulfills the requirements laid out on the same thread.
We would really love if Zig adopted WASIX upstream (I believe we will have very cool demos to showcase why this may be useful for others!), but merging upstream is not a requirement for awarding the $5,000 sponsorship.

Keep up the good work!

[emidoots]

Problem statement

Overall I think the problem statement (or lack of) is valid: WASI as it stands today is quite weak, there's a lot it can't do, and that is harming WASM adoption overall. I've heard this from my younger brother and others who work professionally using WASM at various CDNs using it for edge functions. It's a real problem, and needs a solution.

Solutions

There are three solutions I see:

1. Wait for WASI / the Bytecode Alliance to propose more WASI semantics. So far this has been a.. rather slow, non-existent even, process? But 5 days ago (I wonder if that's a coincidence?), they began work on WASI preview 2 - so although I have no clue what they're going to do with it, the lifeless corpse is twitching a bit?
2. Adopt WASIX. Putting aside the controversy that seems to follow Wasmer, I just don't know that a lot of languages/ecosystems are actually going to adopt this - and WASM is nothing if not 'somewhat' standardized. I'm sure Wasmer is aware of this, which is in part why they are trying to push this proposal so hard.
3. Create our own WASI-like specification, or better see if Bytecode Alliance folks would be open to figuring this out. Zig could actually be a good, strong enough WASM community to propose this sort of thing - and an unbiased enough entity to make it have some teeth - but I imagine this won't happen for the same reason file not found on Ubuntu #1 is the case today: few people have time/interest to do this work. I imagine this would look like 'solve the same problem WASIX aims to, but not by just exposing syscalls to WASM that are hard to secure / problematic, rather by figuring out the proper WASM APIs for those and writing a spec - experiment and iterate on it, prove it out, and make a proposal'.. a slow, arduous process.

Overall I think the situation is 'unfortunate'. It seems unlikely that WASI will improve soon, even more unlikely that WASIX will get any broad adoption by other WASM communities, and it seems unlikely to me 'our own WASI-like specification' would be good long term either (but who knows.)

It may be best to just wait.

Proposal

I expect Zig proposals to follow more of what I wrote above: problem statements, possible solutions. If a company wants to sponsor development of an issue, that's great and reasonable in my view - but the way it was done here was pretty damn poor.

I also see your company using it as a marketing opportunity on Twitter and Reddit[0][1] - I'm not against corporate sponsorship of development, and also not against companies or people publicizing they're sponsoring development. Heck, I've done this. But it feels a bit wrong and distasteful given the proposal is to implement your specification, it wasn't even discussed first, and is structured as some sort of bum rush to implement it and get $5k, no?

You could have made the proposal, mentioned sponsorship being something you are interested in, see what the community said - and then later on publicize the great work a community member did for you after the fact. That would've felt a lot more like 'company works with community to move software forward' rather than 'company pays bounty in hopes of pushing something through and/or getting a marketing opportunity'

I'm not here to call you out for your/Wasmer's iffy behavior as I've done over the years(1) but rather just to let you know this is a really poor way to propose a change to an open source community.

[tschneidereit]

@slimsag

1. Wait for WASI / the Bytecode Alliance to propose more WASI semantics. So far this has been a.. rather slow, non-existent even, process? But 5 days ago (I wonder if that's a coincidence?), they began work on WASI preview 2 - so although I have no clue what they're going to do with it, the lifeless corpse is twitching a bit?

Just as a quick heads-up from within the twitching corpse: WASI Preview2 has been in the works for a long time, is largely done at this point, and a perfectly reasonable thing to target right now.

See this blog post on the Preview 2 roadmap for more details.

[andrewrk]

Alright. @kristoff-it and I took a break from working on our other tasks today to address this issue, and in fact we wrote a blog post about it.

As a reminder, the Zig issue tracker is not a platform for free speech, for discussing bug bounties, for discussing other people's reputations, or for archiving Internet history. It is only, exclusively, and for no other purpose than for working on the Zig compiler and related tools. Any content that is not doing that will be removed, and any users who I subjectively predict will create more of such content in the future will be blocked.

Let us have no more discussion of anything in this topic other than WASIX and its technical merits, or lackthereof.

[daddycocoaman]

Random internet person here.

Personally, I would love if Zig (or some additional C compatible languages like Nim or V) supported WASI/WASIX directly.

As primarily a Python dev, I know of an entire dev community that would benefit from having an accessible low level language like Zig with WASI implementations. But I also get the points made about WASM/WASI standards vs how WASIX comes across.

Regardless of how the bounty stuff came across, I hope y'all implement it. WASI would be a great pivot for me to use Zig. 🙏🏿

[githubuser6000]

@daddycocoaman Unless I'm mistaken, there is (some?) support already for WASI

[squeek502]

@daddycocoaman wasm32-wasi is a Tier 1 supported target and has been since Zig 0.8.0.

[daddycocoaman]

Sorry, maybe I wasn't specific enough.

I know WASI is a target but I kinda assumed that the inclusion of WASIX here would also imply that Zig would gain a first-class runtime library as well (likely Wasmer in this case but any runtime, really). Unless I missed that one somehow. 👀

[aazam-gh]

I'd like to work on this :)

[bitnom]

The Consortium For Advanced Super-Friends wants full sockets and multithreading in wasm

[gkgoat1]

One benefit, which may be useful for Zig, of WASIX is that it does not require Rust code within the compilation process. Contrasted with WASI 0.2, which requires a compiler in Rust to 1. generate component type objects and 2. create and assemble the component. Until Zig can support the component model format, WASIX may be an attractive option for use cases involving sockets and external command execution (though that is not the best way of composing programs, nor is WASIX extensible to non-POSIX apis).