A JS-style Text Format

[sunfishcode]

This proposes an official text format for WebAssembly, aimed at browsers to use in "View Source", debugging, and related tooling. It uses a JavaScript-like syntax for readability and familiarity on the Web, though it differs from JS in several respects, as it aims to reflect the underlying WebAssembly language.

This proposal is meant to serve as a beginning. We'd like to establish this as a concrete place to start, so that we can then iterate, as we did with BinaryFormat.md.

You can try out a prototype of this proposal yourself in Firefox Nightly, for example by playing the AngryBots demo with the debugger open and examining the wasm file in the debugger.

Here's a screenshot of it in action.

A more complete description of the grammar and a parser implementation are available here. If this proposal is accepted, we'd like to move this repository under the WebAssembly GitHub organization to serve as the interim spec for the text format during the initial discussion.

[naturaltransformation]

Is this statement still true given that function names may have to be converted from unrestricted byte sequences?

[sunfishcode]

Yes. Arbitrary bytes are escaped so that they can be represented as valid unicode characters. The details are in the full grammar here.

[ghost]

Just to make sure I understand, this $end label creates a block that starts just before the $x = 1?

If this block started after $x = 1 then would it be presented with explicit curly-braces?

[sunfishcode]

This turned out to be a mistake. In the fac.wast file this file is translated from, the block starts after the set_local. I've now updated the example to accurately reflect where the block starts.

[binji]

Having the label at the end is more sane, I agree. Any concern that it is differing from the equivalent JavaScript?

a: {
  break a;
}

[sunfishcode]

Yes, there is a concern. Accompanying it is the concern that large and deeply nested block structures, with extensive use of labels, are much more common in wasm than they are in typical JS. Under such conditions, having the label "where the branch goes" improves readability because one doesn't have to jump far away to find the top of the loopblock and then jump back down to find the bottom.

[binji]

Yeah, I think you're right, it's better to have labels where they go.

Though I just thought of another concern; shadowing a label name becomes a bit harder to detect because you have to look for the closest nested in either direction, e.g.

{
  loop $a {
    ...
    br $a;
  }
$a:
  ...
}

But one could argue that this is like going to the doctor and saying "it hurts when I do this"... :)

[sunfishcode]

Indeed; "don't do that" ;-). Alternatively, one might imagine prohibiting shadowing loop labels with block labels or vice versa. My guess is that we can address this at a higher level.

[distransient]

Could ^ as xor cause confusion?

[yurydelendik]

Is there a reason why it would cause? (Assuming that majority of people familiar with C or JavaScript)

There is always an options to leave additional alternative for infix operators in form of function call similar to e.g. i32.min, so xor operator will have i32.xor in addition to ^.

[distransient]

Js users will be familiar with it as the exponentiation operator

[yurydelendik]

Js users will be familiar with it as the exponentiation operator

'^' defined in JS as bitwise-xor (see http://www.ecma-international.org/ecma-262/6.0/#sec-binary-bitwise-operators)

[distransient]

Sorry, my brain's in the wrong gear

[ghost]

@drom I guess it means a display like an assembler code, one operator per line, etc.

I strongly expect that this community group expects an expression based language, that they might accept display as a linear list of opcodes for the MVP but only so long as there is a clear plan and a well articulated expression base text format and a proof-of-concept implementation would make this clear. People might be prepared to defer largely bike-shedding text format issues until after the MVP, but they know the difference between a linear stack machine code and an expression base language.

I don't think the core issues here can be deferred until after the MVP rather there a many issues that constrain the language design and are core issues.

[ghost]

If there are pop's in the language then it is not expression based. Typical expression blocks discard excess values of statements, so a special operator is needed to build values.

block
i32.const 4
f64.const 5.5
call $foo
i64.add
end

{
  const (i64 $c0, i64 $c1) = $foo(4, 5.5d0);
  $c0 + $c1; // but does not consume the stack values, might be used again
}

// So perhaps this:
{
  // Builds arguments from all the values of argument operators.
  // But doesn't the above read much better?
  call_with_values(i64.add, $foo(4, 5.5d0)).
}

block
i32.const 4
f64.const 5.5
get_local 0
br_if[arity=2] 0
i32.const 6
f64.const 7.5
end

{
  br_if $b0, values(4, 5.5d0), $l0
  values(6, 7.5)
} $b0

I don't think these examples demonstrate the real problems, but based on these I assume that this stack code allows operators that return no values to appear anywhere, so it removes that constraint. This would at least allow values to be stored in local variables if they need to be consumed in a pattern that does not fit the stack order.

I think drop is ok in an expression language. Assuming presentation blocks are inserted before void operators not at the head of a block:

i32.const 1
i32.const 2
drop
i32.const 3
i32.add

{
  1 + {drop(2); 3};
}

i32.const 1
i32.const 2
i32.const 3
drop
drop
i32.const 4
i32.add

{
  1 + {drop(2, 3); 4};
}

The problem occurs when values are consumed out of stack order:

{
  const (i32 $c0, i32 c1) = call ...;
  $c1 / $c0;
}

or

{
  const (i32 $c0, i32 c1, i32 c2) = call ...;
  $c2 / $c0;
}

For this out-of-order consumption we need get_value, and then need an efficient way to dispose of the values up the stack, need a way to inform the decoder on their last use get_and_zero_value, so are back to the old block semantics.

Can people think of any other good examples that demonstrate stack machine code that will not present well in an expression based language?

[ghost]

@drom Thank you, there are some ideas from stack machines such a Fourth that I have pondered. For example the Fourth pick operation I have been trying to promote as get_value.

There might be a number of ways to shuffle stack elements around to consume them in the order required with Fourth style operators, compared to simply being able to encode access to the desired element.

Being able to swap an element up the stack to the TOS and then drop it would provide a method to discard values up the stack. I have been promoting get_and_zero_local to gain a similar ability to inform the compiler that a value is discarded in an expressionless encoding.

I think there was a point in wasm development, with the AST, in which it was not necessary to compare the stack elements at control flow merge points, rather just the local variables. If we allow swap and other changes to the stack then this might increase the decoding burden.

We seem to have a good alternative in the prior approach, of just building up values within blocks on the values stack, and not allowing them to be changed. They could be referenced by index, rather than shuffling them around. Downside is a deeper decode-time type stack, but perhaps it would be worth it.

In any case we don't yet have swap and pick etc, so the current proposal just does not look tenable it just does not allow access to the values on the stack, rather seems to force definitions into local variables. If wasm is going to be so dependent on local variables then why not just simplify the decoder a lot and remove the stack, make it a register base code.

[qwertie]

@JSStats I'm surprised you moved your comments here, as it was my impression that you were not talking mainly about the text format, but about the more fundamental idea (applicable to the binary format) of whether Wasm should stick with a more traditional AST. It certainly affects the text format, of course, but as I was saying in #697, it doesn't seem to require fundamental changes in the text syntax.

the current proposal just does not look tenable it just does not allow access to the values on the stack, rather seems to force definitions into local variables

I'm sure the backers of the new stack machine (whoever they are) think it's "tenable", and so do I. I imagine they will add a new operator or two, if data shows that such operators would provide a large benefit. But I'll bet avoiding locals won't be found important enough for MVP, even if multi-value support were added, which itself seems unlikely. That said, I personally prefer an expression-based language and, if I had the opportunity, would have loved to look for alternate ways of achieving similar efficiency gains in the traditional AST regime.

[drom]

@JSStats Forth (not Fourth) language. Has a lot to offer to stack machine designers. In fact it influenced most of Virtual stack machines like Java Bytecode and .NET CIL. The syntax is a bit unusual, but the concept of stack manipulation words: drop, dup, swap, over ... is shown it's efficiency over the years.

Also, as @flagxor pointed out here: #261 (comment) Forth has very powerful concept structural control flow operations.

[drom]

Here is basic comparison table https://github.com/drom/wast-forth/blob/master/stack-manipulation.csv
of "Stack Manipulation" operations in FCode, Java Bytecode, and CIL

[ghost]

@drom I think wasm is still ok wrt 'structural control flow operations'. Perhaps it could do more to support loop analysis, to make the stride explicit etc. But what about structured local variables, these are important in languages too?

Your list ignores the pick operation, to be able to pick a value up the stack. The drop operator also frustrates structured local variables as it ends the scope of a variable at a point that is not structured in a block, and combined with swap completely breaks block scoping of variables. I just don't buy the claimed benefits of the stack-machine, they are not necessary for multiple value support, and being able to present structured code was a worthy goal for wasm in my opinion.

It's interesting that CIL is restricted to pop and dup and I wonder if these are for performance reasons, to make validation efficient. If so then perhaps wasm would hit the same barriers and not be able to introduce pick/swap in future, and if these are essential for good multiple value support in this design then the change in direction to the stack machine was never justified!

Can someone answer if wasm decoders will be required to record and compare the values stack for changes at every control flow path merge point, and what is the cost? In the expressionless encoding experiment this was a show stopper.

There will still remains the possibility of canonicalizing on a different code, to consider the stack-machine code as just a compressed encoding of the canonical code, and to present the canonical code in a structured manner. But then it is not possible in general to specify the encoding in the canonical code, just as it is not expected that the choices of encoding that a compressor uses be expressed in the source input to the compressor. e.g. use a functional form of SSA as the wasm language that developers actually work with.

[drom]

@JSStats I have added pick operation to the table. In fact it is my favorite "stack manipulation" pokemon. I have even done RTL for the "PICKy: stack machine" with pick support: http://goo.gl/DsvhL . You can do so many interesting tricks with pick, you can significantly reduce number of local variables, and number of get_local / set_local operations. Also, defectively dup = 0 pick and over = 1 pick. It gives you sort of "immutable" access deep into the value stack.

But: you have to limit pick power. Giving it only "constant-depth" access. Variable depth pick is call for disaster and can't be validated statically. With all that power only Forth/FCode make use of it. Both JVM and CIL not using it's power.

[ghost]

@drom Thank you, and good to hear. Here's an example that I am not sure how to handle?

block
i32.const 1  // let this be stack slot 0
i32.const 2  // let this be stack slot 1
pick 0
pick 1
i32.add
pick 0
pick 1
i32.mul
i32.add
// Gets messy here, need to keep the result but drop slots 0 and 1??
????
end

In structured code, with the prior block semantics, the unused definitions are just discarded, so this could have been presented in a structured format e.g.

{
 const $c0 = 1;
 const $c1 = 2;
 $c0 + $c1 + $c0 * $c1
}

Adding pick aka get_value alone might reduce the burden of checking the stack at each control flow merge point, as it does not mutate the stack. It would have allow better multiple-value support as the multiple values could just be pushed on the stack and still accessed. It would allow the function arguments to be on the stack for the callee rather than in local variables.

[drom]

@JSStats let me annotate your first example:

               [ stack notation ]
i32.const 1    [ c1 ]
i32.const 2    [ c1 c2 ]
pick 0         [ c1 c2 c2 ]
pick 1         [ c1 c2 c2 c2 ]
i32.add        [ c1 c2 (c2+c2) ]
pick 0         [ c1 c2 (c2+c2) (c2+c2) ]
pick 1         [ c1 c2 (c2+c2) (c2+c2) (c2+c2) ]
i32.mul        [ c1 c2 (c2+c2) ((c2+c2)*(c2+c2)) ]
i32.add        [ c1 c2 ((c2+c2)+((c2+c2)*(c2+c2))) ]
// Gets messy here, need to keep the result but drop slots 0 and 1??
????

why you were caring c1? and unused copy of c2?

[drom]

@JSStats your second example:

{
 const $c0 = 1;
 const $c1 = 2;
 $c0 + $c1 + $c0 * $c1
}

assuming correct operation precedence:

in stack machine:

               [ stack notation ]
i32.const 1    [ c0 ]
i32.const 2    [ c0 c1 ]
pick 1         [ c0 c1 c0 ]
pick 1         [ c0 c1 c0 c1 ]
i32.mul        [ c0 c1 (c0*c1) ]
i32.add        [ c0 (c1+(c0*c1)) ]
i32.add        [ (c0+(c1+(c0*c1))) ]

In this particular case you don't really need to drop data, just use it.

[ghost]

Tried to give $c0 and $c1 absolute indexes for the example. With relative offsets:

block
i32.const 1  [ c1 ]
i32.const 2  [ c1 c2 ]
pick 1       [ c1 c2 c1 ]
pick 1       [ c1 c2 c1 c2 ]
i32.add      [ c1 c2 (c1+c2) ]
pick 2       [ c1 c2 (c1+c2) c1 ]
pick 2       [ c1 c2 (c1+c2) c1 c2 ]
i32.mul      [ c1 c2 (c1+c2) (c1 * c2) ]
i32.add      [ c1 c2 ((c1+c2) + (c1 * c2)) ]
// Gets messy here, need to keep the result but drop slots 0 and 1??
????
end

So the challenge is how to keep it structured, while informing the decoder that $c0 and $c1 are no longer used (or even better at their last use), and can both of these constrains be met?? I guess swap could be used to consume $c0 and $c1 at the i32.mul rather than duplicating them, but where would that leave the code structure? Can we have our cake and eat it too?

[ghost]

@drom Interesting example, but it did require re-ordering the operations which broken the intent of the example. Lets assume that the i32.add and i32.mul are actually function calls and that they need to be called in the order given in the example.

[drom]

@JSStats I understand the challenge. Here are 3 popular Forth solutions:

1. Good stack scheduling / ordering algorithm to relax for the need to go deep into stack. Like my answer to your second example performs the same computations just in different order.
2. the chess match above ends with: swap drop swap drop --> checkmate#. It is often works when function returns only one result. For this specific case Forth use nip operation. : nip swap drop ;
3. If you plan storing this result in the local variable here is solution: (set_local $foo) drop drop.

[drom]

@JSStats compiling following ASM.JS into WASM using binaryen:

test1

asm.js

function test1 (c1, c2) {
    c1 = c1 |0;
    c2 = c2 |0;
    return ((c1 + c2) + (c1 * c2));
}

wasm (11 byte):

(func $test1 (param $0 i32) (param $1 i32) (result i32)
    (i32.add
      (i32.add
        (get_local $0)
        (get_local $1)
      )
      (i32.mul
        (get_local $0)
        (get_local $1)
      )
    )
  )

stack machine (9 byte)

(func $test1
pick 1       [ c1 c2 c1 ]
pick 1       [ c1 c2 c1 c2 ]
i32.add      [ c1 c2 (c1+c2) ]
pick 2       [ c1 c2 (c1+c2) c1 ]
pick 2       [ c1 c2 (c1+c2) c1 c2 ]
i32.mul      [ c1 c2 (c1+c2) (c1 * c2) ]
i32.add      [ c1 c2 ((c1+c2) + (c1 * c2)) ]
nip          [ c1 ((c1+c2) + (c1 * c2)) ]
nip          [ ((c1+c2) + (c1 * c2)) ]
)

test2

asm.js

function test2 (c1, c2) {
  c1 = c1 |0;
  c2 = c2 |0;
  return (c1 + (c2 + (c1 * c2)));
}

with binaryen today I am getting the following code (11 byte):

(func $test2 (param $0 i32) (param $1 i32) (result i32)
    (i32.add
      (get_local $0)
      (i32.add
        (get_local $1)
        (i32.mul
          (get_local $0)
          (get_local $1)
        )
      )
    )
  )

good stack machine code would be (5 byte):

(func $test2
pick 1         [ c0 c1 c0 ]
pick 1         [ c0 c1 c0 c1 ]
i32.mul        [ c0 c1 (c0*c1) ]
i32.add        [ c0 (c1+(c0*c1)) ]
i32.add        [ (c0+(c1+(c0*c1))) ]
)

[ghost]

@drom Ok, thank you for the solutions. I also see potential changing the function arguments to be on the stack and accessed by pick aka get_value.

But it's not clear how you 'end game' could be represented in an expression. Also I think the following would inform the decoder earlier that c1 and c2 are consumed, which might make a difference to register pressure in a baseline compiler:

block
i32.const 1  [ c1 ]
i32.const 2  [ c1 c2 ]
pick 1       [ c1 c2 c1 ]
pick 1       [ c1 c2 c1 c2 ]
i32.add      [ c1 c2 (c1+c2) ]
swap 2       [ (c1+c2) c2 c1 ]  // But this messes up the structure.
swap 1       [ (c1+c2) c1 c2 ]
i32.mul      [ (c1+c2) (c1*c2) ]  // Lower register pressure here now.
i32.add      [ ((c1+c2) + (c1 * c2)) ]
end

Another 'move' in this 'chess game' is that we can invalidate a stack slot and define it to be a validation error to consume it. Let pick_and_void duplicate a stack element to the TOS and fill the source stack element with this void value. (Or pick_and_zero) Now we can keep the structure (no stack shuffling) and inform the compiler at the last use:

block <arity=1>
i32.const 1      [ c1 ]
i32.const 2      [ c1 c2 ]
pick 1           [ c1 c2 c1 ]
pick 1           [ c1 c2 c1 c2 ]
i32.add          [ c1 c2 (c1+c2) ]
pick_and_void 2  [ void c2 (c1+c2) c1 ]
pick_and_void 2  [ void void (c1+c2) c1 c2 ]
i32.mul          [ void void (c1+c2) (c1 * c2) ]  // lower register pressure now
i32.add          [ void void ((c1+c2) + (c1 * c2)) ]
// Assume the block discards unused values.
end

[drom]

@JSStats yes, I like the pick_and_void move! 👍 I have done similar commands in hardware stack machines. Very efficient.

[ghost]

Attempting to sketch a formatting rule with pick_and_void or pick_and_zero suggests that it might be necessary to made it a validation error to use pick when pick_and_void could have been used. This might be sufficient to uniquely define the code corresponding to structured uses of lexical constants which would be implemented via the pick operators.

[drom]

@JSStats is the "sketch" you are doing, for validation phase or stack scheduling?

[drom]

@JSStats in Forth pick operation has an evil twin brother -- roll that removes the element deep from the stack. ANS94 6.2.2150 : ROLL ( xu xu-1 ... x0 u -- xu-1 ... x0 xu ) but this behavior not considered to be nice. pick_and_void is much better.

[ghost]

@drom Just trying to understand if the structure can be preserved to keep a 'familiar' text format. If both pick and pick_and_void are valid where pick_and_void could be used then this extra information would need to be explicit in the text format and that would not be a 'familiar' named constant reference in the text format. A validation rule might address this.

Problem with roll is that it messes up the stack, and can that fit in with a structured text format, I don't know?

Here are some more examples that illustrate some more challenging corner cases for the text formatter. I have not found any show stoppers yet, but some to consider would be welcomed.

block <arity=1>
call $fn_returning_i32_i32_i32
i32.add
i32.add
end

{
  const (i32 $c0, i32 $c1, i32 $c2) = $fn_returning_i32_i32_i32();
  const i32 $c3 = $c1 + $c2;
  $c0 + $c3;
}

block <arity=1>
call $fn_returning_i32_i32_i32    [ 0, 1, 2]
pick_and_void 2     [ void, 1, 2, 0]
pick_and_void 2     [ void, void, 2, 0, 1]
pick_and_void 2     [ void, void, void, 0, 1, 2]
i32.add             [ void, void, void, 0, 1 + 2]
i32.add             [ void, void, void, 0 + (1 + 2)]
end                 [ 0 + (1 + 2)]

{
  const (i32 $c0, i32 $c1, i32 $c2) = $fn_returning_i32_i32_i32();
  const i32 $c3 = $c0;
  const i32 $c4 = $c1;
  const i32 $c5 = $c2;
  const i32 $c6 = $c4 + $c5;
  $c3 + $c6;
}

[drom]

@JSStats I was thinking about the name for the pick_and_void command, and came out with two candidates take and pull 😄

[drom]

@JSStats All this pick / pick_and void discussion bigger then the scope of this issue. We should probably move it to the different thread.

[ghost]

@drom Could move discussion specific to the pick proposal here #685 also see #698 although that was directed at the expressionless encoding proposal but same motivation. Would not worry about names, if you can find show-stoppers or inefficiencies etc that is what is needed at this point, we need to be able to put forward a counter proposal that plausibly retains the text format structure and addresses the efficiency use cases as best it can.

@flagxor did open this general issue for discussion here, and the use case being discussed is retaining the structure of the text format (a JS-style text format) rather than giving up as some implementers have proposed. No one has objected yet!

[drom]

@sunfishcode any idea how your text format proposal would be affected by new "stack machine" trend?

[sunfishcode]

@drom With the current stack-machine changes, our text format will require only a few additional features; possibly the addition of syntax for a "first" expression (similar to block, but returns the first value rather than the last) and possibly also syntax for a scoped or restricted "let" expression. I don't know what the actual syntax for these will be yet, but it's mostly just a matter of aesthetics :-).

As @flagxor mentioned above, this proposal has been removed from consideration for standardization in wasm, at least for now. The text format proposed here will continue to be developed at https://github.com/mbebenita/was and in Firefox.