Binary encoder & decoder (#268)

Add working (post-order) encoder & decoder, and several additions to make it useful. For example, you can now invoke the interpreter like
```
wasm module.wast -o module.wasm
```
to convert from text to binary. In a while, the same will also be possible in the inverse direction.

I also extended the script language with (input <file>) and (output <file>) commands. Both are supposed to be able to handle both wast and wasm eventually. The former allows including other scripts or binary modules, the latter allows conversion as part of a script.

Finally, the command line now supports an -e <script> option, which enables to give commands directly. This is useful, for example, when intermingled with binary module arguments, e.g., to invoke exports:
```
wasm module.wasm -e '(invoke "foo")'
```

Furthermore, extends the test runner with encoding & decoding of all test files via a wast->wasm transcoding and loading the resulting modules. Works for all current tests.

There are still a number of smaller TODOs left, to be addressed in follow-ups. Other follow-up work: wasm->wast reverse transcoding, more aggressive tests and testing capabilities. We'll probably also need to revise the use of different integer types throughout the spec, which is not always consistent with what the binary format supports.

Author: rossberg

ml-proto/README.md

@@ -5,10 +5,13 @@ This repository implements a prototypical reference interpreter for WebAssembly.
 Currently, it can
 
 * *parse* a simple S-expression format,
+* *decode* the binary format (work in progress),
 * *validate* modules defined in it,
-* *execute* invocations to functions exported by a module.
+* *execute* invocations to functions exported by a module,
+* *encode* the binary format,
+* *prettyprint* the S-expression format (work in progress).
 
-The file format is a (very dumb) form of *script* that cannot just define a module, but also batch a sequence of invocations.
+The S-expression format is a (very dumb) form of *script* that cannot just define a module, but in fact a sequence of them, and a batch of invocations, assertions, and conversions to each one. As such it is different from the binary format, with the additional functionality purely intended as testing infrastructure. (See [below](#scripts) for details.)
 
 The interpreter can also be run as a REPL, allowing to enter pieces of scripts interactively.
 
@@ -61,17 +64,34 @@ Either way, in order to run the test suite you'll need to have Python installed.
 You can call the executable with
 
 ```
-wasm [option] [file ...]
+wasm [option | file ...]
 ```
 
-where `file` is a script file (see below) to be run. If no file is given, you'll get into the REPL and can enter script commands interactively. You can also get into the REPL by explicitly passing `-` as a file name. You can do that in combination to giving a module file, so that you can then invoke its exports interactively, e.g.:
+where `file`, depending on its extension, either should be an S-expression script file (see below) to be run, or a binary module file to be loaded.
+
+A file prefixed by `-o` is taken to be an output file. Depending on its extension, this will write out the preceding module definition in either S-expression or binary format. This option can be used to convert between the two in both directions, e.g.:
 
 ```
-./wasm module.wast -
+wasm -d module.wast -o module.wasm
+wasm -d module.wasm -o module.wast
 ```
-Note however that the REPL currently is too dumb to allow multi-line input. :)
 
-See `wasm -h` for (the few) options.
+The `-d` option selects "dry mode" and ensures that the input module is not run, even if it has a start section.
+In the second case, the produced script contains exactly one module definition (work in progress).
+
+Finally, the option `-e` allows to provide arbitrary script commands directly on the command line. For example:
+
+```
+wasm module.wasm -e '(invoke "foo")'
+```
+
+If neither a file nor any of the previous options is given, you'll land in the REPL and can enter script commands interactively. You can also get into the REPL by explicitly passing `-` as a file name. You can do that in combination to giving a module file, so that you can then invoke its exports interactively, e.g.:
+
+```
+wasm module.wast -
+```
+
+See `wasm -h` for (the few) additional options.
 
 
 ## S-Expression Syntax
@@ -168,9 +188,13 @@ cmd:
   ( assert_return_nan (invoke <name> <expr>* ))        ;; assert return with floating point nan result of invocation
   ( assert_trap (invoke <name> <expr>* ) <failure> )   ;; assert invocation traps with given failure string
   ( assert_invalid <module> <failure> )                ;; assert invalid module with given failure string
+  ( input <string> )                                   ;; read script or module from file
+  ( output <string> )                                  ;; output module to file
 ```
 
-Invocation is only possible after a module has been defined.
+Commands are executed in sequence. Invocation, assertions, and output apply to the most recently defined module (the _current_ module), and are only possible after a module has been defined. Note that there only ever is one current module, the different module definitions cannot interact.
+
+The input and output commands determine the requested file format from the file name extension. They can handle both `.wast` and `.wasm` files. In the case of input, a `.wast` script will be recursively executed.
 
 Again, this is only a meta-level for testing, and not a part of the language proper.
 
@@ -202,11 +226,15 @@ The implementation consists of the following parts:
 
 * *Parser* (`lexer.mll`, `parser.mly`, `desguar.ml[i]`). Generated with ocamllex and ocamlyacc. The lexer does the opcode encoding (non-trivial tokens carry e.g. type information as semantic values, as declared in `parser.mly`), the parser the actual S-expression parsing. The parser generates a full AST that is desugared into the kernel AST in a separate pass.
 
+* *Pretty Printer* (`prettyprint.ml[i]`). Turns a module AST back into the textual S-expression format. (Work in progress)
+
+* *Decoder*/*Encoder* (`decode.ml[i]`, `encode.ml[i]`). The former (work in progress) parses the binary format and turns it into an AST, the latter does the inverse.
+
 * *Validator* (`check.ml[i]`). Does a recursive walk of the kernel AST, passing down the *expected* type for expressions, and checking each expression against that. An expected empty type can be matched by any result, corresponding to implicit dropping of unused values (e.g. in a block).
 
 * *Evaluator* (`eval.ml[i]`, `values.ml`, `arithmetic.ml[i]`, `int.ml`, `float.ml`, `memory.ml[i]`, and a few more). Evaluation of control transfer (`br` and `return`) is implemented using local exceptions as "labels". While these are allocated dynamically in the code and addressed via a stack, that is merely to simplify the code. In reality, these would be static jumps.
 
-* *Driver* (`main.ml`, `script.ml[i]`, `error.ml`, `print.ml[i]`, `flags.ml`). Executes scripts, reports results or errors, etc.
+* *Driver* (`main.ml`, `run.ml[i]`, `script.ml[i]`, `error.ml`, `print.ml[i]`, `flags.ml`). Executes scripts, reports results or errors, etc.
 
 The most relevant pieces are probably the validator (`check.ml`) and the evaluator (`eval.ml`). They are written to look as much like a "specification" as possible. Hopefully, the code is fairly self-explanatory, at least for those with a passing familiarity with functional programming.
 
@@ -215,6 +243,6 @@ In typical FP convention (and for better readability), the code tends to use sin
 
 ## What Next?
 
-* Binary format as input and output.
+* More tests.
 
-* Compilation to JS/asm.js.
+* Compilation to JS/asm.js?

ml-proto/given/lib.ml

@@ -1,5 +1,8 @@
 module List =
 struct
+  let rec make n x =
+    if n = 0 then [] else x :: make (n - 1) x
+
   let rec take n xs =
     match n, xs with
     | 0, _ -> []

ml-proto/given/lib.mli

@@ -2,6 +2,7 @@
 
 module List :
 sig
+  val make : int -> 'a -> 'a list
   val take : int -> 'a list -> 'a list
   val drop : int -> 'a list -> 'a list
 

ml-proto/given/source.ml

@@ -1,6 +1,6 @@
 type pos = {file : string; line : int; column : int}
 type region = {left : pos; right : pos}
-type 'a phrase = { at : region; it : 'a}
+type 'a phrase = {at : region; it : 'a}
 
 
 (* Positions and regions *)
@@ -9,9 +9,14 @@ let no_pos = {file = ""; line = 0; column = 0}
 let no_region = {left = no_pos; right = no_pos}
 
 let string_of_pos pos =
-  string_of_int pos.line ^ "." ^ string_of_int (pos.column + 1)
+  if pos.line = -1 then
+    string_of_int pos.column
+  else
+    string_of_int pos.line ^ "." ^ string_of_int (pos.column + 1)
+
 let string_of_region r =
-  r.left.file ^ ":" ^ string_of_pos r.left ^ "-" ^ string_of_pos r.right
+  r.left.file ^ ":" ^ string_of_pos r.left ^
+  (if r.right = r.left then "" else "-" ^ string_of_pos r.right)
 
 let before region = {left = region.left; right = region.left}
 let after region = {left = region.right; right = region.right}