The Optimistic Runtime Parser (#289)

This is a re-write of the runtime parser. It supports Fluent Syntax 0.7, runs
against the reference fixtures, has half the lines of code, and is as fast in
SpiderMonkey as the old one (and slightly faster in V8).


        Goals

  1. Support 100% of Fluent Syntax 0.7. This includes the indentation
     relaxation, dropping tabs and CR as syntax whitespace, normalizing new
     lines to LF, and only allowing numbers and identifiers as variant keys.

  2. Maintain good performance. The parser is used in performance-critical code
     paths. Back in the days of Firefox OS it had to be both fast _and_ produce
     tightly packed results so that translations don't take up too much space
     on the device. I think the storage requirements can be relaxed these days.

  3. Write code which will be easy to maintain in the future. The parser was
     first written even before Fluent branched off from L20n. It's seen many
     changes and additions over the last two years. As new features accrued it
     became hard to maintain it and also to keep track of all known bugs. My
     goal for the re-write was not only to clean it up but also to define the
     conformance story for the future and to improve the testing
     infrastructure.


        Design

The parser focuses on minimizing the number of false negatives at the expense
of increasing the risk of false positives. In other words, it aims at parsing
_valid_ Fluent messages with a success rate of 100%, but it may also parse some
invalid messages which the reference parser would reject. The parser doesn't
perform any validation and may produce entries  which wouldn't make sense in
the real world. For best results users are advised to validate translations
with the fluent-syntax parser pre-runtime.

The main parser loop iterates over the beginnings of messages and terms. This
is to efficiently skip over comments (which have no use on runtime), and to
recover from errors. When a fatal error is encountered, the parser instantly
bails out of the currently-parsed message and moves on to the next one. Errors
are discarded and are not visible to the users of `FluentResource`. The do
carry a minimal description of what went wrong which may be useful when reading
the code and for debugging, though.

The parser makes an extensive use of sticky regexes which can be anchored to
any offset of the source string without slicing it. In some places, it's easier
to just check the character currently at the cursor, so it does a fair share of
that, too.


        Conformance

My original plan was to base the parser on the EBNF and only parse well-formed
syntax. In this PR, I went for something a bit wider than that: a superset of
well-formed syntax. The main deviation from the EBNF is related to parsing
`VariantExpressions` and `CallExpressions`. The EBNF verifies that the they are
called on `Terms` and `Functions` respectively. The optimistic parser doesn't
differentiate between `Messages`, `Terms` and `Functions`. I decided to
implement it this way because this code might soon change anyways (see
projectfluent/fluent#176).

Another deviation is that the parser treats commas in argument lists  as
whitespace, similar to how Clojure treats them in sequence lists. I might
suggest we upstream this in the spec, too, because it makes the implementation
of args lists _much_ simpler.

I based this PR on top of the `zeroseven` branch. The `fluent-syntax` parser
already supports Syntax 0.7 and passes the [reference
fixtures](https://github.com/projectfluent/fluent/tree/master/test/fixtures).
This made it possible to also turn on the reference testing in the runtime
parser, too. `make fixtures` creates the parsed results for all reference
fixtures; for now they must be verified manually before they're committed.
`make test` can be used in development to assert that the output of the runtime
parser still matches the committed one.

Author: stasm

eslint_src.json

@@ -68,7 +68,6 @@
         "no-mixed-spaces-and-tabs": 2,
         "no-tabs": 2,
         "prefer-arrow-callback": 1,
-        "prefer-const": 2,
         "prefer-template": 2,
         "prefer-spread": 2,
         "quotes": [

fluent/makefile

@@ -29,17 +29,24 @@ clean:
 	@echo -e " $(OK) clean"
 
 BEHAVIOR_FTL := $(wildcard ../fluent-syntax/test/fixtures_behavior/*.ftl)
+REFERENCE_FTL := $(wildcard ../fluent-syntax/test/fixtures_reference/*.ftl)
 STRUCTURE_FTL := $(wildcard ../fluent-syntax/test/fixtures_structure/*.ftl)
 BEHAVIOR_JSON := $(BEHAVIOR_FTL:../fluent-syntax/test/fixtures_behavior/%.ftl=test/fixtures_behavior/%.json)
+REFERENCE_JSON := $(REFERENCE_FTL:../fluent-syntax/test/fixtures_reference/%.ftl=test/fixtures_reference/%.json)
 STRUCTURE_JSON := $(STRUCTURE_FTL:../fluent-syntax/test/fixtures_structure/%.ftl=test/fixtures_structure/%.json)
 
-fixtures: $(BEHAVIOR_JSON) $(STRUCTURE_JSON)
+fixtures: $(BEHAVIOR_JSON) $(REFERENCE_JSON) $(STRUCTURE_JSON)
 
 .PHONY: $(BEHAVIOR_JSON)
 $(BEHAVIOR_JSON): test/fixtures_behavior/%.json: ../fluent-syntax/test/fixtures_behavior/%.ftl
 	@node test/fixtures_behavior/make_fixtures.js -- $< > $@
 	@echo -e " $(OK) $@"
 
+.PHONY: $(REFERENCE_JSON)
+$(REFERENCE_JSON): test/fixtures_reference/%.json: ../fluent-syntax/test/fixtures_reference/%.ftl
+	@../tools/parse.js --runtime --silent $< > $@
+	@echo -e " $(OK) $@"
+
 .PHONY: $(STRUCTURE_JSON)
 $(STRUCTURE_JSON): test/fixtures_structure/%.json: ../fluent-syntax/test/fixtures_structure/%.ftl
 	@../tools/parse.js --runtime --silent $< > $@

fluent/src/builtins.js

@@ -11,7 +11,7 @@
  * `FluentType`.  Functions must return `FluentType` objects as well.
  */
 
-import { FluentNumber, FluentDateTime } from "./types";
+import { FluentNumber, FluentDateTime } from "./types.js";
 
 export default {
   "NUMBER": ([arg], opts) =>

fluent/src/bundle.js

@@ -1,5 +1,5 @@
-import resolve from "./resolver";
-import FluentResource from "./resource";
+import resolve from "./resolver.js";
+import FluentResource from "./resource.js";
 
 /**
  * Message bundles are single-language stores of translations.  They are
@@ -138,7 +138,8 @@ export default class FluentBundle {
    * @returns {Array<Error>}
    */
   addResource(res) {
-    const errors = res.errors.slice();
+    const errors = [];
+
     for (const [id, value] of res) {
       if (id.startsWith("-")) {
         // Identifiers starting with a dash (-) define terms. Terms are private
@@ -196,16 +197,16 @@ export default class FluentBundle {
       return this._transform(message);
     }
 
-    // optimize simple-string entities with attributes
-    if (typeof message.val === "string") {
-      return this._transform(message.val);
-    }
-
     // optimize entities with null values
-    if (message.val === undefined) {
+    if (message === null || message.value === null) {
       return null;
     }
 
+    // optimize simple-string entities with attributes
+    if (typeof message.value === "string") {
+      return this._transform(message.value);
+    }
+
     return resolve(this, args, message, errors);
   }
 

fluent/src/error.js

@@ -0,0 +1 @@
+export default class FluentError extends Error {}

fluent/src/index.js

@@ -7,8 +7,9 @@
  *
  */
 
-export { default as FluentBundle } from "./bundle";
-export { default as FluentResource } from "./resource";
-export { FluentType, FluentNumber, FluentDateTime } from "./types";
+export { default as FluentBundle } from "./bundle.js";
+export { default as FluentResource } from "./resource.js";
+export { default as FluentError } from "./error.js";
+export { FluentType, FluentNumber, FluentDateTime } from "./types.js";
 
-export { ftl } from "./util";
+export { ftl } from "./util.js";