Implement fluent.bundle namespace package

.gitattributes

@@ -1,3 +1,3 @@
-tests/syntax/fixtures_reference/crlf.ftl eol=crlf
-tests/syntax/fixtures_reference/cr.ftl eol=cr
-tests/syntax/fixtures_structure/crlf.ftl eol=crlf
+fluent.syntax/tests/syntax/fixtures_reference/crlf.ftl eol=crlf
+fluent.syntax/tests/syntax/fixtures_reference/cr.ftl eol=cr
+fluent.syntax/tests/syntax/fixtures_structure/crlf.ftl eol=crlf

.gitignore

@@ -1,4 +1,4 @@
 .tox
 *.pyc
 .eggs/
-fluent.egg-info/
+*.egg-info/

.travis.yml

@@ -4,9 +4,14 @@ python:
   - "2.7"
   - "3.5"
   - "3.6"
+  - "pypy"
+  - "pypy3"
   - "nightly"
+env:
+  - PACKAGE=fluent.syntax
+  - PACKAGE=fluent.runtime
 install: pip install tox-travis
-script: tox
+script: cd $PACKAGE; tox
 notifications:
   irc:
     channels:

README.md

@@ -30,6 +30,251 @@ you're a tool author you may be interested in the formal [EBNF grammar][].
 [EBNF grammar]: https://github.com/projectfluent/fluent/tree/master/spec
 
 
+Installation
+------------
+
+python-fluent consists of two packages:
+
+* `fluent.syntax` - includes AST classes and parser. Most end users will not
+  need this directly. Documentation coming soon!
+
+  To install:
+
+        pip install fluent.syntax
+
+
+* `fluent.runtime` - methods for generating translations from FTL files.
+  Documentation below.
+
+  To install:
+
+        pip install fluent.runtime
+
+  (The correct version of ``fluent.syntax`` will be installed automatically)
+
+
+PyPI also contains an old `fluent` package which is an older version of just
+`fluent.syntax`.
+
+Usage
+-----
+
+To generate translations using the ``fluent.runtime`` package, you start with
+the `FluentBundle` class:
+
+    >>> from fluent.runtime import FluentBundle
+
+You pass a list of locales to the constructor - the first being the desired
+locale, with fallbacks after that:
+
+    >>> bundle = FluentBundle(["en-US"])
+
+
+You must then add messages. These would normally come from a `.ftl` file stored
+on disk, here we will just add them directly:
+
+    >>> bundle.add_messages("""
+    ... welcome = Welcome to this great app!
+    ... greet-by-name = Hello, { $name }!
+    ... """)
+
+To generate translations, use the `format` method, passing a message ID and an
+optional dictionary of substitution parameters. If the the message ID is not
+found, a `LookupError` is raised. Otherwise, as per the Fluent philosophy, the
+implementation tries hard to recover from any formatting errors and generate the
+most human readable representation of the value. The `format` method therefore
+returns a tuple containing `(translated string, errors)`, as below.
+
+    >>> translated, errs = bundle.format('welcome')
+    >>> translated
+    "Welcome to this great app!"
+    >>> errs
+    []
+
+    >>> translated, errs = bundle.format('greet-by-name', {'name': 'Jane'})
+    >>> translated
+    'Hello, \u2068Jane\u2069!'
+
+    >>> translated, errs = bundle.format('greet-by-name', {})
+    >>> translated
+    'Hello, \u2068name\u2069!'
+    >>> errs
+    [FluentReferenceError('Unknown external: name')]
+
+You will notice the extra characters `\u2068` and `\u2069` in the output. These
+are Unicode bidi isolation characters that help to ensure that the interpolated
+strings are handled correctly in the situation where the text direction of the
+substitution might not match the text direction of the localized text. These
+characters can be disabled if you are sure that is not possible for your app by
+passing `use_isolating=False` to the `FluentBundle` constructor.
+
+Python 2
+--------
+
+The above examples assume Python 3. Since Fluent uses unicode everywhere
+internally (and doesn't accept bytestrings), if you are using Python 2 you will
+need to make adjustments to the above example code. Either add `u` unicode
+literal markers to strings or add this at the top of the module or the start of
+your repl session:
+
+    from __future__ import unicode_literals
+
+
+Numbers
+-------
+
+When rendering translations, Fluent passes any numeric arguments (int or float)
+through locale-aware formatting functions:
+
+    >>> bundle.add_messages("show-total-points = You have { $points } points.")
+    >>> val, errs = bundle.format("show-total-points", {'points': 1234567})
+    >>> val
+    'You have 1,234,567 points.'
+
+
+You can specify your own formatting options on the arguments passed in by
+wrapping your numeric arguments with `fluent.runtime.types.fluent_number`:
+
+    >>> from fluent.runtime.types import fluent_number
+    >>> points = fluent_number(1234567, useGrouping=False)
+    >>> bundle.format("show-total-points", {'points': points})[0]
+    'You have 1234567 points.'
+
+    >>> amount = fluent_number(1234.56, style="currency", currency="USD")
+    >>> bundle.add_messages("your-balance = Your balance is { $amount }")
+    >>> bundle.format("your-balance", {'amount': amount})[0]
+    'Your balance is $1,234.56'
+
+Thee options available are defined in the Fluent spec for
+[NUMBER](https://projectfluent.org/fluent/guide/functions.html#number). Some of
+these options can also be defined in the FTL files, as described in the Fluent
+spec, and the options will be merged.
+
+Date and time
+-------------
+
+Python `datetime.datetime` and `datetime.date` objects are also passed through
+locale aware functions:
+
+    >>> from datetime import date
+    >>> bundle.add_messages("today-is = Today is { $today }")
+    >>> val, errs = bundle.format("today-is", {"today": date.today() })
+    >>> val
+    'Today is Jun 16, 2018'
+
+You can explicitly call the `DATETIME` builtin to specify options:
+
+    >>> bundle.add_messages('today-is = Today is { DATETIME($today, dateStyle: "short") }')
+
+See the [DATETIME
+docs](https://projectfluent.org/fluent/guide/functions.html#datetime). However,
+currently the only supported options to `DATETIME` are:
+
+* `timeZone`
+* `dateStyle` and `timeStyle` which are [proposed
+  additions](https://github.com/tc39/proposal-ecma402-datetime-style) to the ECMA i18n spec.
+
+To specify options from Python code, use `fluent.runtime.types.fluent_date`:
+
+    >>> from fluent.runtime.types import fluent_date
+    >>> today = date.today()
+    >>> short_today = fluent_date(today, dateStyle='short')
+    >>> val, errs = bundle.format("today-is", {"today": short_today })
+    >>> val
+    'Today is 6/17/18'
+
+You can also specify timezone for displaying `datetime` objects in two ways:
+
+* Create timezone aware `datetime` objects, and pass these to the `format` call
+  e.g.:
+
+        >>> import pytz
+        >>> from datetime import datetime
+        >>> utcnow = datime.utcnow().replace(tzinfo=pytz.utc)
+        >>> moscow_timezone = pytz.timezone('Europe/Moscow')
+        >>> now_in_moscow = utcnow.astimezone(moscow_timezone)
+
+* Or, use timezone naive `datetime` objects, or ones with a UTC timezone, and
+  pass the `timeZone` argument to `fluent_date` as a string:
+
+        >>> utcnow = datetime.utcnow()
+        >>> utcnow
+        datetime.datetime(2018, 6, 17, 12, 15, 5, 677597)
+
+        >>> bundle.add_messages("now-is = Now is { $now }")
+        >>> val, errs = bundle.format("now-is",
+        ...    {"now": fluent_date(utcnow,
+        ...                        timeZone="Europe/Moscow",
+        ...                        dateStyle="medium",
+        ...                        timeStyle="medium")})
+        >>> val
+        'Now is Jun 17, 2018, 3:15:05 PM'
+
+
+Custom functions
+----------------
+
+You can add functions to the ones available to FTL authors by passing
+a `functions` dictionary to the `FluentBundle` constructor:
+
+
+    >>> import platform
+    >>> def os_name():
+    ...    """Returns linux/mac/windows/other"""
+    ...    return {'Linux': 'linux',
+    ...            'Darwin': 'mac',
+    ...            'Windows': 'windows'}.get(platform.system(), 'other')
+
+    >>> bundle = FluentBundle(['en-US'], functions={'OS': os_name})
+    >>> bundle.add_messages("""
+    ... welcome = { OS() ->
+    ...    [linux]    Welcome to Linux
+    ...    [mac]      Welcome to Mac
+    ...    [windows]  Welcome to Windows
+    ...   *[other]    Welcome
+    ...   }
+    ... """)
+    >>> print(bundle.format('welcome')[0]
+    Welcome to Linux
+
+These functions can accept positioal and keyword arguments (like the `NUMBER`
+and `DATETIME` builtins), and in this case must accept the following types of
+arguments:
+
+* unicode strings (i.e. `unicode` on Python 2, `str` on Python 3)
+* `fluent.runtime.types.FluentType` subclasses, namely:
+  * `FluentNumber` - `int`, `float` or `Decimal` objects passed in externally,
+    or expressed as literals, are wrapped in these. Note that these objects also
+    subclass builtin `int`, `float` or `Decimal`, so can be used as numbers in
+    the normal way.
+  * `FluentDateType` - `date` or `datetime` objects passed in are wrapped in
+    these. Again, these classes also subclass `date` or `datetime`, and can be
+    used as such.
+  * `FluentNone` - in error conditions, such as a message referring to an argument
+    that hasn't been passed in, objects of this type are passed in.
+
+Custom functions should not throw errors, but return `FluentNone` instances to
+indicate an error or missing data. Otherwise they should return unicode strings,
+or instances of a `FluentType` subclass as above.
+
+
+Known limitations and bugs
+--------------------------
+
+* We do not yet support `NUMBER(..., currencyDisplay="name")` - see [this python-babel
+  pull request](https://github.com/python-babel/babel/pull/585) which needs to
+  be merged and released.
+
+* Most options to `DATETIME` are not yet supported. See the [MDN docs for
+  Intl.DateTimeFormat](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat),
+  the [ECMA spec for
+  BasicFormatMatcher](http://www.ecma-international.org/ecma-402/1.0/#BasicFormatMatcher)
+  and the [Intl.js
+  polyfill](https://github.com/andyearnshaw/Intl.js/blob/master/src/12.datetimeformat.js).
+
+Help with the above would be welcome!
+
+
 Discuss
 -------
 

fluent.runtime/fluent/runtime/__init__.py

@@ -0,0 +1,78 @@
+from __future__ import absolute_import, unicode_literals
+
+import babel
+import babel.numbers
+import babel.plural
+
+from fluent.syntax import FluentParser
+from fluent.syntax.ast import Message, Term
+
+from .builtins import BUILTINS
+from .resolver import resolve
+
+
+class FluentBundle(object):
+    """
+    Message contexts are single-language stores of translations.  They are
+    responsible for parsing translation resources in the Fluent syntax and can
+    format translation units (entities) to strings.
+
+    Always use `FluentBundle.format` to retrieve translation units from
+    a context.  Translations can contain references to other entities or
+    external arguments, conditional logic in form of select expressions, traits
+    which describe their grammatical features, and can use Fluent builtins.
+    See the documentation of the Fluent syntax for more information.
+    """
+
+    def __init__(self, locales, functions=None, use_isolating=True):
+        self.locales = locales
+        _functions = BUILTINS.copy()
+        if functions:
+            _functions.update(functions)
+        self._functions = _functions
+        self._use_isolating = use_isolating
+        self._messages_and_terms = {}
+        self._babel_locale = self._get_babel_locale()
+        self._plural_form = babel.plural.to_python(self._babel_locale.plural_form)
+
+    def add_messages(self, source):
+        parser = FluentParser()
+        resource = parser.parse(source)
+        # TODO - warn/error about duplicates
+        for item in resource.body:
+            if isinstance(item, (Message, Term)):
+                if item.id.name not in self._messages_and_terms:
+                    self._messages_and_terms[item.id.name] = item
+
+    def has_message(self, message_id):
+        if message_id.startswith('-'):
+            return False
+        return message_id in self._messages_and_terms
+
+    def format(self, message_id, args=None):
+        message = self._get_message(message_id)
+        if args is None:
+            args = {}
+        return resolve(self, message, args)
+
+    def _get_message(self, message_id):
+        if message_id.startswith('-'):
+            raise LookupError(message_id)
+        if '.' in message_id:
+            name, attr_name = message_id.split('.', 1)
+            msg = self._messages_and_terms[name]
+            for attribute in msg.attributes:
+                if attribute.id.name == attr_name:
+                    return attribute.value
+            raise LookupError(message_id)
+        else:
+            return self._messages_and_terms[message_id]
+
+    def _get_babel_locale(self):
+        for l in self.locales:
+            try:
+                return babel.Locale.parse(l.replace('-', '_'))
+            except babel.UnknownLocaleError:
+                continue
+        # TODO - log error
+        return babel.Locale.default()

fluent.runtime/fluent/runtime/builtins.py

@@ -0,0 +1,10 @@
+from .types import fluent_date, fluent_number
+
+NUMBER = fluent_number
+DATETIME = fluent_date
+
+
+BUILTINS = {
+    'NUMBER': NUMBER,
+    'DATETIME': DATETIME,
+}

fluent.runtime/fluent/runtime/errors.py

@@ -0,0 +1,15 @@
+from __future__ import absolute_import, unicode_literals
+
+
+class FluentFormatError(ValueError):
+    def __eq__(self, other):
+        return ((other.__class__ == self.__class__) and
+                other.args == self.args)
+
+
+class FluentReferenceError(FluentFormatError):
+    pass
+
+
+class FluentCyclicReferenceError(FluentFormatError):
+    pass