Evaluate CLDR as database for cultures

[jzaefferer]

We're currently having various issues with the culture files generate from .NET (see label "cultures-bug"). We're considering moving to CLDR as the database for these files.

They're actually working on providing JSON files: http://unicode.org/cldr/trac/ticket/2733

We need to build a prototype to figure out if we can transform the CLDR data into something we can use here directly, or if we need to adapt the Globalize API to the CLDR data.

/cc @clarkbox @SlexAxton @Krinkle

[SlexAxton]

I know they don't love keeping it working (they do, it just is a hassle), but Dojo has scripts to take a lot of the CLDR info and turn it into usable JSON data and functions:

http://bugs.dojotoolkit.org/browser/dojo/util/trunk/buildscripts/cldr

It has some hefty pre-reqs, but that's to be expected with xlst and whatnot. I'd imagine you could do it a touch cleaner in node these days, but the core logic would be the same. I'm definitely looking into doing this, but I'd love for CLDR to just give us this stuff natively. :D

Adam Peller is the expert on this stuff and is/was active on that cldr ticket.

[jzaefferer]

Thanks Alex, appreciate your input! More puzzle pieces to work with.

[jzaefferer]

There's some activity on the cldr ticket: http://unicode.org/cldr/trac/ticket/2733#comment:22

Looks like for now our best bet is to just wait a few more weeks and hope for a release of those tools.

[jzaefferer]

Btw. Tim Wood of Moment.js had general interest in using CLDR, but outlined some issues here: moment/moment#315 - he now also closed that ticket, since he doesn't have time to work on it. If we ever want to add relative time functions to Globalize, Moment.js would be a good starting point.

[jzaefferer]

There's a Ldml2JsonConverter in CLDR now: http://unicode.org/cldr/trac/changeset/7886 Need to try that out.

[jzaefferer]

Google's Closure library use CLDR as the datasource. I haven't yet figured out how they import the data, but the data itself is in this file: http://docs.closure-library.googlecode.com/git/closure_goog_i18n_datetimesymbols.js.source.html

The equivalent of our format method is in DateTimeFormat: http://docs.closure-library.googlecode.com/git/class_goog_i18n_DateTimeFormat.html
Source, using the above DateTimeSymbols: http://docs.closure-library.googlecode.com/git/closure_goog_i18n_datetimeformat.js.source.html

The source list the tokens it supports, which I can't find in the API document. The header says its based on CLDR standards: "Datetime formatting functions following the pattern specification as defined in JDK, ICU and CLDR, with minor modification for typical usage in JS. Pattern specification: (Refer to JDK/ICU/CLDR)"

[jzaefferer]

Twitter published a JavaScript port of their Ruby CLDR wrapper 8 months ago: https://github.com/twitter/twitter-cldr-js

[ericelliott]

CLDR JSON page and CLDR data in JSON format.

[jzaefferer]

Thanks @dilvie for those links. Those look like it could just load them with JSON.parse, then output the properties we actually need. Using the full CLDR files seems like a bad idea, since they are pretty massive.

Its nice to see that there are properties for relative time formatting, e.g. "6 months ago". That's not something Globalize supports right now, but we could consider adding it.

[ericelliott]

@jzaefferer Agreed. There's a lot of data that is not necessary most of the time. Maybe somebody could create a custom build script similar to http://projects.jga.me/jquery-builder/, so we get only the features we really need. That would actually be really nice in Globalize -- especially if all you need are string translations and number formatting. Or string translations and relative time (gentle nudge to anybody with more time than I can spare at the moment).

[scottgonzalez]

It would be nice if we could just filter, as opposed to reformatting, the data. This would allow anyone who has full CLDR already available to just use that instead of duplicating some data with the Globalize files. We'll have to see if using the existing structure would become awkward.

[andyearnshaw]

Hey guys, I noticed this issue earlier when I was investigating how to convert the CLDR data to JSON format. You need the CLDR core.zip and tools.zip files, then you need to use the Ldml2JsonConverter in the tools to convert the data.

You can use what I have so far as an example, in the tools folder at https://github.com/andyearnshaw/Intl.js.

[ericelliott]

@andyearnshaw Thanks for the tips!

[rxaviers]

What about coverage. Does CLDR data cover all culture data needed by Globalize? @jzaefferer have you or anyone looked into it already?