diff --git a/meetings/task-force/#103-2020-08-31.md b/meetings/task-force/#103-2020-08-31.md new file mode 100644 index 0000000000..10592681b4 --- /dev/null +++ b/meetings/task-force/#103-2020-08-31.md @@ -0,0 +1,428 @@ +## Executive summary ([Original Doc](https://docs.google.com/document/d/1-6t6Yl5RHZI9QZwBDrFrl1fqSKSA4IMs1ef60IxD3lU/edit#)) + +*Participants:* +- DAF: David Filip +- ECH: Elango Cheran +- MIH: Mihai Nita +- NIC: Nicolas Bouvrette +- RCA: Romulo Cintra +- SMY: Stanislaw Malolepszy +- ZIB: Zbigniew Braniecki + +The discussion was focused on issue [#103](https://github.com/unicode-org/message-format-wg/issues/103). + +Despite the title of issue #103, the main topic of discussion was more precisely: supporting in-message and full-message selectors vs full-message selectors only. + +### Full-message selection: +``` +ICU: {count, plural, + =1 {You deleted # file from the folder {folder_name}!} + other {You deleted # files from the folder {folder_name}!} +} + +Fluent: {$count -> + [one] You deleted {$count} file from the folder {$folder_name}! + *[other] You deleted {$count} files from the folder {$folder_name}! +} +``` + +### In-message selection (sub-selectors): + +``` +ICU: You deleted {count, plural, =1{# file} other{# files}} from the folder {folder_name}! + +Fluent: You deleted {$count -> + [one] {$count} file + *[other] {$count} files + } from the folder {$folder_name}! + +``` + +The participants agreed that the two forms (in-message / full-message) are functionally equivalent. They can both represent the same kind of messages, and it is possible to algorithmically convert between them without data loss. + +We can also allow for an in-message selection in the developer syntax but not allow it in the data model. So some of the verbosity can be mitigated when in developer-facing syntax / tooling. + +We then collected pros and cons for the two options. + +Message Level Selectors +Pros +Friendly for translators +L10n tools friendly +More “implementor” friendly +Cons +Unfriendly for developers (verbose) +Verbose (to move through the wire) +No accurate round-tripping (version control?) + +Sub-selectors +Pros +Friendly for developers +Cons +Unfriendly for translators +Hard to track what happens on the boundary of segments +Harder to grep for text +Integration with TMS is more difficult + + + +> Approval Stamps for Executive Summary + +*ECH,NIC,DAF,SMY,RCA* + + + +## Notes from meeting on 2020-08-31 + + +SMY: We have to be careful about decisions because of tradeoffs. Flexibility vs disallowing bad practices. + +MIH: It’s like programming languages (ex: static vs dynamic). + +NIC: We’ve tried to document pros and cons. Maybe we can find more concrete factual data to support each argument. If we want to support this in TMS, then we need to do work to integrate. + + +MIH: Did I capture the similarities and differences correctly in the updated comparison slides doc? + +Slides doc: https://docs.google.com/presentation/d/1xi4cyLmLVADNXSb-xNtZoz1xZNbcwQa4khuqt-21MOo/edit#slide=id.g94e31aa88b_1_0 + +SMY: Edge cases = the combinatorial number of combinations of cases. Let’s start with NIC’s suggestion of specific use cases + +MIH: Core decision to make is “do we allow selector type of constructs within the message or not” ? In other words, do we let our message pattern cases be full sentences + +DAF: From a round-trip localization point of view, I am for the message-level selectors. I don’t think the other approach is possible for GUIs. + +MIH: I’ve seen that, actually, but it is very unfriendly to translators. You as the translator have to coordinate translations across sub-message parts to ensure subject / verb / inflection agreement, and it is very difficult. Also, it ruins integrations with Translation Memories (TMs). + +ZIB: (chat) in context of what Nicolas said, I deeply recommend https://www.youtube.com/watch?v=2ajos-0OWts + +ZIB: I think the fundamental question is are we designing a system for today or for 10 years from now? Role model example is CSS. Previously, they questioned CSS as unnecessary and unsupported. + +MIH: In the localization industry, things move very slowly. So I think 10 years from now, things will look similar to what they are now, just my opinion. + +ZIB: Responding, wearing an optimistic hat. Unprecedented situation, we have the backing of Unicode and W3C and avenue for landing straight into JS. Landing into JS gives us adoption by the largest pool of developers and reach. We have more push power than before. + +SMY: Just as ZIB is an optimist, I’m perceived as a pessimist, which is why we work very well. I urge the side of modesty and caution. We got here to this meeting to discuss how selection works, but it is good to discuss principles of design. The crucial difference between CSS, OpenGL, and localization is that there were millions of dollars to be made with CSS and OpenGL, but localization is already established. But the quality of translation is not from supporting 4 plurals at the same time, but the quality of the text that we put in there. So we have to support a format that enables translators to write great prose translations. + +MIH: I would like to exclude from discussion about what tooling will look like in 10 years. Both CSS and OpenGL are developer oriented technologies. What is not going away for localization is the linguistic part -- some languages will require one to bring information from outside to inside a selector. The main argument is that the 2 selection models are perfectly equivalent, they allow the same things, and I have code that can convert between the 2. + +DAF: I would agree that they are equivalent in expressivity, and certainly there is a difference in verbosity. But verbosity also makes things translation friendly, and the verbosity won’t cost extra money due to TM leveraging. + +SMY: I wanted to add on what DAF said to a similar effect. I’ve seen the rise of Translation Memory (TM) and machine translation (MT). So I would be interested in which model supports these functionalities better. + +DAF: I asked in my first monthly meeting if we’re trying to build a universal Rule Based translation engine, and it sounds like we are, in some ways. + +MIH: If both approaches map to each other exactly, choosing one over the other right now doesn’t mean we can’t change it later. + +And it doesn’t prevent us from having a different syntax that has “internal selection” (nested selector sub-messages). We can still convert to the data model. + +SMY: For conversion, this is a lossy conversion, right? + +DAF: I would support a layered approach. At the syntax level, we can support the nested selector approach in syntax and then have full-message messages when sending off to translation. +In reaction to SMY, the conversion would not be necessarily lossy. In L10n we work with this paradigm of extraction and merging. L10n happens between these brackets. And it is always assumed that merger has the full knowledge of the extraction process. So you should be able to work losslessly in your proprietary bracket, knowing what you have done to expand to the canonical format.. + +ZIB: I am mesmerized by the sentiment that we are talking about having the data model user-friendly and the syntax is okay to be verbose. Shouldn’t the syntax be user-friendly and allow the data model to handle more and be more flexible? + +MIH: Yes, and no. There are no “generic humans” in this equation. Developers can handle certain syntax that translators cannot, we can’t put them into the same bucket. What we can’t currently handle in the data model but I see in some proprietary tools is to automatically add missing plural cases. In this example, you really need to do this at the full-message level. + +DAF: I think there is some misunderstanding in hearing ZIB’s reaction. It’s an interoperability effort. We are trying to come up with an interoperability vehicle, not prescribe the whole thing. We can have the more verbose representation be the canonical, but syntax can always be adjusted as we see fit. So in your proprietary bracket, you can make the syntax be whatever you want, so long as you can convert to the canonical representation. + +SMY: So to make sense of that for the Fluent case, it sounds like Fluent can have its own syntax, so long as it can be converted into the canonical format. Right? + +DAF: Yes. Example: Markdown is simplified, but you can always convert to HTML, and you can allow plain HTML in Markdown to support the full range of HTML. + +MIH: For me, the core question is do we support selection happening within (nested in) the message pattern, or does it happen at the message level. + +DAF: I think having the internal selection would be too costly. If we support internal selection as part of the standard, it would need to create too many syntax peculiarities. + +MIH: Should we try to list the pros and cons of each approach on the topic of allowing nested selectors? + + + + + +| Message Level Selectors | | Sub-selectors | | +| :---: | --- | :---: | --- | +| Pros | Cons | Pros | Cons | +| Friendly for translators

L10n tools friendly

More “implementor” friendly | Unfriendly for developers (verbose)

Verbose (to move through the wire)

Developers must decide to make a message a selector even if the target language doesn’t need it. | Friendly for developers | Unfriendly for translators

Hard to track what happens on the boundary of segments

Harder to grep for text

Integration with TMS is more difficult | + + + + +MIH: Nesting breaks a lot of things in a TMS, like validation (do the number of placeholders match?), not just translation (MT) and memory (TM). + + +============================= + +#### ICU MF (source lang) + +``` + +You have {COUNT, plural, + one {1 unread message} + other {{COUNT} unread messages}} +in your inbox. + +Canonical data model MFWG (source lang) + +SWITCHES: [{plural, COUNT}] +CASES: { + [one]: You have 1 unread message in your inbox. + [other]: You have 1 unread message in your inbox. +} + +``` + +SMY: Can I bring your attention to discussion from pros and cons to the issue of whether we can have the data model be the backend and having syntaxes be the front end? + +In the example of translating to Polish, do we want this to be driven by the translator themselves, or do we always try to represent the maximum number of plural cases? + + +#### Canonical data model MFWG (target lang) + +``` + +SWITCHES: [{plural, COUNT}] +CASES: { + [one]: Masz 1 wiadomość w skrzynce odbiorczej. + [few]: Masz {COUNT} wiadomości w skrzynce odbiorczej. + [other]: Masz {COUNT} wiadomości w skrzynce odbiorczej. +} + +ICU MF (target lang) + +{COUNT, plural, + one {Masz 1 wiadomość w skrzynce odbiorczej.} + few {Masz {COUNT} wiadomości w skrzynce odbiorczej.} + other {Masz {COUNT} wiadomości w skrzynce odbiorczej.} + +Masz {COUNT, plural, + one {1 wiadomość} + few {{COUNT} wiadomości} + other {{COUNT} wiadomości} +w skrzynce odbiorczej. + +``` + + +MIH: I think we always want to represent the maximum number of plural cases. Translators always should be shown the full set of translations, and let TM handle the redundancy. + +Now, if we want to collapse the representation in syntax to a more compact form, that can be handled by the TMS as necessary (ex: can’t afford the extra bandwidth caused by verbosity). + +Now, the duplication of identical message pattern cases in the translation source (in English) itself can be wrong because developers introduce typos and inconsistencies. + +DAF: It sounds like it’s not as developer friendly as we think, it’s just that developers religiously hate verbosity. + +SMY: On the pro side, scalability is better? + +MIH: In what way? + +SMY: You can encode very complex sentences in a succinct manner? + +MIH: This is just a restatement of verbosity. + +These are edge cases, really. I tried to count in our internal codebase how many messages have selectors, and it was maybe 2-3%, sometimes 5% depending on the application, so let’s say 5%. + +ZIB: I think the argument you made is a fallacy. There is a lot of tooling that make selectors discouraged and thus less likely. If we had more representation of selection, then it would make more natural sounding messages in other languages. English might be an exception where selection is less necessary and most other languages have more selection messages than English. + +SMY: So it’s a self-fulfilling prophecy to say that because we don’t have it, we don’t need it. + +DAF: It can cost you market share in Japan or Poland if you don’t have natural-sounding translations, even if for instance Czech Republic is more forgiving. + +ZIB: It’s hard to get signal on direct correlation of impact of translation and + +Closest thing to having data is Paypal. A person at Paypal was exploring using Fluent for translation, because they saw that the older users had less trust of the product if the translation was not natively written. + +SMY: I agree, although I wasn’t trying to be data-driven here. The trust issue is important, as a user of Polish language software. But the major problem isn’t about plural support so much as it is just bad translations. + +DAF: I agree with ZIB and SMY. I totally agree with the approach that English is just another language/translation. But I think that the translator must be given the whole message and all the lines she needs to serve all cases. + +MIH: I don’t disagree with anything that was said here. We can discuss whether the syntax is friendly or not to developers in certain ways. I don’t think the space saving considerations are significant. I don’t think these issues matter so much as whether we make it easy for translators or not? Does the data model support making things easy for translators? We can design syntaxes to be concise or verbose for developers however we want (ex: JS friendly, Java friendly). But should this be in the data model? + +SMY: I wanted to add one thing to the list of pros and cons. Both approaches can be difficult to debug. Look at the following example. If we translate this into Arabic, there are 1300 variants/cases. It’s likely to make small mistakes that create inconsistencies. + + +``` +activity-needed-calculation-plural = { NUMBER($totalHours) -> + [one] {$totalHours} hour + *[other] {$totalHours} hours +} is achievable in just over { NUMBER($periodMonths) -> + [one] {$periodMonths} month + *[other] {$periodMonths} months +} if { NUMBER($people) -> + [one] {$people} person + *[other] {$people} people +} record { NUMBER($clipsPerDay) -> + [one] {$clipsPerDay} clip + *[other] {$clipsPerDay} clips +} a day. +``` + + + +MIH: I don’t find this more readable, though. I think you end up with more errors than just spaces and commas, you get problems of sentence agreement (https://en.wikipedia.org/wiki/Agreement_(linguistics)). + +I still don’t see this example as an argument to represent nested selector messages in the data model. + +SMY: I’m trying to picture the scenario in 10 years from now, let’s say you have a UI, you have different use case scenarios. Can we support it? + +MIH: I don’t think the existence of new tooling will make a difference, because the linguistic concerns of sentence agreement will still be there. It still makes things mentally complex. + +SMY: This is a bit of an extreme case, yes, and I’m just trying to play devil’s advocate a little bit. + +Can you split the message into separate sentences, translate, and then combine somehow back into one sentence via concatenation? + +MIH: The problem is that the combination of sentence parts has to be managed by the translator. And sentences still need to agree: +“You deleted 21 files. You can recover them from the Recycle Bin” + +DAF: In XLIFF 1.2, support of segmentation was added as an afterthought, it was not well supported. So the model was changed in XLIFF 2. In the XLIFF 2 data model, you can do segmentation at the time of translation. It is something that can potentially be forced at the time of extraction of messages from the source, but it is not a general best practice, segmentation is a linguistic process, so it should happen in L10n tools not in the engineering Extractor. + +MIH: Look at slide 19 of my l10n concepts doc. Segmentation is localization level. Taking a full message for translation, it turns it into a single Text Unit. That Text Unit is split into segments. The only benefit of segmentation is increased leverage. But has drawbacks (for example, how to deal with spaces between sentences in CJK, Thai, etc.) + +DAF: As a supporting detail, according to the XLIFF standard, Text Units are indivisible, but it is okay for Segments inside a unit to be split or joined. + +MIH: A Text Unit cannot be altered, but it can be split up into whatever number of Segments, and it gets recombined back into a Text Unit when returning the translated Text Unit. + +Extraction produces Text Units, which is usually paragraphs. Segmentation converts Text Units into Segments, which are usually sentences. + +For example, if you have a source like, “There are 21 files. Together, they are 82 MB.” The word “they” refers to the 21 files. So the sentences have to be taken together. + +NIC: I see, so it is up to the author to keep the context of the sentences within a text unit. + +MIH: Are there any other pros and cons? + +NIC: Is one approach or another more compatible with TMSes? I have a gut feeling, but not sure. + +MIH: I have more than just gut feeling, I tried both and implemented internally at Google, the approach without internal selection is more compatible with TMSes. I worked on an internal TMS tool a couple of years ago, and it takes the full message approach. For plural message, it supports the representation of missing plural cases (English as ONE and OTHER, but Russian might have 4 cases, and the tool supports the expansion to the plural cases in the target language and Cartesian product of cases when there are multiple selectors in the source message). + +NIC: Perhaps that is important so that we avoid the pattern of trying to solve a problem with a solution but not actually solving it. + +DAF: Perhaps we can add to the goals a set of examples that show the mappings of different syntaxes to the (verbose) canonical data model (without internal selection) so that we can demonstrate to people the equivalence of the different implementations/syntaxes with the full message selection based data model. + +We can add these mappings to the goals document, for example “Mapping to Fluent”, “Mapping to ICU”. + +MIH: I have code that does that mapping algorithmically for ICU MessageFormat messages. And note, there is no concept of full message selection in ICU. Fluent is similar in that regard (of no full message selection), right? + +DAF: I think this is great progress. I think that we here can say that we all understand each other. And we have the pros and cons. We can make the notes of this meeting available, and provide a digestible summary to the full meeting. + +ECH: We also don’t have to provide just one proposal from the group, we could provide a small number of proposals along with the notes and summary to help the group think and decide. + +DAF: I would want us to sleep on it to not make a rash decision. After all, our decision making process gives all the power to decide to the monthly plenary meeting. + +RCA: I have been listening all along, and I agree with most of the opinions of the meeting, even if they diverge a little bit from each other. I like ZIB’s thought to think about 10 years in the future and think about what we have now. We can support existing tools. But I don’t think it is a correct way to limit the ability in the future of what we can represent so that we don’t regret things in 10 years’ time. + +NIC: What if we have an example with 3 selectors? Or 1000 selectors? + +MIH: Sure, there is no problem in the data model to represent 1000 selectors, and the tooling can be adapted to support that, too. The issue is that if we allow nested selectors, then we move the complexity out of the syntax and into the head of the translator. + +DAF: This brings me back to the question of are we trying to support natural language generation, or are we trying to create a rule based machine translation system? I don’t think we should try to create a rule based machine translation system. + +SMY: For the simple message example, _________ For example, if you have the message: + +“You have {X} available items.” + +In many languages, you need message-level selectors. + + +______ + + + +MIH: You cannot add this as a translator. ____ You can do this for gender. _____ You need to have a tool that supports messages that are formatted this way + + +SMY: This is a con of message-level selectors. Unless + +DAF: The other way around, it won’t be there, either. The linguist won’t be able to add options in their GUI. + +ZIB: It’s not necessary in Fluent. That would be terrible to force the translator to work with plurals for languages that don’t need it. Isolation of languages (per-language support) is one of the fundamental design principles of Fluent. + +Does this mean that every single message needs selection? + +MIH: If it is a plural message, then you will need selection. + +SMY: But there are messages with noun declensions (aka inflections) + +MIH: Messages without placeholders don’t need selection. + +SMY: But for messages with noun declensions, it sounds like + +DAF: Developers are responsible for making their messages internationalizable. They don’t need to know everything about the possible target languages to do so. + +ZIB: Are we talking about making all messages that are plurals into selection messages (?), or are we saying ___________. + +SMY: I think what we’re saying is that the data model for a message with a single placeholder is a message level selection with a single case/variant. + +DAF: A feature request that Steven Loomis filed for XLIFF 2.2 is to allow passing on semantic data for placeholders as metadata. I think it would be a great feature complementing the current core capability. + +ZIB: It is not a bug in the system that there is ambiguity in the system as to whether an argument should be an integer or a string or not. It is just + +I am less concerned about this direction. I am more concerned about the concern that DAF brought up with the example of gettext tooling, where if you have a message with plurals, you have to use a different call site and go off somewhere to fetch the message. + +SMY: I think the confusion is because I used the phrase “simple message”, but what I meant was _______. + +MIH: If you have a placeholder, and the placeholder is numeric, you should have validation that says that “you probably want a message level selection string”. + +SMY: So are we saying that any placeholder implies a complex message? + +MIH: I would say no. + +DAF: I can’t think of any example that wouldn’t. + +ECH: What about values coming in from runtime, example you have a date. They are represented via as a placeholder, but it doesn’t require selection. + +DAF: Well, depending on the use of the date in the sentence in Czech, it can either be in nominative or accusative case, so there is selection there. + +NIC: What about CJK languages that don’t have any plurals? + +MIH: Well, not supporting plurals properly is bad i18n. (?) + +DAF: I wanted to add as a follow up topic how to support the per-language message expansion in XLIFF. I would think that the only option is to create separate XLIFF files (with set @trgLang) for each locale with all the relevant cases covered.. + +MIH: I will send you a document that proposes supporting this in XLIFF. +Tools that wouldn’t be aware of the solution would need to create separate files for each target locale, as you said, but there could be all options in one XLIFF and L10n tools could filter the relevant ones by target locale. + +DAF: That sounds very clever, and should be possible to implement as a module, development of XLIFF 2.2 is just about to start.. + +MIH: Yes, it’s a module proposal for XLIFF 2.x (whichever it hits on time). I will send it your way. + +SMY: It would be good to create an executive summary. + +ECH: Fill in the blanks in the notes, too. + +SMY: When is the next meeting? It’s Sept 21 -- 3 weeks from now. + +RCA: Should we, on Slack, if you want to meet before the next plenary meeting, we can use that. + +SMY: We only talked about 2 items in our Tier 1 issues in our Task Force issues at https://github.com/unicode-org/message-format-wg/projects/3. We can meet again in 1-2 weeks to discuss the other 2 issues #103 and #106, which are really the same thing. + +MIH: I think #104 is the same thing, no? + +SMY: #104 is about how to represent selection in the data model. But #103 and #106 are different, and are about “do we allow” and “how”. + +RCA: Should we schedule a meeting to discuss those remaining topics on Sept 14? + +DAF: I was going to argue against meeting again in 2 weeks. If we meet in 2 weeks, we won’t have enough time to prepare, discuss, summarize and present to the main meeting afterwards.. Also I feel this should be finalized before progressing.. + +RCA: I think 2 weeks would be a good checkpoint if nothing else, so that we don’t allow 3 weeks to go by without more progress. + +MIH?: Sure, we can check if the meeting results are ready to present to the monthly meeting, let’s talk on Slack? + + +SMY?: Agree with DAF, there are dependencies, we shouldn’t progress before this is settled.. + +MIH: I volunteer for the first pass on the executive summary, should be checked by “the other camp” + +DAF: Slack sounds good to me, let’s add individual approval stamps for the executive summary.. + + + + + + + + + + + + +