time: decimal comma not supported in fractional seconds

[gopherbot]

by dmitri.m:

What steps will reproduce the problem?
If possible, include a link to a program on play.golang.org.
The Go time package expects the decimal mark in fractional seconds to always be a dot.
Attempting to parse a time string that uses a decimal comma results in an error:
http://play.golang.org/p/d8qQasN0z1

Decimal commas are standard in many locales, but there is currently no way to parse the
fractional part of the second in a time string if it's separated by a comma.
https://en.wikipedia.org/wiki/Decimal_mark#Countries_using_Arabic_numerals_with_decimal_comma

What is the expected output?
In case the provided layout also uses a decimal comma, the time string should be parsed
without errors.

What do you see instead?
parsing time "2013-08-19 22:56:01,234" as "2006-01-02 15:04:05,000":
cannot parse "234" as ",000"

Which compiler are you using (5g, 6g, 8g, gccgo)?
6g

Which operating system are you using?
OS X 10.8.4

Which version are you using?  (run 'go version')
go version devel +5037426bea2f Mon Aug 19 23:09:24 2013 +0400 darwin/amd64

Please provide any additional information below.
Note that (*time.Time).Format() currently outputs decimal commas just fine since it
doesn't see them as a special character. But making the straightforward change to
support decimal commas in Parse() - by adding them to the stdFracSecond{0,9} classes -
breaks this due to hardcoded '.' in formatNano().

[robpike]

Comment 1:

Here is what I wrote in mail responding to a CL to fix this:
Thanks for filing the issue but I might prefer to leave it unaddressed
for the moment. Proper locale treatment is coming (I hope soon) and this
seems like the wrong place to start insinuating special locale handling
into the standard library. Strconv can't do it yet, for instance.
I'll mark the bug as to be fixed for Go 1.3.

Labels changed: added go1.3, removed go1.2maybe.

[robpike]

Comment 2:

Status changed to Accepted.

[robpike]

Comment 3:

Labels changed: added priority-soon, removed priority-triage.

[robpike]

Comment 4:

Labels changed: removed go1.3.

[rsc]

Comment 5:

Labels changed: added go1.3maybe.

[rsc]

Comment 6:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 7:

Labels changed: added repo-main.

[gopherbot]

Comment 8 by phmmnd:

I'm not sure this is just a localization issue.
According to http://en.wikipedia.org/wiki/ISO_8601 the latest version (ISO 8601:2004(E))
suggests that a comma is the preferred separator, which means some software will use a
comma even when not localized.
As a real life use case, I'm currently trying to parse a log file generated by log4j
where the most commonly used date formatters use a comma (eg:
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/helpers/ISO8601DateFormat.html)

[gopherbot]

Comment 9 by mike.gaffney:

I am running into the same issue as described in comment #8. The default format for
log4j 1.2.x uses a comma. Here is an example copied from one of our log files: 
2014-04-02 12:00:57,376
There's a lot of software using log4j out there.

[actgardner]

Would a PR be welcome to add this functionality? We have a workaround in our codebase but it'd be great to have support in the standard library.

[bradfitz]

@alanctgardner, if you already have a fix, feel free to send it so we can see how invasive it might be.

@mpvl, this bug was punted in 2013 because proper locale support was coming. Is it still coming?

[gopherbot]

CL https://golang.org/cl/23063 mentions this issue.

[mpvl]

Date localization is definitely still in the planning, but fairly low on the priority list at the moment. Extraction, segmentation, numbers, among other things are higher on the list right now.

That said, I think it is not unreasonable for the standard lib to support case 2 of the play example:
{"2006-01-02 15:04:05,000", "2013-08-19 22:56:01,234"}, // decimal comma
as it is unrelated to localization, I would say, and a localization that wants to build upon the existing time package would need such support.

The third case:
{"2006-01-02 15:04:05.000", "2013-08-19 22:56:01,234"},
seems to be more a localization feature.

[actgardner]

That makes sense to me - the patch I uploaded doesn't support using a comma where a period is specified in the format string (and vice versa). That does seem like localization.

The patch does support the case where there's no fractional seconds part in the format but there are fractional seconds with a comma in the string, like:

{"2006-01-02 15:04:05", "2013-08-19 22:56:01,234"}

but I would be willing to remove that if it's too localization-related. The real crux of the issue is that there's no way to parse a date that uses a comma as a delimiter using any format string.

[cprima]

I am completely new to Go, thanks for this Issue which saved me a ton of time.

I ended up strings-Replace'ing the log4j-comma.
Ugly, but in this code of mine are uglier things for sure! ;)

[AlekSi]

In https://golang.org/cl/23063 @robpike said:

While I understand the desire, this is not the right fix. As listed in the issue comments, the right answer is proper localization support and that should come soon. I hope.

But hacking it up like this in various places is not the answer. It sets a bad precedent.

But the next day @mpvl said there:

That said, I think it is not unreasonable for the standard lib to support case 2 of the play example:
{"2006-01-02 15:04:05,000", "2013-08-19 22:56:01,234"}, // decimal comma
as it is unrelated to localization, I would say, and a localization that wants to build upon the existing time package would need such support.

That sends a mixed message. Can you please decide about if you want to support 2006-01-02 15:04:05,000 as layout?

My use-case: I parse supervisord log, and there is no way to configure timestamps format. strings.Replace works, but ugly.

[raneq]

I am using @grafana 's promtail to parse logs from java, python and go applications. Not all of them use the same format that go assumes as "neutral" and "unlocalized".

@AlekSi and @actgardner have strong points

I could expect apps not being flexible enough to configure log formatting, but not the programming language behind any of them.

IMO, little bugs like this make golang much less attractive for developers looking for a versatile language.

Regarding localization, claiming that using a english standard like decimal dot is "unlocalized" sounds pretty colonialist, and the fact that "proper localization" is still coming, tells to me that diversity for different languages and cultures is totally not a priority for @golang

As constructive data, I analyzed a bit the linux /usr/share/i18n/locales/ dir and found that the most used separators are:

    208 "."
    134 ","
      7 "<U002E>" -> same as regular ascii dot
      3 "<U066B>" -> arabic decimal separator ٫

So adding latin and arabic comma would be a great step.

[RobbanHoglund]

I have the same problem with colon delimiter before fraction of a second.
2020-06-12_11:48:20:466

cannot parse "466" as ".000""

[ChangSurrey]

I find it hard to understand why this is a localization issue.

Using comma as the millisecond delimiter is the default format used by log4j2.

https://logging.apache.org/log4j/2.x/manual/garbagefree.html#Layouts

[jvsteiner]

This is absolutely not a localization issue. It's a developer usability issue. let's say I have a timestamps like the following:

2020/6/17 06:31:59.573
2020/6/17 06:31:59,573
2020/6/17 06:31:59:573
2020/6/17 06:31:59|573

The first one, I can parse, the rest I cannot. It doesn't matter at all that some other maniac developer is feeding me a timestamp format that makes no sense at all, and is non standard as hell. The pattern matching technique in go is just bad, and inflexible. Microsoft Excel has a better mechanism, see: here.

Get it together people.

[networkimprov]

Now under consideration in proposal: #36145

[gopherbot]

Change https://golang.org/cl/300996 mentions this issue: time: support "," as separator for fractional seconds

[gopherbot]

Change https://golang.org/cl/345438 mentions this issue: time: propagate "," separator for fractional seconds into Format

[gopherbot]

Change https://golang.org/cl/350149 mentions this issue: [release-branch.go1.17] time: propagate "," separator for fractional seconds into Format

[dsnet]

Unfortunately, this change broke compliance with RFC 3339, where we now permit , when the RFC specifies that it should not. See #54571.

Given that RFC 3339 is more common than ISO 8601, I think it was a mistake to loosen this behavior. In my opinion it would have been better to introduce a , into the format where it means either , or . and left . to mean exactly ..

[gopherbot]

Change https://go.dev/cl/425045 mentions this issue: time: fix Parse regression of sub-second separator in RFC3339