No way to generate or parse timezone as produced by datetime.isoformat()

[gvanrossum]

BPO	24954
Nosy	@tim-one, @warsaw, @jcea, @abalkin, @vstinner, @jwilk, @mcepl, @merwok, @bitdancer, @karlcow, @flying-sheep, @mihaic, @Fak3, @berkerpeksag, @vadmium, @ztane, @boxed, @jstasiak, @offby1, @deronnax, @pbryan, @pganssle, @sirex, @mariocj89, @sonots
Dependencies	bpo-31800: datetime.strptime: Support for parsing offsets with a colon
Files	datetime.patch datetime2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2015-08-28.22:34:41.660>
labels = ['extension-modules', 'easy', 'type-feature', '3.7']
title = 'No way to generate or parse timezone as produced by datetime.isoformat()'
updated_at = <Date 2018-07-05.19:25:05.940>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2018-07-05.19:25:05.940>
actor = 'gvanrossum'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2015-08-28.22:34:41.660>
creator = 'gvanrossum'
dependencies = ['31800']
files = ['40675', '40692']
hgrepos = ['319']
issue_num = 24954
keywords = ['patch', 'easy']
message_count = 24.0
messages = ['249305', '249815', '252284', '252294', '252364', '252366', '252449', '270530', '270531', '270543', '270573', '270575', '270579', '270826', '299213', '299237', '299239', '299241', '299333', '299339', '304596', '305015', '305016', '321108']
nosy_count = 36.0
nosy_names = ['tim.peters', 'barry', 'jcea', 'roysmith', 'belopolsky', 'nagle', 'vstinner', 'jwilk', 'mcepl', 'eric.araujo', 'Arfrever', 'r.david.murray', 'davydov', 'cvrebert', 'karlcow', 'SilentGhost', 'perey', 'flying sheep', 'mihaic', 'aymeric.augustin', 'Roman.Evstifeev', 'berker.peksag', 'martin.panter', 'piotr.dobrogost', 'kirpit', 'ztane', 'Anders.Hovm\xc3\xb6ller', 'jstasiak', 'Eric.Hanchrow', 'deronnax', 'pbryan', 'shanmbic', 'p-ganssle', 'sirex', 'mariocj89', 'sonots']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'commit review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24954'
versions = ['Python 3.7']

[gvanrossum]

The datetime isoformat() function for an aware datetime appends the timezone in the form +HH:MM or -HH:MM. But the %z format produces (strftime) or parses (strptime) either +HHMM or -HHMM. I looked it up on Wikipedia, and the ISO 8601 standard indeed uses the colon. It would be nice if there was a new format character that could produce or parse the colon...

[abalkin]

This request is similar to (if not a duplicate of) bpo-15873.

Gnu date uses the following extensions to %z instruction of strftime/strptime:

   %z     +hhmm numeric timezone (e.g., -0400)

   %:z    +hh:mm numeric timezone (e.g., -04:00)

   %::z   +hh:mm:ss numeric time zone (e.g., -04:00:00)

   %:::z  numeric time zone with : to necessary precision (e.g., -04, +05:30)

I think it is reasonable to add those to python.

[shanmbic]

I modified the "format_utcoffset" function in "_datetimemodule.c", to accept one more parameter "secondsrequired" . It is a boolean variable (PyObject) , which when set to true, the function will return the offset formatted as "+HH:MM:SS" or "-HH:MM:SS". I also modified the "wrap_strftime" function to also accept the strings "%:z" and "%::z" as format specifiers. This is my first contribution to python , so my approach of modifying a function's default parameters might not be correct. Any help is appreciated
Thanks

[vadmium]

I added some comments on Rietveld. I guess the documentation should get notices and What’s New entries saying it is new in 3.6. Test cases would be good, including negative ones to check error handling is sensible if the Z is missing. Also shanmbic, perhaps look at signing the contributor agreement <https://www.python.org/psf/contrib/\> if you haven’t already.

Adding to the list of format codes means that the statement at the top of the list about them all being required by C89 will need fixing.

For parsing, perhaps the existing %z code could be extended to accept colons, without needing to specify a new %:z code. Although if %:z is added for formatting, it should also be supported for parsing.

I am not convinced that it is worth adding %::z or %:::z to Python. The documentation of tzinfo.utcoffset() says it returns the offset “in minutes”, hinting that sub-minute offsets are not supported. RFC 3339 acknowledges that sub-minute offsets exist in history, but AFAIK neither RFC 3339 nor ISO 8601 support them. So I think it is too specialized to build %::z into Python. And I don’t imagine the %:::z necessary precision version would be used much either.

[shanmbic]

Yeah , I agree that including '%:::z' might not be necessary at all . Including '%:z' and '%::z' format specifiers will give an upper edge to Python. Although , if not going with '%::z' or seconds offset, for now only '%:z' can be a valuable addition .
I have rectified my code, corrected the indentations and have added cases , where it will also handle any strings of form '%:z' and '%::z' i.e. any character other than 'z' also raising exceptions for cases "%:" or "%::", where no character is specified (Invalid string).
In case , if we only go with '%:z' format specifier , I will remove the code for '%::z'.
P.S. I have filled the contributor agreement form. Thanks for the awesome help here. :)

[abalkin]

I think it is very likely that bpo-5288 will result in at least seconds being allowed in UTC offsets in Python 3.6. I am assuming that much in my PEP-495 work in bpo-24773.

[vadmium]

Okay I am happy to be wrong about Python allowing seconds resolution.

I notice that bpo-12006 was committed, which adds a second table of Python-only codes for ISO 8601 values. The proposed entries here could be incorporated into that table, and the text above it modified to suit. That would solve my C89 library problem.

[abalkin]

Since this is closely related to bpo-15873, I am merging the nosy lists.

As far as I can tell, the patch just needs tests and a final decision on bpo-5288. I don't think it is at all controversial, but we need to relax the offset restrictions before ::z code can go in.

shanmbic, you can use my PEP-495 branch to test. See <https://github.com/abalkin/cpython.git#bpo-24773-s3\>.

[abalkin]

Sorry, the tracker messed up the URL: go to

https://github.com/abalkin/cpython.git

and select branch:bpo-24773-s3

[ztane]

"Be conservative in what you do, be liberal in what you accept from others" they say. Also Z as a timezone designator is also widely used in iso 8601 timestamps. I believe the effort should be made to *parse* *any/all* of the ISO 8601 supported time-zone codes with one conversion, the list is not that long, just 'Z', HH, HH:MM, HHMM, longest match. Literal 'Z' really does not need to be supported for *output* at all, but for input, please.

Otherwise this will still go down the road of iso8601 library, which just tries to support all the YYmmddTHHMMSS.FFFFFFzzzz variants. It uses regular expressions to parse the dates as it is faster than trying N different formats with strptime

[abalkin]

"Be conservative in what you do, be liberal in what you accept from others"

I would agree to this approach with respect to a proposed fromisoformat() method (see bpo-15873), but setptime seems to be about specifying an exact format. For example, "%Y%m%d" will not accept "2016-07-16".

[ztane]

Alexander: that is true, because they are *separate* conversion flags.

However even the POSIX standard strptime has some leniency: '%mand%daccept the numbers *without* leading zeroes. This actually also means that one cannot use%Y%m%d` to detect an invalid ISO timestamp:

    >>> datetime.datetime.strptime('111122', '%Y%m%d')
    datetime.datetime(1111, 2, 2, 0, 0)

The arrow library depends on the supposed "strict" behaviour of strptime that has never been guaranteed, which often results in very buggy behaviour under some conditions.

----

(Also, it must be noted that GNU date program doesn't use these formats to *parse* dates, and POSIX strptime in *C* library outright ignores any timezone information)

[abalkin]

Antti, while I see some convenience in making %z parsing promiscuous, there is clear utility in adding %:z to strftime. If we do that, not allowing the same for parsing will be odd. Let's start with that. A case for a promiscuous %z can be made later.

On a separate issue, note that datetime.isoformat() has recently grown a timespec option. See bpo-19475. I wonder if it would make sense to add a tzspec option to control the way timezone is formatted.

[boxed]

The arrow library depends on the supposed "strict" behaviour of strptime that has never been guaranteed, which often results in very buggy behaviour under some conditions.

Well… the arrow library accepts all sorts of broken input and gives you a date back. I think they even think that’s a feature and not a bug so no use in trying to help people who are adamant that they don’t want to be helped :/

[sonots]

Any progress here? I want %:z strftime.

By the way, ruby strptime works as:

irb(main):008:0* Time.strptime('2015-01-01T01:01:01+09:00', '%Y-%m-%dT%H:%M:%S%z')
=> 2015-01-01 01:01:01 +0900
irb(main):009:0> Time.strptime('2015-01-01T01:01:01+0900', '%Y-%m-%dT%H:%M:%S%z')
=> 2015-01-01 01:01:01 +0900
irb(main):010:0> Time.strptime('2015-01-01T01:01:01UTC', '%Y-%m-%dT%H:%M:%S%z')
=> 2015-01-01 01:01:01 UTC
irb(main):011:0> Time.strptime('2015-01-01T01:01:01Z', '%Y-%m-%dT%H:%M:%S%z')
=> 2015-01-01 01:01:01 UTC
irb(main):020:0> Time.strptime('2015-01-01T01:01:01Asia/Tokyo', '%Y-%m-%dT%H:%M:%S%z')
=> 2015-01-01 01:01:01 +0900

irb(main):015:0* Time.strptime('2015-01-01T01:01:01+09:00', '%Y-%m-%dT%H:%M:%S%Z')
=> 2015-01-01 01:01:01 +0900
irb(main):016:0> Time.strptime('2015-01-01T01:01:01+0900', '%Y-%m-%dT%H:%M:%S%Z')
=> 2015-01-01 01:01:01 +0900
irb(main):017:0> Time.strptime('2015-01-01T01:01:01UTC', '%Y-%m-%dT%H:%M:%S%Z')
=> 2015-01-01 01:01:01 UTC
irb(main):018:0> Time.strptime('2015-01-01T01:01:01Z', '%Y-%m-%dT%H:%M:%S%Z')
=> 2015-01-01 01:01:01 UTC
irb(main):019:0> Time.strptime('2015-01-01T01:01:01Asia/Tokyo', '%Y-%m-%dT%H:%M:%S%Z')
=> 2015-01-01 01:01:01 +0900

[gvanrossum]

Alexander, can you summarize the status of this issue? Maybe we can move forward for 3.7?

[abalkin]

This issue is waiting for the final decision on bpo-5288. If sub-minute offsets support is accepted, I still don't think we need %::z because %:z can simply add non-zero :seconds as needed. There are also some review comments on the latest patch that have not been addressed. Overall this is an easy issue and it would be quite doable to get it resolved for 3.7.

[gvanrossum]

Please do! I have no opinion on %::z but maybe you can find inspiration in the Zen of Python. :-)

[nagle]

As the original author of the predecessor bug report (bpo-15873) in 2012, I would suggest that there's too much bikeshedding here. I filed this bug because there was no usable ISO8601 date parser available. PyPi contained four slightly different buggy ones, and three more versions were found later.

I suggested following RFC3339, "Date and Time on the Internet: Timestamps", section 5.6, which specifies a clear subset of ISO8601. Five years later, I suggest just going with that. Fancier variations belong in non-standard libraries.

Date parsing should not be platform-dependent. Using an available C library was convenient, but not portable.

Let's get this done.

[abalkin]

John,

An RFC3339 parser is beyond the scope of this issue which is limited to adding str[fp]time code(s) to produce and consume RFC3339-formatted timezones.

We can still have fromisoformat() constructor implemented in 3.7, but someone needs to address the issues raised at bpo-15873.

[mariocj89]

Wrote https://bugs.python.org/issue31800 without realising this issue was open (Thanks for bringing it up Martin Panter).

bpo-31800 basically just adds the ability to parse NN:NN to the already existing python isoformat function when %z is specified.
Rather than adding "%:z".

As available as well in linux strptime: http://man7.org/linux/man-pages/man3/strptime.3.html

I'd really like to see a way to parse isoformatted dates and this is the only thing in the middle. Happy to continue with the other issue/PR or help out here if needed.

Thanks! ^^

[abalkin]

I am going to merge PR 4015 submitted for bpo-31800. That issue asks to make %z strptime format accept offsets with : separators. Given that a similar feature has been added to glibc several years ago, I don't see much need for bikeshedding. This issue will remain open and will focus on strftime.

[gvanrossum]

SGTM.

[pganssle]

I believe this can be consolidated with bpo-15873 and closed, since that is finished and available in Python 3.7.

[ThomasWaldmann]

Seems like there is still no way to have a custom format (like something including the suggested %:z) that generates the tzoffset in the same way as datetime.isoformat() would generate it: +HH:MM.

Thus, see topic of this issue, this can't be closed yet.

[ThomasWaldmann]

dt = datetime.now().astimezone()
dt.isoformat()
'2022-08-13T20:13:25.892454+02:00'
dt.strftime("%z")
'+0200'
dt.strftime("%:z")
':z'

[gvanrossum]

Indeed. There are some old patches attached to this diff. Would you like to clean one of those up and submit it as a PR?

[ThomasWaldmann]

@gvanrossum thanks for pointing me to the patches.

I had a look at them, but then decided to implement in a slightly different and easier way. PR coming soon.

[gvanrossum]

Closed by gh-95983. Thanks again TW!