Skip to content

gh-91581: Check result of utc_to_seconds #91582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 12, 2022

Conversation

pganssle
Copy link
Member

This result could fail. On Linux here's a minimal reproducer:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only because subsequent calculations happened to fail as well. Better to fail fast.

Fixes GH-91581

@pganssle pganssle requested a review from abalkin as a code owner April 15, 2022 17:18
@ghost
Copy link

ghost commented Apr 15, 2022

All commit authors signed the Contributor License Agreement.
CLA signed

@pganssle pganssle closed this Apr 15, 2022
@pganssle pganssle reopened this Apr 15, 2022
@pganssle pganssle force-pushed the check_results_seconds_error branch from 9113132 to c22c81d Compare April 16, 2022 00:25
@pganssle pganssle force-pushed the check_results_seconds_error branch from c22c81d to 0242b88 Compare May 9, 2022 17:26
Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it does not have user-facing effect, it does not need a NEWS entry. If it changes the error message, it would be worth to add a test for this.

@serhiy-storchaka
Copy link
Member

I would add tests for Dec 30, 1 BCE and Dec 31, 1 BCE.

@pganssle pganssle force-pushed the check_results_seconds_error branch from 0242b88 to 7a0af95 Compare May 11, 2022 17:11
@pganssle
Copy link
Member Author

Thanks for the review Serhiy!

If it does not have user-facing effect, it does not need a NEWS entry. If it changes the error message, it would be worth to add a test for this.

It changes the error message but not in any way that I am prepared to make assertions about — this is a mostly cosmetic change for consistency. I'm not even sure that the error message is the same across the C and Python implementations.

I generally like to add a NEWS entry for code changes anyway, in case the change had an unexpected effect and someone is looking for hints as to what might have caused the behavior change in the changelog.

I would add tests for Dec 30, 1 BCE and Dec 31, 1 BCE.

Done, and when doing that I refactored the relevant tests to get us a lot closer to the actual limits we're expecting. This turned up some other issues around these edge cases, so I've added this comment to #91012. Fixing that should let us make these tests a lot cleaner.

As part of the refactor I ended up tweaking the pure Python implementation of utcfromtimestamp, since it was erroneously erroring out in cases where the C version wasn't.

Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions and minor suggestions.

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes pythonGH-91581
@pganssle pganssle force-pushed the check_results_seconds_error branch from 28d8ca1 to 55a77c1 Compare May 12, 2022 14:16
@pganssle pganssle added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels May 12, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @pganssle for commit 55a77c1 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 12, 2022
@pganssle pganssle added the extension-modules C modules in the Modules dir label May 12, 2022
@pganssle
Copy link
Member Author

Buildbot failures seem unrelated:

  • buildbot/AMD64 FreeBSD Shared PR — Build done. Details
  • buildbot/x86 Gentoo Installed with X PR — Build done. Details
  • buildbot/x86 Gentoo Non-Debug with X PR — Build done. Details

I'm going to go ahead and merge this.

@pganssle pganssle merged commit 83c0247 into python:main May 12, 2022
@miss-islington
Copy link
Contributor

Thanks @pganssle for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10, 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 12, 2022
…thonGH-91582)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes pythonGH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
@bedevere-bot
Copy link

GH-92745 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 12, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 12, 2022
…thonGH-91582)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes pythonGH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label May 12, 2022
@bedevere-bot
Copy link

GH-92746 is a backport of this pull request to the 3.10 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 12, 2022
…thonGH-91582)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes pythonGH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.9 only security fixes label May 12, 2022
@bedevere-bot
Copy link

GH-92748 is a backport of this pull request to the 3.9 branch.

miss-islington added a commit that referenced this pull request May 14, 2022
…-91582)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes GH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
miss-islington added a commit that referenced this pull request May 14, 2022
…-91582)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes GH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
ambv pushed a commit that referenced this pull request May 16, 2022
…-91582) (GH-92748)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes GH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
hello-adam pushed a commit to hello-adam/cpython that referenced this pull request Jun 2, 2022
…thonGH-91582) (pythonGH-92748)

The `utc_to_seconds` call can fail, here's a minimal reproducer on
Linux:

TZ=UTC python -c "from datetime import *; datetime.fromtimestamp(253402300799 + 1)"

The old behavior still raised an error in a similar way, but only
because subsequent calculations happened to fail as well. Better to fail
fast.

This also refactors the tests to split out the `fromtimestamp` and
`utcfromtimestamp` tests, and to get us closer to the actual desired
limits of the functions. As part of this, we also changed the way we
detect platforms where the same limits don't necessarily apply (e.g.
Windows).

As part of refactoring the tests to hit this condition explicitly (even
though the user-facing behvior doesn't change in any way we plan to
guarantee), I noticed that there was a difference in the places that
`datetime.utcfromtimestamp` fails in the C and pure Python versions, which
was fixed by skipping the "probe for fold" logic for UTC specifically —
since UTC doesn't have any folds or gaps, we were never going to find a
fold value anyway. This should prevent some failures in the pure python
`utcfromtimestamp` method on timestamps close to 0001-01-01.

There are two separate news entries for this because one is a
potentially user-facing change, the other is an internal code
correctness change that, if anything, changes some error messages. The
two happen to be coupled because of the test refactoring, but they are
probably best thought of as independent changes.

Fixes pythonGH-91581
(cherry picked from commit 83c0247)

Co-authored-by: Paul Ganssle <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extension-modules C modules in the Modules dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error condition in _datetimemodule.c is unchecked with datetime.fromtimestamp on local times > Y10k
4 participants