Skip to content

[WIP] Support nano second time encoding. #4400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

hmaarrfk
Copy link
Contributor

@hmaarrfk hmaarrfk commented Sep 2, 2020

Not too sure i have the bandwidth to complete this seeing as cftime and datetime don't have nanoseconds, but maybe it can help somebody.

  • Closes Unable to decode a date in nanoseconds #4183
  • Tests added
  • Passes isort . && black . && mypy . && flake8
  • User visible changes (including notable bug fixes) are documented in whats-new.rst
  • New functions/methods are listed in api.rst

@aldanor
Copy link

aldanor commented Nov 27, 2020

Is there anything preventing to merge this?

@aldanor
Copy link

aldanor commented Nov 27, 2020

Oh, that requires cftime._cftime support first? :/

@aldanor
Copy link

aldanor commented Nov 27, 2020

But the test already passes (i.e. you can at least do a .encoding={.... 'nanoseconds'} and avoid float conversion?

@hmaarrfk
Copy link
Contributor Author

yeah, i'm not too sure. I think the idea is that this breaks compatibility with netcdf times, so the resulting file is thus not standard.

For my application, us timing is enough.

@aldanor
Copy link

aldanor commented Nov 29, 2020

I think netcdf lists "nanoseconds" as a valid unit though?

@aldanor
Copy link

aldanor commented Nov 29, 2020

I'm working on an application where nanosecond-resolution is critical and took me days to find why my timestamps are all scrambled or off-by-1 after I write them with xarray and them read them back... would probably much rather prefer if it threw an exception instead of corrupting your data silently.

Non-standard netcdf or not, if it was possible to just store them as plain int64s and read them back as is, that would help a ton...

@hmaarrfk
Copy link
Contributor Author

I think you should be able to define your own custom encoder if you want it to be a datetime.

But inevitably, you will have to define your own save and load functions.

Python, by definition of being such a loose language, allows you to do things that the original developers never really imagined.

this can sometimes lead to silent corruption.like the one you've experienced.

@aldanor
Copy link

aldanor commented Nov 30, 2020

Yea, well, in this case it's not about Python... M8[ns] datatype is simply an int64 underneath, why not just store it as that, no bells and whistles required, no corruption possible, no funky conversions?

@dcherian
Copy link
Contributor

The merge conflict suggests this was fixed on main by #4684

@dcherian dcherian closed this Mar 26, 2023
@hmaarrfk
Copy link
Contributor Author

nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to decode a date in nanoseconds
3 participants