Skip to content

Support for attributes with different dtypes when serialising to netcdf4 #1500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lewisacidic opened this issue Aug 3, 2017 · 4 comments

Comments

@lewisacidic
Copy link
Contributor

At the moment, bool and dates aren't supported as attributes when serializing to netcdf4:

>>> da = xr.DataArray(range(5), attrs={'test': True})
>>> da 
<xarray.DataArray (dim_0: 5)>
array([0, 1, 2, 3, 4])
Dimensions without coordinates: dim_0
Attributes:
    test:     True

>>> da.to_netcdf('test_bool.nc')
...
TypeError: illegal data type for attribute, must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got b1

>>> da = xr.DataArray(range(5), attrs={'test': pd.to_datetime('now')})
>>> da
<xarray.DataArray (dim_0: 5)>
array([0, 1, 2, 3, 4])
Dimensions without coordinates: dim_0
Attributes:
    test:     2017-08-03 13:02:29

>>> da.to_netcdf('test_dt.nc')
...
TypeError: Invalid value for attr: 2017-08-03 13:02:29 must be a number string, ndarray or a list/tuple of numbers/strings for serialization to netCDF files

I assume bool attributes aren't supported by netcdf4-python and dates are difficult (could always just write these as a string), but this would be really nice to have if possible.

As an aside, using h5netcdf works for bools, but coerces them to int64.

@shoyer
Copy link
Member

shoyer commented Aug 3, 2017

These aren't supported by the netCDF4 file format, so there's not a straightforward way to add them.

We could add some custom serialization logic (like we do for variable dtypes like bool), but without the ability to put an attribute on an attribute I don't see how we could round-trip this safely.

I suppose as a last resort we could write such attributes as bytestrings in some custom serialization format, but then it has the downside of not being readable by any other tool.

@shoyer
Copy link
Member

shoyer commented Aug 3, 2017

I guess this could work as part of your PR #1421 to support using pickle for array values.

@stale
Copy link

stale bot commented Jul 19, 2019

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

@stale stale bot added the stale label Jul 19, 2019
@stale stale bot closed this as completed Aug 18, 2019
@half-adder
Copy link

Still relevant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants