Changing dtype on v0.13.0 causes Dataset attributes to be lost

[robbibt]

MCVE Code Sample

import numpy as np
import pandas as pd
import xarray as xr

np.random.seed(123)

times = pd.date_range("2000-01-01", "2001-12-31", name="time")
annual_cycle = np.sin(2 * np.pi * (times.dayofyear.values / 365.25 - 0.28))

base = 10 + 15 * annual_cycle.reshape(-1, 1)
tmin_values = base + 3 * np.random.randn(annual_cycle.size, 3)
tmax_values = base + 10 + 3 * np.random.randn(annual_cycle.size, 3)

ds = xr.Dataset({"tmin": (("time", "location"), tmin_values),
                 "tmax": (("time", "location"), tmax_values),},
                {"time": times, "location": ["IA", "IN", "IL"]})

# Assign an attribute
ds = ds.assign_attrs(CRS = 'EPSG:4326')

# Change dtype
ds.astype(np.float32)

Expected Output

ds to be returned with variables of dtype np.float32, with attributes (e.g. CRS = 'EPSG:4326') still included in the dataset.

Problem Description

On xarray version 0.12.1, changing the dtype of a dataset preserves any attached attributes, e.g:

<xarray.Dataset>
Dimensions:   (location: 3, time: 731)
Coordinates:
  * location  (location) <U2 'IA' 'IN' 'IL'
  * time      (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2001-12-31
Data variables:
    tmin      (time, location) float32 -8.03737 -1.7884412 ... -4.543927
    tmax      (time, location) float32 12.980549 3.3104093 ... 3.8052793
Attributes:
    CRS:      EPSG:4326

However, on xarray version 0.13.0, changing the dtype of a dataset silently drops any attached attributes, e.g:

<xarray.Dataset>
Dimensions:   (location: 3, time: 731)
Coordinates:
  * time      (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2001-12-31
  * location  (location) <U2 'IA' 'IN' 'IL'
Data variables:
    tmin      (time, location) float32 -8.03737 -1.7884412 ... -4.543927
    tmax      (time, location) float32 12.980549 3.3104093 ... 3.8052793

This causes issues with large geospatial analyses (e.g. OpenDataCube workflows), as we need to change dtype to reduce memory, but also preserve CRS information that is used for downstream tools.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.8 (default, Jan 14 2019, 11:02:34) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] python-bits: 64 OS: Linux OS-release: 4.14.133-113.112.amzn2.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C.UTF-8 LANG: None LOCALE: en_US.UTF-8 libhdf5: 1.10.0 libnetcdf: 4.6.0

xarray: 0.13.0
pandas: 0.24.2
numpy: 1.16.2
scipy: 1.3.1
netCDF4: 1.3.1
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.24
cfgrib: None
iris: None
bottleneck: None
dask: 2.3.0
distributed: 2.3.2
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 40.6.3
pip: 19.2.3
conda: None
pytest: 3.5.0
IPython: 7.8.0
sphinx: None

[dcherian]

Should we switch keep_attrs to True by default or be determined by the global keep_attrs setting? (also I don't know why this worked in 0.12.1)

xarray/xarray/core/dataset.py

Lines 4388 to 4401 in 6ece6a1

	@staticmethod
	def _unary_op(f, keep_attrs=False):
	@functools.wraps(f)
	def func(self, *args, **kwargs):
	variables = OrderedDict()
	for k, v in self._variables.items():
	if k in self._coord_names:
	variables[k] = v
	else:
	variables[k] = f(v, *args, **kwargs)
	attrs = self._attrs if keep_attrs else None
	return self._replace_with_new_dims(variables, attrs=attrs)
	return func

The more general solution for DataArrays is to finish up #2070 (see also #2049). @robbibt are you you up for doing this?

[dcherian]

I guess the simpler solution is to just add an astype method to Dataset and specify keep_attrs=True like

xarray/xarray/core/dataset.py

Lines 5062 to 5063 in 6ece6a1

	def real(self):
	return self._unary_op(lambda x: x.real, keep_attrs=True)(self)

[max-sixty]

Should we switch keep_attrs to True by default or be determined by the global keep_attrs setting? (also I don't know why this worked in 0.12.1)

Do we have a view that this should be False by default, or is it the case for historical reasons?

[shoyer]

It's also not clear to me how this managed to change/break but I think we should consider it a bug.

I agree that the right fix is to write an explicit astype method, but I don't think there's a good reason for keep_attrs=False in this case. astype isn't really changing the data, just changing how the data is represented so I think it makes sense to always preserve attributes.

[davidggphy]

The problem is still there, as it is quite unexpected.

I hacked it using .astype() directly in the np.array

Test code

import xarray as xr

da = xr.DataArray(
    np.arange(56).reshape((7, 8)),
    coords={
        'x': list('abcdefg'),
        'y': 10 * np.arange(8)
        },
    dims=['x', 'y'],
    attrs={"at_str":"at1_value","at_float":123.,"at_int":123}
)

print("Original xr.DataArray")
print(f"attrs: {da.attrs}")
print(f"dtype: {da.dtype}")

print(f"\nAfter casting to int64: {da.astype(np.int64).attrs}" )
print(f"After casting to int32: {da.astype(np.int32).attrs}" )
print(f"After casting to float32: {da.astype(np.float32).attrs}")

da.values = da.values.astype(np.float32)
print("\nChanging the np.array")
print(f"attrs: {da.attrs}")
print(f"dtype: {da.dtype}")

Output:

Original xr.DataArray
attrs: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
dtype: int64

After casting to int64: {}
After casting to int32: {}
After casting to float32: {}

Changing the np.array
attrs: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
dtype: float32

xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 14:38:56) 
[Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 18.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.6.2

xarray: 0.15.0
pandas: 1.0.1
numpy: 1.18.1
scipy: 1.3.0
netCDF4: 1.5.1.2
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: None
distributed: None
matplotlib: 3.1.0
cartopy: None
seaborn: 0.9.0
numbagg: None
setuptools: 41.0.1
pip: 19.1.1
conda: None
pytest: 5.0.1
IPython: 7.7.0
sphinx: None

[rhkleijn]

Dataset.astype() preserves attributes by default since xarray 0.16.1 with #4314. Running the scripts above with 0.16.2 I get the output below, which confirms attributes are now preserved. Can this issue be closed?

Output 1

<xarray.Dataset>
Dimensions:   (location: 3, time: 731)
Coordinates:
  * time      (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2001-12-31
  * location  (location) <U2 'IA' 'IN' 'IL'
Data variables:
    tmin      (time, location) float32 -8.03737 -1.7884412 ... -4.543927
    tmax      (time, location) float32 12.980549 3.3104093 ... 3.8052793
Attributes:
    CRS:      EPSG:4326

Output 2

Original xr.DataArray
attrs: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
dtype: int64

After casting to int64: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
After casting to int32: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
After casting to float32: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}

Changing the np.array
attrs: {'at_str': 'at1_value', 'at_float': 123.0, 'at_int': 123}
dtype: float32

[mathause]

Yes, this is fixed so I am closing. Thanks for noting @rhkleijn