unstacking an integer array yields a RuntimeWarning after upgrade to numpy 1.24.1

[itcarroll]

What happened?

After upgrading numpy from 1.23.5 to 1.24.1, calling the unstack method on an xarray.DataArray with integer data produces the warning <__array_function__ internals>:200: RuntimeWarning: invalid value encountered in cast. I think this relates to "ongoing work to improve the handling and promotion of dtypes" (Numpy 1.24.0 Release Notes), and is catching the fact that the method attempts to provide nan as a fill value on an integer array.

What did you expect to happen?

In the case below, where there is no need for a fill value, I do not expect to get a warning.

Minimal Complete Verifiable Example

import xarray as xr
import numpy as np
# np.seterr(all='raise') # uncomment to convert warning to error

da = xr.DataArray(
    data=np.array([[0]], dtype=int),
    coords={'x': [0], 'y': [1]},
    )
da = da.stack({'z': ['x', 'y']})
da.unstack()

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

<__array_function__ internals>:200: RuntimeWarning: invalid value encountered in cast
<xarray.DataArray (x: 1, y: 1)>
array([[0]])
Coordinates:
  * x        (x) int64 0
  * y        (y) int64 1

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.10.9 (main, Dec 15 2022, 18:18:30) [Clang 14.0.0 (clang-1400.0.29.202)] python-bits: 64 OS: Darwin OS-release: 21.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: None LOCALE: (None, 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.0

xarray: 2022.12.0
pandas: 1.5.2
numpy: 1.24.1
scipy: 1.10.0
netCDF4: 1.6.2
pydap: None
h5netcdf: 1.1.0
h5py: 3.7.0
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.12.1
distributed: None
matplotlib: 3.6.2
cartopy: 0.21.1
seaborn: None
numbagg: None
fsspec: 2022.11.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.6.3
pip: 22.1.2
conda: None
pytest: None
mypy: None
IPython: 8.8.0
sphinx: None

[keewis]

I just encountered this one as well. It seems the issue is here:

xarray/xarray/core/variable.py

Lines 1815 to 1820 in 43ba095

	data = np.full_like(
	self.data,
	fill_value=fill_value,
	shape=new_shape,
	dtype=dtype,
	)

where fill_value = np.nan and dtype = "int64".

As mentioned in numpy/numpy#8017 (comment), numpy now warns instead of silently casting nan to int.

Can we do anything about that here (besides silencing the warning, but I'm not sure if that actually makes sense), or do we need to lobby for nullable dtypes in numpy or a numpy-adjacent library?

Edit: as it seems there are at least 4 deferred NEPs (NEPs 12, 24, 25, and 26) on the topic of missing values this might be more tricky than I expected. So I guess that means that we might have to try finding a workaround.

[itcarroll]

Would you consider xarray itself to be "numpy-adjacent"? If XArray fully adopted (i don't know how!) the NetCDF4 _FillValue attribute, this and some other challenges, could be solved rigorously.

[blackbox-tech]

This new warning has appeared when I call DataSet.unstack(), with xarray 2023.5.0 and numpy 1.24

I suspect this warning is related to the stricter type checking in numpy 1.24.
https://numpy.org/doc/stable/release/1.24.0-notes.html#numpy-now-gives-floating-point-errors-in-casts

[max-sixty]

I no longer get the warning on numpy 1.26.1 — please reopen if others do

[tguruswamy]

I see this on numpy 1.26.2 using the MWE (and my own code)

  numpy                          1.26.2        py311h24aa872_0                
  numpy-base                     1.26.2        py311hbfb1bba_0
  xarray                         2023.6.0      py311h06a4308_0 

IPython 8.15.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import xarray as xr
   ...: import numpy as np
   ...: # np.seterr(all='raise') # uncomment to convert warning to error
   ...: 
   ...: da = xr.DataArray(
   ...:     data=np.array([[0]], dtype=int),
   ...:     coords={'x': [0], 'y': [1]},
   ...:     )
   ...: da = da.stack({'z': ['x', 'y']})
   ...: da.unstack()
.../lib/python3.11/site-packages/numpy/core/numeric.py:407: RuntimeWarning: invalid value encountered in cast
  multiarray.copyto(res, fill_value, casting='unsafe')
Out[1]: 
<xarray.DataArray (x: 1, y: 1)>
array([[0]])
Coordinates:
  * x        (x) int64 0
  * y        (y) int64 1

[max-sixty]

@tguruswamy I don't get that on the latest xarray — do you?

[tguruswamy]

You're right -- 2023.12.0 is ok. Thanks for the pointer! A quick bisect suggests this was solved in v2023.8.0.

[keewis]

this should have been closed by #8061

[itcarroll]

Agree the issue was resolved with v2023.8.0.