Abnormal process termination when using bottleneck function on xarray data after transposing and having a dimension with length 1

[RubendeBruin]

When running the following example the python interpreter exists abnormally (exit code -1073741819)

The code that causes the exception is the maxnan() function in bottleneck.

This error only occurs when:

1. The data contains a dimension with length 1
2. The data is transposed
3. bottleneck is used

So I suspect that the transpose function changes the data somehow such that bottleneck can not handle it anymore.

• Calling bn.maxnan on a normal ndarray works fine
• Calling data.max() before transposing works fine
• Running the example without using bottleneck (either by not installing it or by disabling it in OPTIONS) works fine
• Running the example with len(time) > 1 works fine

What happened:

Process finished with exit code -1073741819 (0xC0000005)

What you expected to happen:

Script should run without crashing

Minimal Complete Verifiable Example:

from collections import OrderedDict
import numpy as np
import xarray as xr

xr.show_versions()

n_time = 1  # 1 : Fails, 2 : everything is fine

from xarray.core.options import OPTIONS
OPTIONS["use_bottleneck"] = True   # Set to False for work-around

# Build some dataset
dirs = np.linspace(0,360, num=121)
freqs = np.linspace(0,4,num=192)
spec_data = np.random.random(size=(n_time,192,121))

dims = ('time', 'freq', 'dir')
coords = OrderedDict()
coords['time'] = range(n_time)
coords['freq'] = freqs
coords['dir'] = dirs

xdata = xr.DataArray(
            data=spec_data, coords=coords, dims=dims, name='Spec name',
        ).to_dataset()

xdata = xdata.transpose(..., "freq")  # remove this line and the script will run

tm = xdata.max()
print('Done')

Anything else we need to know?:

Uhm... it was really hard to dig this deep? :-)

Environment:

Tested with python 3.8 and 3.9 on win64. Only required packages are xarray and bottleneck. For example:

name: ws
dependencies:
  - python=3.9
  - xarray
channels:
  - defaults
  - conda-forge

note: xarray requires pandas which installs bottleneck. To install without bottleneck pin pandas to 1.2.4

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: AMD64 Family 23 Model 96 Stepping 1, AuthenticAMD
byteorder: little
LC_ALL: None
LANG: None
LOCALE: ('English_United States', '1252')
libhdf5: None
libnetcdf: None

xarray: 0.20.1
pandas: 1.2.4
numpy: 1.21.4
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
setuptools: 59.1.1
pip: 21.3.1
conda: None
pytest: None
IPython: None
sphinx: None

[RubendeBruin]

tests on another machine (also win64) with the same result.

Running under WSL/Ubuntu results in a Segmentation Fault

[max-sixty]

Thanks for the detail @RubendeBruin . I'm guessing this a bottleneck issue — would you be up for trying to run bottleneck's nanmax function on the underlying numpy array (xdata.data), and seeing whether that has the same effect?

[RubendeBruin]

I can reproduce it with calling bn.nanmax directly, but I can not reproduce it without the xarray.transpose() function.

• If I call nanmax on the internal data of xarray then nanmax fails with a segfault:

np_data = xdata['Spec name'].data
bn.nanmax(np_data) # Segfault

• But if I create a copy of that data and then call nanmax then it works fine.

np_data = xdata['Spec name'].data
new_data = np_data.copy()
bn.nanmax(new_data) # works

I suspect that the xarray.transpose function does something with the data-structure (lazy reshuffling of dimensions?) that triggers the fault in bottleneck.

Full code:

from collections import OrderedDict
import numpy as np
import xarray as xr

xr.show_versions()

n_time = 1  # 1 : Fails, 2 : everything is fine

from xarray.core.options import OPTIONS
OPTIONS["use_bottleneck"] = True   # Set to False for work-around

# Build some dataset
dirs = np.linspace(0,360, num=121)
freqs = np.linspace(0,4,num=192)
spec_data = np.random.random(size=(n_time,192,121))

dims = ('time', 'freq', 'dir')
coords = OrderedDict()
coords['time'] = range(n_time)
coords['freq'] = freqs
coords['dir'] = dirs

xdata = xr.DataArray(
            data=spec_data, coords=coords, dims=dims, name='Spec name',
        ).to_dataset()

xdata = xdata.transpose(..., "freq")

import bottleneck as bn
np_data = xdata['Spec name'].data

new_data = np_data.copy()
bn.nanmax(new_data) # works

bn.nanmax(np_data) # Segfault
print('direct bn call done')

[keewis]

It's just a simple numpy.ndarray.transpose:

In [1]: import numpy as np
   ...: import bottleneck as bn
   ...: 
   ...: n_time = 1
   ...: spec_data = np.random.random(size=(n_time,192,121))
   ...: 
   ...: bn.nanmax(spec_data.transpose(0, 2, 1))
Segmentation fault

numpy.transpose returns a view, so I guess that's what causes bottleneck to segfault? Not sure, though, especially since changing the order does not trigger the segfault: spec_data.transpose(1, 0, 2)... maybe bottleneck doesn't like views with a first dimension of size 1?

[RubendeBruin]

Ok, then it is clearly a bottleneck/numpy issue. I will raise it there and close it here.

Thanks!