recent versions of sparse and dask seem to be incompatible with our tests

[github-actions]

Workflow Run URL

Python 3.9 Test Summary Info

FAILED xarray/tests/test_plot.py::TestFacetGrid::test_can_set_norm - ValueErr...
FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_line_plot
FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_pcolormesh_plot
FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_contour_plot

[keewis]

this seems to be caused by changes to either dask or sparse, not sure which.

[Illviljan]

This is showing up in the normal CI now, see
https://github.com/pydata/xarray/pull/5667/checks?check_run_id=3329247566

[dcherian]

This looks like dask.

__________________________________ test_chunk __________________________________

    @requires_dask
    def test_chunk():
        s = sparse.COO.from_numpy(np.array([0, 0, 1, 2]))
        a = DataArray(s)
        ac = a.chunk(2)
        assert ac.chunks == ((2, 2),)
        assert isinstance(ac.data._meta, sparse.COO)
>       assert_identical(ac, a)


self = <COO: shape=(2,), dtype=bool, nnz=0, fill_value=False>
other = <function _broadcast_trick_inner at 0x7f325aef91f0>

    def func(self, other):
        if _disables_array_ufunc(other):
            return NotImplemented
>       return ufunc(other, self)
E       TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'bitwise_and'>, '__call__', <function _broadcast_trick_inner at 0x7f325aef91f0>, <COO: shape=(2,), dtype=bool, nnz=0, fill_value=False>): 'curry', 'COO'

cc @jrbourbeau

[jrbourbeau]

Thanks for the ping @dcherian. Running against Dask's git history, dask/dask#7939 is the place where xarray/tests/test_sparse.py::test_chunk started failing. We probably dropped some relevant compatibility code when we stopped supporting NumPy 1.17 and pandas 0.25.

Since xarray has a clearly defined policy for minimum dependencies, would dropping NumPy 1.17 and pandas 0.25 be an acceptable solution here?

[keewis]

We already require pandas>=1.0 and I guess we can bump numpy (NEP29 allows it for releases after Jul 26), but I'm not sure why that would be an issue for the upstream-dev CI. For reference, the installed versions arenumpy=1.22.0.dev0+716.g20807c743 and pandas=1.4.0.dev0+436.gfcf3f24e02.

Interestingly, the installation of dask from github started failing two days ago (see the last succeeding and the first failing install – unless I'm misinterpreting the logs?). Any idea how to fix that? The most recent run correctly installed dask so I guess this was a temporary issue (or dask resolved this really quickly)

[jrbourbeau]

Ah, that's a good point @keewis. Stepping back a bit, my current situation locally is I'm able to reproduce the xarray/tests/test_sparse.py::test_chunk failure with:

• the latest commit to dask (dask/dask@76822db)
• the latest commit to xarray (e26aec9)
• numpy=1.21.1
• pandas=1.3.1

Rolling dask back to the 2021.7.2 release, but keeping everything else the same, test_chunk passes. After running git bisect between the latest dask commit and the 2021.7.2 release, dask/dask#7939 is the point where test_chunk started failing (cc @jakirkham for visibility). I'll try to dig in a bit more to see if I can find the root cause of the failure

EDIT: I should note that the test_chunk failure has a similar traceback to dask/dask#5259

[jakirkham]

Would double check that CI is pulling the latest sparse

xref: dask/dask#7939 (comment)

[jrbourbeau]

Good point. CI is currently pulling in the latest sparse=0.12.0 release (which is also what I'm running locally).

It also appears there were some ordering concerns related to dask/dask#5259 when test_chunk was added (xref #3202 (comment))

[keewis]

FWIW, the upstream-dev CI installs sparse=0.12.0+55.ge706c2e (i.e. the most recent state of the repository). However, since our other CI is also failing I guess this does not change much.

[jakirkham]

Yeah was just mentioning that since we had older version of sparse pulled while developing that PR at one point and it caused issues. Sounds like that is not the case here