Local test failure in test_backends

[max-sixty]

I'm happy to debug this further but before I do, is this an issue people have seen before? I'm running tests on master and hit an issue very early on.

FWIW I don't use netCDF, and don't think I've got that installed

Code Sample, a copy-pastable example if possible

========================================================================== FAILURES ==========================================================================
_________________________________________________________ ScipyInMemoryDataTest.test_bytesio_pickle __________________________________________________________

self = <xarray.tests.test_backends.ScipyInMemoryDataTest testMethod=test_bytesio_pickle>

    @pytest.mark.skipif(PY2, reason='cannot pickle BytesIO on Python 2')
    def test_bytesio_pickle(self):
        data = Dataset({'foo': ('x', [1, 2, 3])})
        fobj = BytesIO(data.to_netcdf())
        with open_dataset(fobj, autoclose=self.autoclose) as ds:
>           unpickled = pickle.loads(pickle.dumps(ds))
E           TypeError: can't pickle _thread.lock objects

xarray/tests/test_backends.py:1384: TypeError

Problem description

[this should explain why the current behavior is a problem and why the expected output is a better solution.]

Expected Output

Skip or pass backends tests

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: d00721a3560f57a1b9226c5dbf5bf3af0356619d python: 3.6.4.final.0 python-bits: 64 OS: Darwin OS-release: 17.4.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

xarray: 0.7.0-38-g1005a9e # not sure why this is tagged so early. I'm running on latest master
pandas: 0.22.0
numpy: 1.14.0
scipy: 1.0.0
netCDF4: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: None
distributed: None
matplotlib: 2.1.2
cartopy: None
seaborn: 0.8.1
setuptools: 38.5.1
pip: 9.0.1
conda: None
pytest: 3.4.0
IPython: 6.2.1
sphinx: None

[shoyer]

I think this test would pass if you had dask installed.

SciPy comes with read/write support for netCDF3, which is why this is being picked up.

[smithsp]

I think that I am encountering this myself. Here is my (relevant) environment:

$ conda list dask
# packages in environment at /fusion/projects/codes/conda/omfit:
#
# Name                    Version                   Build  Channel
dask                      0.17.0                     py_0    conda-forge
dask-core                 0.17.0                     py_0    conda-forge
$ conda list xarray
# packages in environment at /fusion/projects/codes/conda/omfit:
#
# Name                    Version                   Build  Channel
xarray                    0.10.3                   py27_0    conda-forge
$ conda list numpy
# packages in environment at /fusion/projects/codes/conda/omfit:
#
# Name                    Version                   Build  Channel
numpy                     1.12.1          py27_blas_openblas_201  [blas_openblas]  conda-forge
numpydoc                  0.8.0                    py27_0  

Here is the error:

  File "/home/smithsp/OMFIT-source/src/omfit.py", line 30, in <module>
    main()
  File "/home/smithsp/OMFIT-source/src/omfit.py", line 25, in main
    OMFIT['__userScript__'].runNoGUI(**_tmp)
  File "/home/smithsp/OMFIT-source/src/omfit_tree.py", line 1286, in runNoGUI
    return self.run(**kw)
  File "/home/smithsp/OMFIT-source/src/omfit_tree.py", line 1271, in run
    result = self.__run__(**kw)
  File "/home/smithsp/OMFIT-source/src/omfit_tree.py", line 1177, in __run__
    tmp=execGlobLoc(self, kw, _relLoc, {}, OMFITscriptsDict, prerun=_prerun, postrun=_postrun)
  File "/home/smithsp/OMFIT-source/src/omfit_tree.py", line 127, in f_locked
    return f(*args,**kw)
  File "/home/smithsp/OMFIT-source/src/omfit_tree.py", line 279, in execGlobLoc
    exec(compile(exeString, filename, "exec"),GlobLoc)
  File "/home/smithsp/OMFIT-source/regression/test_dataset.py", line 138, in <module>
    pickle.dump(OMFIT['TS_ds'],f,pickle.HIGHEST_PROTOCOL)
  File "/home/smithsp/OMFIT-source/src/classes/sortedDict.py", line 1295, in _OMFITdump
    return pypickle.dump(x,*args,**kw)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 1376, in dump
    Pickler(file, protocol).dump(obj)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/fusion/projects/codes/conda/omfit/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
TypeError: can't pickle thread.lock objects

Let me know if you want a smaller code sample to test.

[dcherian]

@max-sixty Is this still an issue?

[max-sixty]

I haven't tried in a while without dask - it's a very minor issue given that it only requires installing dask

[dcherian]

Ooops I didn't see that #1927 was open. Sorry for the noise.

[mathause]

Without really looking - I think this can be closed given that we test all but dask now: https://github.com/pydata/xarray/blob/master/ci/requirements/py38-all-but-dask.yml#L1