Perf: use Scipy engine by default for netcdf3?

[mangecoeur]

Not really a bug, but I'm finding that the scipy backend is considerably faster than the netCDF backend for netCDF 3 files (using dataset: http://rda.ucar.edu/datasets/ds093.1/). Using Anaconda python with MKL. Not sure if this is always faster, but if it is perhaps xarray should default to scipy backend for netCDF 3 files?

[shoyer]

Yes, I have found this to be true in my experience, as well. The scipy backend also works more smoothly when using multiple threads with dask, because it releases the GIL.

I would be happy to accept a patch that adds this as the default behavior. We would need to detect netCDF3 vs netCDF4 files automatically when doing file reading by looking at the first few bytes from the file. See these docs for file format specs, which include the necessary details:
https://www.hdfgroup.org/HDF5/doc/H5.format.html#Superblock
http://www.unidata.ucar.edu/software/netcdf/docs/file_format_specifications.html

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity
If this issue remains relevant, please comment here; otherwise it will be marked as closed automatically