Subsetting not reducing memory consumption with mfdataset on Windows

[jorgsk]

On Windows, with the following code I run out of memory:

d = xr.open_mfdataset(filepath)
sed = d.sediment_mass_per_unit_area.isel(time=-1)
sed.load()
d.close()

but when using "open_dataset" I load the data instantly:

d = xr.open_dataset(filepath)
sed = d.sediment_mass_per_unit_area.isel(time=-1)
sed.load()
d.close()

On linux both methods use minimal memory. The target file is 1.5 GB, and the target variable is 1460x862x900 floats. Filepath is a single file, not multiple files. Xarray 0.9.6 used on both systems, installed with conda.

[jhamman]

your 2 code snippets are identical.

[jorgsk]

Sorry about that, I've updated the second snippet. Also added more information in case it helps.

[fmaussion]

Is the dask version also identical on both systems?

[jorgsk]

I found that dask was version 0.11 on Windows but 0.15 on Linux. Upgrading the Windows version to 0.15 fixed the issue, indicating that the problem was the version of dask.

[fmaussion]

Yes, I recall @rabernat worked extensively on this. Possibly related:

• round-trip performance with save_mfdataset / open_mfdataset #1340
• Excessive memory usage when printing multi-file Dataset #1481

@shoyer , @rabernat , maybe if you have some time it would be good to revisit these issues and see if they can be closed?