We need a fast path for open_mfdataset

[rabernat]

It would be great to have a "fast path" option for open_mfdataset, in which all alignment / coordinate checking is bypassed. This would be used in cases where the user knows that many netCDF files all share the same coordinates (e.g. model output, satellite records from the same product, etc.). The coordinates would just be taken from the first file, and only the data variables would be read from all subsequent files. The only checking would be that the data variables have the correct shape.

Implementing this would require some refactoring. @jbusecke mentioned that he had developed a solution for this (related to #1704), so maybe he could be the one to add this feature to xarray.

This is also related to #1385.

[jhamman]

@rabernat - Depending on the structure of the dataset, another possibility that would speed up some open_mfdataset tasks substantially is to implement the step of opening each file and getting its metadata in in some parallel way (dask/joblib/etc.) and either returning the just dataset schema or a picklable version of the dataset itself. I think this will only be able to work with autoclose=True but it could be quite useful when working with many files.

[jbusecke]

I did not really find an elegant solution. What I did was just specify all dims and coords as drop_variables and then update those from a master file with

ds.update(ds_master)

Perhaps this could be generalized in a sense, by reading all coords and dims just from the first file.

[jbusecke]

Would these two options be necessarily mutually exclusive?

I think parallelizing the read in sounds amazing.

But isnt there some merit in skipping some of the checks all together, if the user is sure about the structure of the data contained in the many files?

I am often working with the aforementioned type of data (many files either contain a new timestep or a different variable, but most of the dimensions/coordinates are the same).

In some cases I am finding that reading the data "lazily" consumes a significant amount of the time in my workflow. I am unsure how hard this would be to achieve, and perhaps it is not worth it after all.

Just putting out a few ideas, while I wait for my xr.open_mfdataset to finish :-)

[jhamman]

@jbusecke - No. These options are not mutually exclusive. The parallel open is, in my opinion, the lowest hanging fruit so that's why I started there. There are other improvements that we can tackle incrementally.

[jbusecke]

Awesome, thanks for the clarification.
I just looked at #1981 and it seems indeed very elegant (in fact I just now used this approach to parallelize printing of movie frames!) Thanks for that!

[dcherian]

I am currently motivated to fix this.

1. Over in concat prealigned objects #1413 (comment) @rabernat mentioned

allowing the user to pass join='exact' via open_mfdataset. A related optimization would be to allow the user to pass coords='minimal' (or other concat coords options) via open_mfdataset.

2. @shoyer suggested calling decode_cf later here though perhaps this wont help too much: slow performance with open_mfdataset #1385 (comment)

Is this all that we can do on the xarray side?