Use apply_ufunc in xESMF regridding package

[mrocklin]

I would like to call attention to JiaweiZhuang/xESMF#3 (comment) . It seems like the xESMF package does regridding in a way that at least some XArray users find sensible. It should probably make use of, but does not currently use apply_ufunc, and is not particularly parallelizable (or at least that is my understanding). It could be that some modest development by someone more familiar with XArray could have a large impact by properly using apply_ufunc within that codebase.

I apologize for posting an issue about another package in this issue tracker. Feel free to close.

cc @JiaweiZhuang

[JiaweiZhuang]

Thanks for bringing this up... I've made more experiments and realized that Numba is actually faster than scipy.sparse, and also shows excellent parallel efficiency. See this notebook for all details. Thus I consider switch to Numba and add parallel support in the next version. It should fit better than xr.apply_ufunc in this case. Let's discuss in the original thread if you have further suggestions.

[mrocklin]

I would think that the use of numba for sparse matrix multiply would be orthogonal to the choice of parallelizing with apply_ufunc. I think that one can likely do both.

[JiaweiZhuang]

I agree that they can be both implemented, and dask is useful for out-of-core. If anyone would like to contribute, please see JiaweiZhuang/xESMF#3 (comment) for my preliminary experiments with xr.apply_ufunc.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically