interpolate/sample array at point

[cossatot]

This looks like an excellent project, thanks for all the work.

I am wondering if there is functionality for interpolation or sampling of a Dataset at a point (or sequence of points). The SciPy way of doing this uses scipy.ndimage.map_coordinates and is completely backwards for (geo)spatial data, requiring the point's coordinates to be converted to 'array' coordinates before interpolation.

Does this already have an implementation, given how super common an operation it is? If not, would it be difficult to wrap pandas.Series.interpolate() up? I might write a function to do this, but I don't known the xray internals well enough to know where to stick it.

Thanks,
Richard

[shoyer]

Hi Richard,

Your question is actually very timely. We don't have any routines yet to do interpolation, although one of my colleagues (not sure if he's on GitHub) was looking into 1-dimensional interpolation last week.

A contribution to add interpolation to xray would certainly be very welcome!

I would not recommend wrapping the pandas routines -- they just use their own wrapper over scipy and only interpolate in 1D. They also have somewhat unusual API -- they support interpolation only to fill in missing values (marked with NaN).

Scipy has a wide variety of interpolation options, most of which I have not used. I'm guessing you're most immediately interested in doing something with map_coordinates? What were you thinking for the function signature?

As for where to put it, we have two options:

1. Add a method DataArray.interpolate.
2. If we don't think we can come up with a generic enough API to hold all interpolation strategies in a single method (quite likely!), add a new module xray.interpolation.

[cossatot]

Stephan,

I think that I could contribute some functions to do 'nearest' and linear interpolation in n-dimensions; these should be able to take advantage of the indexing afforded by xray and not have to deal with Scipy's functions, because they only require selecting values in the neighborhood of the points.

As far as I can tell, higher-order interpolation (spline, etc.) requires fitting functions to the entirety of the dataset, which is pretty slow/ram-intensive with large datasets, and many of the fuctions require the data to be on a regular grid (I am not sure what the xray / netcdf requirements are here). For this, it's probably better to use the scipy functions (at least I personally don't have the knowledge to write anything comparable without more study).

For the function signature, I was thinking about something simple, like:

xray.interpolate(point[s], array, field[s], order), or
DataArray.interpolate(point[s], order) where points are the points to interpolate field values from the array, and order is like in map_coordinates (0=nearest, 1=linear, 3=cubic, etc.).

This could return a Series or DataFrame.

But thinking about this a little more, there are kind of two sides to interpolation: What I think of as 'sampling', where we pull values at points from within a grid or structured array (like in map_coordinates), and then the creation of arrays from unstructured (or just smaller) point sets. Both are equally common in geoscience, and probably require pretty different treatment programmatically, so it might be wise to make room for both early.

[shoyer]

So I would definitely still wrap the scipy functions for any interpolation routines for xray. It's not worth rewriting any of them twice! Scipy is an optional dependency for xray, but it would be fine to require it for the interpolation routines.

Yes, there do seems to be a few different interpolation routines in scipy. Naming the function you're thinking of something like xray.interpolation.map_coordinates would leave plenty of room for alternative interpolation methods.

In xray, we have DataArray objects (corresponding to a particular variable in a netCDF file along with its coordinates) as well as Datasets (corresponding to entire netCDFs). I think it makes a bit more sense to interpolate individual variables, so I guess you would want this function to have a signature something like the following:

Parameters
------------
source : DataArray
    Gridded source data.
destination : ndarray or dict-like (e.g., xray.Dataset or pandas.DataFrame)
    Array of points of points to sample at (n_obs * n_dim)
    or mapping from dimension names to coordinate values.
order : int, optional
mode : {'constant', 'nearest'}, optional
cval : scalar, optional
   Maybe defaulting to NaN instead of 0?

Returns
---------
interpolated : DataArray
    Data array with interpolated values. Coordinates and
    dimensions are copied from points.

You don't need to necessarily handle all the edge cases of possible input to destination, but it would be nice to handle destinations with labeled dimensions (e.g., in the form of a dictionary or xray.Dataset), and return labeled data in the form of an xray DataArray, with the same coordinate values (if they were supplied).

The DataArray constructor isn't in the currently released version of xray but it will be a major feature of the next release (hopefully to be released in a week or two) and it might be worth using here. It should be clearly documented in the docstring of DataArray.__init__ but let me know if you have any questions.

[shoyer]

Actually, I'm thinking now that we could probably (theoretically) fit all interpolation strategies into a single interpolate method, where the specific type of interpolation (e.g., grid -> grid, grid -> unstructured, unstructured -> grid) is selected based on the shape of source and destination.

That said... perhaps its better to stick with a simpler function for now and we can figure out the bigger picture later. Looking at all the scipy functions, there are a lot of hypothetical options and I'm not sure which we'd want to keep in the final method. Also, kitchen sink type functions (e.g., pandas.DataFrame.plot) are not necessarily well-advised.

[nfaggian]

For what its worth, I wrote this today. Its a long way from being useful but I find it's working well enough to fill gaps in data after a reindex()

from scipy import interpolate, ndimage

def linterp(data, index, interp_index, order=1):
    """
    Parameters
    ----------
        data: nd-array (cube).
        index: index (floats) associated with the cube.
        interp_index: float interpolation poing. 
    Returns
    -------
        interpolated: nd-array
            An interpolated field.
    """

    # Form a cube of the values, which we will imagine is our function f()
    cube = np.array(data, dtype=np.float)

    # Form a relationship, and this can be non-linear, between slice indexes and
    # the "index".
    m = interpolate.interp1d(index, range(len(index)))

    # Form a set of coordinates to sample over - x
    y, x = np.mgrid[0:data[0].shape[0], 0:data[0].shape[1]]
    z = np.ones_like(y) * m(interp_index)

    # Perform the sampling f(x), map coordinates is performing a linear
    # interpolation of the coordinates in the cube.
    return ndimage.map_coordinates(
        cube,
        [z, y, x],
        order=order,
        cval=np.nan)

[den-run-ai]

+1

[saulomeirelles]

Hi All,

This is indeed an excellent project with great potential!

I am wondering if there is any progress on the interpolation issue. I am working with an irregular time series which I would pretty much like to upsample using xray.

Thanks for all the effort!

Saulo

[shoyer]

@saulomeirelles I don't have any progress to share on this issue. If this issue is important to you, contributions would be gratefully accepted!