netCDF IO suport

[jhamman]

I'd like to propose read_netcdf (netCDF (Network Common Data Format)) a new Pandas I/O api feature with a similar top level interface as the other reader functions. This format is widely used in the scientific community. Furthermore, netCDF4 is implemented on top of the HDF5 library, making this a natural extension to functionality already in the api.

Most likely this would sit on top of the exiting Python/numpy interface to netCDF, and because each variables metadata is stored in the file header, no complicated parsing would necessary. Multidimensional variables could be handled in a similar manner as hdf.

This may have been brought up in the past but my search here and on the google didn't bring anything up.

[jtratner]

Interested in contributing to make this happen?

[jhamman]

Certainly willing to contribute on this. My proposal was meant to see if this would be feature that folks were interested in. I have some project specific implementations that could be generalized but the effort may be best served by a discussion on how to approach this. Presumably, following along the lines of the HDF5 application would be ideal.

[jtratner]

Is it tabular/columnar data?

If so, can you map netCDF data types directly to numpy dtypes? If so, seems like it's pretty clear what should happen.

[jreback]

@jhamman

netCDF is quite similar to the way HDF5 / PyTables work. I think you could make this very similar, as a separate module. I think you could maybe use HDFStore as a base class.

[jhamman]

@jtratner - N-dimensional homogeneous arrays, the netCDF4 package takes care of the loading into numpy arrays.

@jreback - I'll take a look at replicating the HDFStore features in terms of the netCDF4 package. I actually don't think this should be all that difficult.

I'll take a first stab at it over the next week or so and report back.

[jtratner]

looking forward to seeing what you come up with.

[jreback]

@jhamman gr8....be sure to add the dep (netCDF ?) to ci/requirements (you don't have to add to all, but make sure at least 2.7 and 3.3. (you can add where pytables is tested), you can add for different versions (if that matters).

provide plenty of test cases! start with a smaller feature set, you can build/add over time.

pls hook up to travis!

[ebrevdo]

@jhamman You may also be interested in the xray package that @jreback references. It's partly built on pandas and has support for n-dimensional, gridded, datatypes (the initial goal was to naturally represent netcdf3/4, which it does). You may find some of the code in there useful for your read/write functionality. Note that we use the netCDF4 library, which supports netcdf3 + 4, not any HDF5 library.

[jhamman]

Thanks @ebrevdo . This looks very promising.

In fact, it may make sense, as @jreback suggests, to use some of xray's back-ends in pandas.

I'll poke around xray and see how it all works.

[shoyer]

One tricky aspect here is dealing with large variables. It's possible (indeed, somewhat common) to have netCDF files with variables too big to fit into memory. Libraries like netCDF4-python (or xray, for that matter) let you use slicing syntax to only load part of an variable. I may be mistaken, but I don't think pandas has any support (yet) for objects that don't fit entirely into memory.

[jreback]

what is a 'large variable' ? you can easily load part of an object via query or slicing, see docs: http://pandas-docs.github.io/pandas-docs-travis/io.html#io-hdf5

[shoyer]

@jreback You're right, I stand corrected.