Merge branch 'feature/expand-dims-broadcast' of https://github.com/pletchm/xarray into feature/expand-dims-broadcast

Author: pletchm

.travis.yml

@@ -19,7 +19,6 @@ matrix:
     - EXTRA_FLAGS="--run-flaky --run-network-tests"
   - env: CONDA_ENV=py36-dask-dev
   - env: CONDA_ENV=py36-pandas-dev
-  - env: CONDA_ENV=py36-bottleneck-dev
   - env: CONDA_ENV=py36-rasterio
   - env: CONDA_ENV=py36-zarr-dev
   - env: CONDA_ENV=docs
@@ -31,7 +30,6 @@ matrix:
     - CONDA_ENV=py36
     - EXTRA_FLAGS="--run-flaky --run-network-tests"
   - env: CONDA_ENV=py36-pandas-dev
-  - env: CONDA_ENV=py36-bottleneck-dev
   - env: CONDA_ENV=py36-zarr-dev
 
 before_install:

ci/requirements-py36-bottleneck-dev.yml

@@ -20,16 +20,16 @@ dependencies:
   - scipy
   - seaborn
   - toolz
-  # - rasterio  # xref #2683
+  - rasterio
   - bottleneck
   - zarr
   - pseudonetcdf>=3.0.1
   - eccodes
   - cdms2
-# - pynio  # xref #2683
-# - iris>=1.10  # xref #2683
+  - pynio
+  - iris>=1.10
   - pydap
   - lxml
   - pip:
     - cfgrib>=0.9.2
-    - mypy==0.650
+    - mypy==0.660

ci/requirements-py36.yml

@@ -4,6 +4,7 @@ channels:
 dependencies:
   - python=3.7
   - cftime
+  - nc-time-axis
   - dask
   - distributed
   - h5py

ci/requirements-py37-windows.yml

@@ -4,6 +4,7 @@ channels:
 dependencies:
   - python=3.7
   - cftime
+  - nc-time-axis
   - dask
   - distributed
   - h5py

ci/requirements-py37.yml

@@ -45,8 +45,6 @@ For accelerating xarray
 - `bottleneck <https://github.com/kwgoodman/bottleneck>`__: speeds up
   NaN-skipping and rolling window aggregations by a large factor
   (1.1 or later)
-- `cyordereddict <https://github.com/shoyer/cyordereddict>`__: speeds up most
-  internal operations with xarray data structures (for python versions < 3.5)
 
 For parallel computing
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -63,6 +61,8 @@ For plotting
   :ref:`plot-maps`
 - `seaborn <https://stanford.edu/~mwaskom/software/seaborn/>`__: for better
   color palettes
+- `nc-time-axis <https://github.com/SciTools/nc-time-axis>`__: for plotting
+  cftime.datetime objects (1.2.0 or later)
 
 
 Instructions
@@ -111,4 +111,4 @@ To run these benchmark tests in a local machine, first install
 - `airspeed-velocity <https://asv.readthedocs.io/en/latest/>`__: a tool for benchmarking Python packages over their lifetime.
 
 and run
-``asv run  # this will install some conda environments in ./.asv/envs``
+``asv run  # this will install some conda environments in ./.asv/envs``

doc/installing.rst

@@ -23,6 +23,11 @@ Matplotlib syntax and function names were copied as much as possible, which
 makes for an easy transition between the two.
 Matplotlib must be installed before xarray can plot.
 
+To use xarray's plotting capabilities with time coordinates containing
+``cftime.datetime`` objects
+`nc-time-axis <https://github.com/SciTools/nc-time-axis>`_ v1.2.0 or later
+needs to be installed.
+
 For more extensive plotting applications consider the following projects:
 
 - `Seaborn <http://seaborn.pydata.org/>`_: "provides
@@ -226,7 +231,7 @@ Step plots
 ~~~~~~~~~~
 
 As an alternative, also a step plot similar to matplotlib's ``plt.step`` can be
-made using 1D data. 
+made using 1D data.
 
 .. ipython:: python
 
@@ -248,7 +253,7 @@ when plotting data grouped with :py:func:`xarray.Dataset.groupby_bins`.
     plt.ylim(-20,30)
     @savefig plotting_example_step_groupby.png width=4in
     plt.title('Zonal mean temperature')
-    
+
 In this case, the actual boundaries of the bins are used and the ``where`` argument
 is ignored.
 

doc/plotting.rst

@@ -74,7 +74,7 @@ will be used for indexing.  :py:class:`~xarray.CFTimeIndex` enables a subset of
 the indexing functionality of a :py:class:`pandas.DatetimeIndex` and is only
 fully compatible with the standalone version of ``cftime`` (not the version
 packaged with earlier versions ``netCDF4``).  See :ref:`CFTimeIndex` for more
-information. 
+information.
 
 Datetime indexing
 -----------------
@@ -215,7 +215,7 @@ For more examples of using grouped operations on a time dimension, see
 
 
 .. _CFTimeIndex:
-     
+
 Non-standard calendars and dates outside the Timestamp-valid range
 ------------------------------------------------------------------
 
@@ -224,14 +224,14 @@ Through the standalone ``cftime`` library and a custom subclass of
 functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for
 dates from non-standard calendars commonly used in climate science or dates
 using a standard calendar, but outside the `Timestamp-valid range`_
-(approximately between years 1678 and 2262).  
+(approximately between years 1678 and 2262).
 
 .. note::
 
    As of xarray version 0.11, by default, :py:class:`cftime.datetime` objects
    will be used to represent times (either in indexes, as a
-   :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if 
-   any of the following are true: 
+   :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if
+   any of the following are true:
 
    - The dates are from a non-standard calendar
    - Any dates are outside the Timestamp-valid range.
@@ -252,7 +252,7 @@ coordinate with dates from a no-leap calendar and a
    dates = [DatetimeNoLeap(year, month, 1) for year, month in
             product(range(1, 3), range(1, 13))]
    da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo')
-                         
+
 xarray also includes a :py:func:`~xarray.cftime_range` function, which enables
 creating a :py:class:`~xarray.CFTimeIndex` with regularly-spaced dates.  For
 instance, we can create the same dates and DataArray we created above using:
@@ -261,20 +261,20 @@ instance, we can create the same dates and DataArray we created above using:
 
    dates = xr.cftime_range(start='0001', periods=24, freq='MS', calendar='noleap')
    da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo')
-   
+
 For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
 - `Partial datetime string indexing`_ using strictly `ISO 8601-format`_ partial
   datetime strings:
-  
+
 .. ipython:: python
 
    da.sel(time='0001')
    da.sel(time=slice('0001-05', '0002-02'))
 
 - Access of basic datetime components via the ``dt`` accessor (in this case
   just "year", "month", "day", "hour", "minute", "second", "microsecond",
-  "season", "dayofyear", and "dayofweek"): 
+  "season", "dayofyear", and "dayofweek"):
 
 .. ipython:: python
 
@@ -309,31 +309,27 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
    da.differentiate('time')
 
-- And serialization:
+- Serialization:
 
 .. ipython:: python
 
    da.to_netcdf('example-no-leap.nc')
    xr.open_dataset('example-no-leap.nc')
 
-.. note::
-   
-   While much of the time series functionality that is possible for standard
-   dates has been implemented for dates from non-standard calendars, there are
-   still some remaining important features that have yet to be implemented,
-   for example:
+- And resampling along the time dimension for data indexed by a :py:class:`~xarray.CFTimeIndex`:
 
-   - Resampling along the time dimension for data indexed by a
-     :py:class:`~xarray.CFTimeIndex` (:issue:`2191`, :issue:`2458`)
-   - Built-in plotting of data with :py:class:`cftime.datetime` coordinate axes
-     (:issue:`2164`).   
+.. ipython:: python
+
+    da.resample(time='81T', closed='right', label='right', base=3).mean()
+
+.. note::
+    
 
    For some use-cases it may still be useful to convert from
    a :py:class:`~xarray.CFTimeIndex` to a :py:class:`pandas.DatetimeIndex`,
-   despite the difference in calendar types (e.g. to allow the use of some
-   forms of resample with non-standard calendars).  The recommended way of
-   doing this is to use the built-in
-   :py:meth:`~xarray.CFTimeIndex.to_datetimeindex` method:
+   despite the difference in calendar types. The recommended way of doing this
+   is to use the built-in :py:meth:`~xarray.CFTimeIndex.to_datetimeindex`
+   method:
 
    .. ipython:: python
       :okwarning:
@@ -343,14 +339,13 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
        da
        datetimeindex = da.indexes['time'].to_datetimeindex()
        da['time'] = datetimeindex
-       da.resample(time='Y').mean('time')
-   
+
    However in this case one should use caution to only perform operations which
    do not depend on differences between dates (e.g. differentiation,
    interpolation, or upsampling with resample), as these could introduce subtle
    and silent errors due to the difference in calendar types between the dates
-   encoded in your data and the dates stored in memory.  
-  
+   encoded in your data and the dates stored in memory.
+
 .. _Timestamp-valid range: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#timestamp-limitations
 .. _ISO 8601-format: https://en.wikipedia.org/wiki/ISO_8601
 .. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#partial-string-indexing

doc/time-series.rst

@@ -24,14 +24,20 @@ Breaking changes
 - Remove support for Python 2. This is the first version of xarray that is
   Python 3 only. (:issue:`1876`).
   By `Joe Hamman <https://github.com/jhamman>`_.
-- The `compat` argument to `Dataset` and the `encoding` argument to 
+- The `compat` argument to `Dataset` and the `encoding` argument to
   `DataArray` are deprecated and will be removed in a future release.
   (:issue:`1188`)
   By `Maximilian Roos <https://github.com/max-sixty>`_.
+- `cyordereddict` is no longer used as an optional dependency (:issue:`2744`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
 
 Enhancements
 ~~~~~~~~~~~~
 
+- Internal plotting now supports ``cftime.datetime`` objects as time series.
+  (:issue:`2164`)
+  By `Julius Busecke <https://github.com/jbusecke>`_ and
+  `Spencer Clark <https://github.com/spencerkclark>`_.
 - Add ``data=False`` option to ``to_dict()`` methods. (:issue:`2656`)
   By `Ryan Abernathey <https://github.com/rabernat>`_
 - :py:meth:`~xarray.DataArray.coarsen` and
@@ -47,20 +53,25 @@ Enhancements
   report showing what exactly differs between the two objects (dimensions /
   coordinates / variables / attributes)  (:issue:`1507`).
   By `Benoit Bovy <https://github.com/benbovy>`_.
+- Resampling of standard and non-standard calendars indexed by
+  :py:class:`~xarray.CFTimeIndex` is now possible. (:issue:`2191`).
+  By `Jwen Fai Low <https://github.com/jwenfai>`_ and
+  `Spencer Clark <https://github.com/spencerkclark>`_.
 - Add ``tolerance`` option to ``resample()`` methods ``bfill``, ``pad``,
   ``nearest``. (:issue:`2695`)
   By `Hauke Schulz <https://github.com/observingClouds>`_.
-
 - :py:meth:`~xarray.DataArray.integrate` and
   :py:meth:`~xarray.Dataset.integrate` are newly added.
   See :ref:`_compute.using_coordinates` for the detail.
   (:issue:`1332`)
   By `Keisuke Fujii <https://github.com/fujiisoup>`_.
+- :py:meth:`pandas.Series.dropna` is now supported for a
+  :py:class:`pandas.Series` indexed by a :py:class:`~xarray.CFTimeIndex`
+  (:issue:`2688`). By `Spencer Clark <https://github.com/spencerkclark>`_.
 - Allow ``expand_dims`` method to support inserting/broadcasting dimensions
   with size > 1. (:issue:`2710`)
   By `Martin Pletcher <https://github.com/pletchm>`_.
 
-
 Bug fixes
 ~~~~~~~~~
 
@@ -77,6 +88,12 @@ Bug fixes
   :py:class:`CFTimeIndex` now results in a :py:class:`pandas.TimedeltaIndex`
   instead of raising a ``TypeError`` (:issue:`2671`).  By `Spencer Clark
   <https://github.com/spencerkclark>`_.
+- backend_kwargs are no longer ignored when using open_dataset with pynio engine
+  (:issue:'2380')
+  By 'Jonathan Joyce <https://github.com/jonmjoyce>'_.
+- Fix ``open_rasterio`` creating a WKT CRS instead of PROJ.4 with
+  ``rasterio`` 1.0.14+ (:issue:`2715`).
+  By `David Hoese <https://github.com/djhoese`_.
 
 .. _whats-new.0.11.3:
 

doc/whats-new.rst

@@ -45,13 +45,13 @@ class NioDataStore(AbstractDataStore):
     """Store for accessing datasets via PyNIO
     """
 
-    def __init__(self, filename, mode='r', lock=None):
+    def __init__(self, filename, mode='r', lock=None, **kwargs):
         import Nio
         if lock is None:
             lock = PYNIO_LOCK
         self.lock = ensure_lock(lock)
         self._manager = CachingFileManager(
-            Nio.open_file, filename, lock=lock, mode=mode)
+            Nio.open_file, filename, lock=lock, mode=mode, kwargs=kwargs)
         # xarray provides its own support for FillValue,
         # so turn off PyNIO's support for the same.
         self.ds.set_option('MaskedArrayMode', 'MaskedNever')

xarray/backends/pynio_.py

@@ -282,7 +282,10 @@ def open_rasterio(filename, parse_coordinates=None, chunks=None, cache=None,
         # CRS is a dict-like object specific to rasterio
         # If CRS is not None, we convert it back to a PROJ4 string using
         # rasterio itself
-        attrs['crs'] = riods.crs.to_string()
+        try:
+            attrs['crs'] = riods.crs.to_proj4()
+        except AttributeError:
+            attrs['crs'] = riods.crs.to_string()
     if hasattr(riods, 'res'):
         # (width, height) tuple of pixels in units of CRS
         attrs['res'] = riods.res