holoviews / bokeh doesn't like cftime coords

[rabernat]

Code Sample, a copy-pastable example if possible

Consider a simple working example of converting an xarray dataset to holoviews for plotting:

ref_date = '1981-01-01'
ds = xr.DataArray([1, 2, 3], dims=['time'],
                  coords={'time': ('time', [1, 2, 3],
                                   {'units': 'days since %s' % ref_date})}
                  ).to_dataset(name='foo')
with xr.set_options(enable_cftimeindex=True):
    ds = xr.decode_cf(ds)
print(ds)
hv_ds = hv.Dataset(ds)
hv_ds.to(hv.Curve)

This gives

<xarray.Dataset>
Dimensions:  (time: 3)
Coordinates:
  * time     (time) datetime64[ns] 1981-01-02 1981-01-03 1981-01-04
Data variables:
    foo      (time) int64 ...

and

Problem description

Now change ref_date = '0181-01-01' (or anything outside of the valid range for regular pandas datetime index). We get a beautiful new cftimeindex

<xarray.Dataset>
Dimensions:  (time: 3)
Coordinates:
  * time     (time) object 0181-01-02 00:00:00 0181-01-03 00:00:00 ...
Data variables:
    foo      (time) int64 ...

but holoviews / bokeh doesn't like it

/opt/conda/lib/python3.6/site-packages/xarray/coding/times.py:132: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy cftime.datetime objects instead, reason: dates out of range
  enable_cftimeindex)
/opt/conda/lib/python3.6/site-packages/xarray/coding/variables.py:66: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy cftime.datetime objects instead, reason: dates out of range
  return self.func(self.array[key])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj, include, exclude)
    968 
    969             if method is not None:
--> 970                 return method(include=include, exclude=exclude)
    971             return None
    972         else:

/opt/conda/lib/python3.6/site-packages/holoviews/core/dimension.py in _repr_mimebundle_(self, include, exclude)
   1229         combined and returned.
   1230         """
-> 1231         return Store.render(self)
   1232 
   1233 

/opt/conda/lib/python3.6/site-packages/holoviews/core/options.py in render(cls, obj)
   1287         data, metadata = {}, {}
   1288         for hook in hooks:
-> 1289             ret = hook(obj)
   1290             if ret is None:
   1291                 continue

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in pprint_display(obj)
    278     if not ip.display_formatter.formatters['text/plain'].pprint:
    279         return None
--> 280     return display(obj, raw_output=True)
    281 
    282 

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in display(obj, raw_output, **kwargs)
    248     elif isinstance(obj, (CompositeOverlay, ViewableElement)):
    249         with option_state(obj):
--> 250             output = element_display(obj)
    251     elif isinstance(obj, (Layout, NdLayout, AdjointLayout)):
    252         with option_state(obj):

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in wrapped(element)
    140         try:
    141             max_frames = OutputSettings.options['max_frames']
--> 142             mimebundle = fn(element, max_frames=max_frames)
    143             if mimebundle is None:
    144                 return {}, {}

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in element_display(element, max_frames)
    186         return None
    187 
--> 188     return render(element)
    189 
    190 

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in render(obj, **kwargs)
     63         renderer = renderer.instance(fig='png')
     64 
---> 65     return renderer.components(obj, **kwargs)
     66 
     67 

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/renderer.py in components(self, obj, fmt, comm, **kwargs)
    257         # Bokeh has to handle comms directly in <0.12.15
    258         comm = False if bokeh_version < '0.12.15' else comm
--> 259         return super(BokehRenderer, self).components(obj,fmt, comm, **kwargs)
    260 
    261 

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in components(self, obj, fmt, comm, **kwargs)
    319             plot = obj
    320         else:
--> 321             plot, fmt = self._validate(obj, fmt)
    322 
    323         widget_id = None

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in _validate(self, obj, fmt, **kwargs)
    218         if isinstance(obj, tuple(self.widgets.values())):
    219             return obj, 'html'
--> 220         plot = self.get_plot(obj, renderer=self, **kwargs)
    221 
    222         fig_formats = self.mode_formats['fig'][self.mode]

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/renderer.py in get_plot(self_or_cls, obj, doc, renderer)
    150             doc = Document() if self_or_cls.notebook_context else curdoc()
    151         doc.theme = self_or_cls.theme
--> 152         plot = super(BokehRenderer, self_or_cls).get_plot(obj, renderer)
    153         plot.document = doc
    154         return plot

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in get_plot(self_or_cls, obj, renderer)
    205             init_key = tuple(v if d is None else d for v, d in
    206                              zip(plot.keys[0], defaults))
--> 207             plot.update(init_key)
    208         else:
    209             plot = obj

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/plot.py in update(self, key)
    511     def update(self, key):
    512         if len(self) == 1 and ((key == 0) or (key == self.keys[0])) and not self.drawn:
--> 513             return self.initialize_plot()
    514         item = self.__getitem__(key)
    515         self.traverse(lambda x: setattr(x, '_updated', True))

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in initialize_plot(self, ranges, plot, plots, source)
    729         if not self.overlaid:
    730             self._update_plot(key, plot, style_element)
--> 731             self._update_ranges(style_element, ranges)
    732 
    733         for cb in self.callbacks:

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in _update_ranges(self, element, ranges)
    498         if not self.drawn or xupdate:
    499             self._update_range(x_range, l, r, xfactors, self.invert_xaxis,
--> 500                                self._shared['x'], self.logx, streaming)
    501         if not self.drawn or yupdate:
    502             self._update_range(y_range, b, t, yfactors, self.invert_yaxis,

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in _update_range(self, axis_range, low, high, factors, invert, shared, log, streaming)
    525             updates = {}
    526             if low is not None and (isinstance(low, util.datetime_types)
--> 527                                     or np.isfinite(low)):
    528                 updates['start'] = (axis_range.start, low)
    529             if high is not None and (isinstance(high, util.datetime_types)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Similar but slightly different errors arise for different holoviews types (e.g. hv.Image) and contexts (using time as a holoviews kdim).

Expected Output

This should work.

I'm not sure if this is really an xarray problem. Maybe it needs a fix in holoviews (or bokeh). But I'm raising it here first since clearly we have introduced this new wrinkle in the stack. Cc'ing @philippjfr since he is the expert on all things holoviews.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 4.4.111+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

xarray: 0.10.4
pandas: 0.23.0
numpy: 1.14.3
scipy: 1.1.0
netCDF4: 1.4.0
h5netcdf: None
h5py: None
Nio: None
zarr: 2.2.0
bottleneck: None
cyordereddict: None
dask: 0.17.5
distributed: 1.21.8
matplotlib: 2.2.2
cartopy: None
seaborn: None
setuptools: 39.0.1
pip: 10.0.1
conda: 4.3.34
pytest: 3.5.1
IPython: 6.3.1
sphinx: None

[philippjfr]

Thanks for the detailed example, I've been able to reproduce the issue. Part of it is that we need to add the new type(s) to holoviews.util.datetime_types. Secondly it seems like bokeh's date conversion code will also have to be made aware of this new type somehow (I haven't investigated that yet).

[spencerkclark]

I agree it would be very nice to enable plotting data with cftime.datetime coordinates with holoviews. Eventually it would also be great if we could enable it for xarray's built-in plotting too. I'm happy to help out where I can.

[jbusecke]

I encountered this problem right now with the xarray built-in plotting. Does anybody know a workaround for the xarray plotting by any chance?

[spencerkclark]

@jbusecke I recently stumbled upon https://github.com/SciTools/nc-time-axis. You can install it through conda-forge:

$ conda install -c conda-forge nc-time-axis

However, for now you won't be able to use xarray's built in plotting, since xarray will raise an error if you try to plot anything with coordinates that aren't numeric or of type datetime.datetime or np.datetime64 (but nc-time-axis at least gives you matplotlib support). You'll also need to convert your dates to nc_time_axis.CalendarDateTime objects; you can do that with a simple list comprehension (see the example in their README).

[aidanheerdegen]

Darn. Just when I thought the time stuff was sorted. This is (yet another) deal breaker as far as recommending mass adoption goes.

Is there an estimate when, or if, cftime indexes will be supported by xarray's .plot() method?

[rabernat]

Darn. Just when I thought the time stuff was sorted. This is (yet another) deal breaker as far as recommending mass adoption goes.

@aidanheerdegen -- support for unconventional time coordinates (via cftime) is not a trivial problem--there are many special cases and complex logic to deal with. Rather than being discouraged by this bug, I encourage you to take a longer view to see how much this support has improved over the past year. One year ago, xarray could not have decoded this time coordinate at all!

I'm curious what you are referring to with your comment about "mass adoption." To whom are you making recommendations about adoption of xarray? And what do you consider the numerous other dealbreakers to be? I hope you see that these blanket comments could be read as quite discouraging and negative to the volunteer developers working hard on xarray.

Is there an estimate when, or if, cftime indexes will be supported by xarray's .plot() method?

The work on cftimeindex is being done entirely on a volunteer basis by graduate students like @spencerkclark (👏 👏 👏). In order to make xarray development progress faster, we need more contributors to the project. In my view, the ideal place to recruit such contributors is from within the paid computational support staff from major climate modeling centers. These people are ideally poised to understand what features are most important to users, and they are actually paid to help develop tools (like xarray) to help users be more productive. We recently created a contributor guide to make it easier for new contributors to come on board. If you can think of anyone who might be interested in contributing to xarray, please let us know, either here or via a private channel.

[aidanheerdegen]

Hi @rabernat,

I apologise if what I said is discouraging. I didn't intend it that way. It was the result of exasperation, as I think array is a fantastic tool, and I thought with the development of cftime support all barriers to widespread adoption had pretty much been overcome. When I said "another" it was in reference to the previous barrier of not supporting long time, or old time, indices which had since been overcome

As far as recommending, it is to the researchers at the centre of excellence where I am one of the people who is paid to support climate models and the support infrastructure to run and analyse their outputs. I guess I've outed myself as one of those paid computational support staff you referred to.

My initial comment above was a clumsy attempt to highlight what I thought was an important feature to support to further increase xarray adoption. From my perspective as someone who has to support users I'm often having to decide what I think the majority of users will be able to use efficiently, taking into account very wide levels of expertise and motivation. Before the cftime upgrades I did not wholeheartedly evangelise for xarray adoption because I knew there were many cases where it was not simple and easy to use. For every edge and corner case I have to support users when they encounter them. In some ways, having a tool that can do such amazing things as xarray, but which don't work in some circumstances for some datasets is very frustrating for users. It can take a lot of work to find out what doesn't work.

Having said which, we're currently doing half way through a 2 hour training session for xarray for researchers in the CoE who are interested, but not being able to easily plot cftime datasets will harm adoption, and all those who are volunteering their time developing xarray want it to be adopted as widely as possible right?

Thanks for the pointer to the contributor guide, I did read it, and I will try and find some time to make a positive contribution to xarray. I had started down that path already (#2244)

[rabernat]

Thanks for your follow up comment. I understand much better where you are coming from now! We do sincerely appreciate your feedback and contributions. Yes: we definitely want xarray to be adopted widely!

There is no doubt that these weird calendars are a continuous source of frustration. The challenge is compounded by the fast that, basically, only the climate community needs them. All of the fancy time indexing stuff in pandas is most likely there because of its value to the finance community.

Maybe a good path forward would be for @spencerkclark to outline what steps might be needed to get xarray's built in plotting to work with cftimeindex. That way, anyone who urgently needs this feature has some sort of roadmap for starting to implement it themselves.

[aidanheerdegen]

Sounds like a great idea.

[spencerkclark]

I think a quick and dirty fix could be to add nc-time-axis as an optional dependency, and convert any cftime.datetime objects encountered in plotting routines to nc_time_axis.CalendarDateTime objects before passing them to matplotlib.

But I wonder if there is potentially room for improvement upstream in cftime? Should we need to convert to these proxy objects before plotting or might there be a way to make cftime objects themselves more friendly to use with matplotlib? That could have benefits for multiple libraries (not just xarray).

@pelson @lbdreyer @ocefpaf (developers of nc-time-axis and Iris) -- it seems you have grappled with this problem a fair bit already. Have you thought about those questions before?

[shoyer]

From a cursory examination, it sure looks (to me) that nc_time_axis could be made to directly support plotting of cftime.datetime classes. This could probably either be done in a separate package or upstream in cftime.

[pelson]

This could probably either be done in a separate package

Why a separate package and not in nc-time-axis? (Or alternatively, just in cftime, as you say)

FWIW SciTools/nc-time-axis#16 tried to get the ball rolling on clarifications around the CalendarDateTime object. That was necessary in the days when there was a single PhoneyDateTime object, that didn't have calendar information associated with it. It seems that some refactoring is very plausible in nc-time-axis at this point.

[shoyer]

Why a separate package and not in nc-time-axis? (Or alternatively, just in cftime, as you say)

Yes, of course, that would probably be preferred. 👍