Possible backwards incompatibility for unathenticated OPeNDAP access

[d70-t]

We are currently building an intake catalog for a field campaign which refers to several datasets which are accessible via unauthenticated OPeNDAP. One issue that we've come across was that the currently released version (v0.3.1) does not yet support unauthenticated access via the opendap driver (which was introduced in #57). However, it was still possible to access those resources using the netcdf driver. For the current master the situation has turned around, as #62 introduces caching for remotely stored netCDF-files, but that is incompatible with OPeNDAP resources. While I see the usefulness of caching remotely stored netCDF-files, I don't see a way how to properly decide if a URL should be interpreted as downloadable netCDF file or as OPeNDAP resource, so I don't know how to support both, the old and the new behaviour of the netcdf driver.

So, the question would be how should we proceed when building our intake catalog (i.e use netcdf or use opendap as a driver for unauthenticated OPeNDAP access)? And will we have to fear a breaking change at which point all users must update their intake-xarray libraries while we are updating the catalog? The options which I currently see are:

• hope for a quick new release of intake-xarray and use opendapfrom the beginning.
• revert / fix Allow opening of fsspec cache files #62 such that the old behaviour is retained and use netcdf from the beginning
• use netcdf now and opendap later, but that would require a coordinated update between catalog creators and all users

[martindurant]

Happy to do a release here, if that helps

[d70-t]

I think that depends a bit on where you want to go with the intake-xarray project. One way of reasoning would be netcdf driver is for netcdf, no matter where it is stored and opendap driver is for OPeNDAP, no matter how to authenticate, then the current state of the master seems to be good, however that would partly contradict the manual for netcdf and opendap. So the manual should be updated in that case as well and maybe adding #72 would be a good fit in this line.

If you'd instead like to stay to what is written in the docs, than maybe #62 is not a good idea.

Personally I think the first option and a quick new release would be great, but I don't want to spoil other ideas.

[martindurant]

OK, let's wait a bit to hear from the other maintainers. I, personally, am more familiar with the code than with actual usage.

[weiji14]

Agree that tagging a new 0.3.2 release is good (0.3.1 is more than a year old!). It sounds like Option 1 (rewording the manual) would be easiest. Personally I find the current NetCDF driver to be better because it allows for caching (though you're right that it isn't quite intuitive, here's a non-OPeNDAP example), and just use a .netrc file so that authentication can be handled (not explicitly documented, but it works).

[d70-t]

I've opened a potential fix for the docs. While doing this, I came across the default arguments for the opendap driver and am wondering if it wouldn't be more intuitive if the default authentication method would be None in stead.

How do you think about this? Maybe it is a good idea to change this now as well, such that we break multiple things at once in stead of one after another.

[weiji14]

I've opened a potential fix for the docs.

Cool, thanks for being proactive!

While doing this, I came across the default arguments for the opendap driver and am wondering if it wouldn't be more intuitive if the default authentication method would be None in stead.
How do you think about this? Maybe it is a good idea to change this now as well, such that we break multiple things at once in stead of one after another.

It's been mentioned before at #57 (comment). Technically if no password/username is given, it should fallback to no authentication (i.e. None) according to #57 (comment). But perhaps this should be explicitly None, and we'll mark this as a breaking change and tag a v0.4.0 release?

[d70-t]

I can not confirm the behaviour you describe an which was mentioned in 57. When I try to access a dataset where I've removed auth: null in the catalogue, I get the following warning:

UserWarning: OPENID was not set. ESGF connection cannot succeed.

and in fact, a subsequent exception is triggered. So the esgf authentication method from pydap does not fall back gently to no authentication. Also from scrolling over the code, I didn't find a way how this fallback would be implemented.

For completeness: I can access the very same dataset if auth: null is part of the catalogue and I don't have any DAP-related variables in my environment.

I personally wouldn't care if it is 0.3.2 or 0.4.0, but the latter seems to be a bit more fair to other users :-)

[d70-t]

I think with the acceptance of #76, it is pretty clear that #62 is here to stay and thus that unauthenticated access to opendap should go via the opendap driver. Furthermore, I think the possibility of having unauthenticated access to opendap via this driver could be retained in future releases independently of how the netcdf driver evolves. This basically answers the original question of this issue 👍 .

As far as I see, there are currently two remaining open topics:

• Maybe changing the auth: null issue (accept or reject opendap driver: use no authentication as default #77) and
• Defining which version number the upcoming release will have

In order to advance our catalog (my reason for starting this issue), the actual outcome of these questions is not of great importance, but having the current (and hopefully future) status inside of a released version is essential. So @martindurant what do you think? How can we move this issue forward?

[d70-t]

@martindurant thank you very much for including all the suggested changes. However, there's still one thing remaining in this issue: a new release. Initially you mentioned that this would be easy to do for you, so I am wondering when this would happen. What is the current status on this? Are there other currently open topics which should also be solved prior to a next release?

[martindurant]

Thanks for the ping. v0.3.2 now on pypi

[d70-t]

Perfect, thank you so much!