Add ABSStore

[tjcrone]

This adds the ABSStore MutableMapping for reading/writing to Azure Blob Storage. I think this needs more testing so I am happy to have this live in a separate branch for a while. Whatever works best for you.

TODO:

• Add unit tests and/or doctests in docstrings
• Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or
  pytest -v --doctest-modules zarr)
• Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or
  pytest -v zarr)
• PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
• Changes documented in docs/release.rst
• Docs build locally (e.g., run tox -e docs)
• AppVeyor and Travis CI passes
• Test coverage to 100% (Coveralls passes)

[martindurant]

I would like to copy across from the initial issue the idea that file-system implementations should follow a convention of some sort - not necessarily subclassing from my definition, but possibly.
Again, I realise that this is not a file-system exactly, but a mapping (it doesn't produce file-like objects or support generic file-system operations), but you might find it interesting to compare to the simplest file-system implementations I've been involved in: local and http.

It may not be worthwhile from the point of view of zarr, but I imagine it would be appreciated by a wider community to separate the one class into ABSMapping, ABSFileSystem, ABSFile (in this implementation, this only supports one-shot read() and mode='r' - although range requests are available for some blob types).

I will not push the issue further :)

For the PR as it stands, I would prefer the two utility functions to be static methods of the class, since I think they are very much tied to the functionality of this class.

For testing, you would either have to build a mock server (tricky) or use vcrpy as gcsfs does (awkward and too much additional complexity to live within zarr).

[alimanfoo]

Just to say I'm about to go offline for 3 weeks and unfortunately have run out of time to take any action on this. @tjcrone can I suggest you go ahead and merge this into a new branch if that would be useful for you. I noticed you have this going into "zarr-developers:aws_store", it might be good to rename that to "abs_store" or "azure_store" just to make sure we don't get confused between our cloud vendors :-) I leave it to your decision whether to stick with a minimal Mapping implementation or follow @martindurant's suggestion. Either way, if you document the new features as experimental then you should have some latitude to change course in future.

[martindurant]

not "azure_store", if that's ok - the datalake-store does remain a valid option, after all.

[alimanfoo]

Good point, was just thinking anything other than "aws" :-)

…

On Friday, 3 August 2018, Martin Durant ***@***.***> wrote: not "azure_store", if that's ok - the datalake-store does remain a valid option, after all. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QgE5CHcIvAdHSQdSk06aXXH7oHzGks5uNHpqgaJpZM4Vt3Pv> .

-- If I do not respond to an email within a few days, please feel free to resend your email and/or contact me by other means. Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health Big Data Institute Li Ka Shing Centre for Health Information and Discovery Old Road Campus Headington Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 or +44 (0)7866 541624 Email: [email protected] Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: @alimanfoo <https://twitter.com/alimanfoo>

[tjcrone]

@alimanfoo, thanks for catching that. i will change the name of the branch to abs_store, to refer to Azure Blob Storage, which is what it was supposed to be. I will go ahead and merge into that branch which I think will be helpful now that it seems we have people using this feature and it would be great to make it easier for them to test.

Regarding @martindurant's suggestion regarding the overall structure, I don't think that I am knowledgeable enough to make this sort of change myself. I have structured the MutableMapping based on the others there now. If we want to take this in a different direction that's very fine by me. For now I will just keep it simple.

Regarding the location of the utility functions, I am happy to move them into the class. However I think they were originally placed outside the class by @rabernat because they would be ostensibly useful for any blob storage classes. I will go ahead and move them in now. They can easily be moved out if that makes more sense at a later time.

I would greatly appreciate getting some guidance on building automated tests for this when you are back online @alimanfoo.

[alimanfoo]

Thanks @tjcrone, sounds good. FWIW I don't have a lot of experience of testing against remote services like this so would defer to others on the best approach. I might be tempted to reach for mocking first but that's just a hunch.

…

-- If I do not respond to an email within a few days, please feel free to resend your email and/or contact me by other means. Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health Big Data Institute Li Ka Shing Centre for Health Information and Discovery Old Road Campus Headington Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 or +44 (0)7866 541624 Email: [email protected] Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: @alimanfoo <https://twitter.com/alimanfoo>

[tjcrone]

@mzjp2, the ABSStore is now merged into the abs_store branch of this repo, so I recommend continuing your testing with the code in this repo going forward. Thanks.

[jakirkham]

FWIW opened issue ( https://github.com/zarr-developers/zarr/issues/290 ) to discuss cloud support generally. Please feel free to share anything relevant there.

[jakirkham]

So PR ( #345 ) has integrated this into master with some changed. Do we still need the abs_store branch or can we delete it?

[shikharsg]

I think we can delete it