Maintenance on tests, CI and contributor docs

[alimanfoo]

This PR does some maintenance on the testing and CI config, and on the contributor documentation on how to install dependencies and run tests. Changes include:

• Upgrades tox test matrices to use latest versions of numpy.
• Adds use of environment variables ZARR_TEST_ABS, ZARR_TEST_MONGO and ZARR_TEST_REDIS which must be set to "1" in order to run tests that require the corresponding services to be running locally, otherwise those tests are skipped - this simplifies the experience for most contributors who won't want to set up those services or run those tests.
• Reworks the requirements files for simplicity.
• Reworks the contributor guide to show how to install dev environment and run tests, without needing services running locally.
• Uses pytest-doctestplus and pytest-remotedata to ensure that all doctests can also be run without needing services running locally, or needing an internet connection to access remote data.
• Does some tweaking of doctests and handling of numpy compatibility to reduce the number of warnings being emitted during running of unit tests.
• Pins azure-storage-blob for compatibility with the Azure Storage Emulator (resolves Azure tests failing on AppVeyor CI #467).

TODO:

• Changes documented in docs/release.rst
• Docs build locally (e.g., run tox -e docs)
• AppVeyor and Travis CI passes
• Test coverage is 100% (Coveralls passes)

[alimanfoo]

I'm also finding that it's not possible currently to run the tox tests locally because the network stores (redis, mongo, ABS) are not available. It would be good to find some way to avoid running those unless they are on CI.

[jakirkham]

The Azure test failure seems to be an API break. ( #467 ) As a short term fix, we might consider constraining the dependency to a version that we know works.

[alimanfoo]

The Azure test failure seems to be an API break. ( #467 ) As a short term fix, we might consider constraining the dependency to a version that we know works.

I've pinned azure-storage-blob==2.0.1 and ABS tests seem to be passing now on appveyor. xref azure-storage-blog change log. I'm assuming the azure storage emulator does not support 2019-02-02 REST API version, although can't find any release notes.

[alimanfoo]

Appveyor has installed Azure Storage Emulator version 5.9:

"%EMULATOR_LOC%" start
Windows Azure Storage Emulator 5.9.0.0 command line tool

[alimanfoo]

Travis has passed. Appveyor is passing. @jakirkham, @shikharsg, I'd be grateful if you could take a look at this PR.

[alimanfoo]

Hm, I'm looking at the contributing guide (docs/contributing.rst), that could use updating. Also I'd like to find a way to avoid contributors unintentionally running the pymongo, redis or ABS tests when they don't have the relevant services installed locally and getting test failures. Doesn't make for a nice contributor experience. Needs some thought.

[jakirkham]

Seems reasonable to me. Thanks for tackling this @alimanfoo! 🙂

[jakirkham]

We should be able to do things like this to avoid running tests without dependencies.

[alimanfoo]

We should be able to do things like this to avoid running tests without dependencies.

Yes that works as a way of avoiding tests if the client libraries are not installed. But the current contributing docs can lead a developer to install the client libraries, but these tests also require a service (or service emulator) to be running locally, and so the redis, mongo and azure tests fail. This is annoying for a potential contributor, it would be better to guide them to a situation where they don't have the service client libraries installed, which requires some reworking of requirements files and contributor docs.

[alimanfoo]

This has ended up being a bit more of a change than originally planned, but think it needed doing. I've updated the issue description with info on the changes included here.

[alimanfoo]

@jakirkham if you wouldn't mind re-reviewing I'd be grateful.

[shikharsg]

All looks good to me. Thanks for the fix @alimanfoo

[alimanfoo]

Thanks @shikharsg. Just to add, to make life even simpler for contributors I've added some logic to check for an environment variable ZARR_TEST_ABS=1, if not set then azure blob service tests will be skipped. Similar for mongo and redis. I think this is a bit easier to manage because it means those tests will be skipped unless the developer specifically switches them on via the environment variable.

[alimanfoo]

Upside of the new environment variables is that tox works again locally.

[alimanfoo]

Coveralls is currently down for maintenance, so think it's OK to merge this without coveralls passing.

[shikharsg]

Thanks @shikharsg. Just to add, to make life even simpler for contributors I've added some logic to check for an environment variable ZARR_TEST_ABS=1, if not set then azure blob service tests will be skipped. Similar for mongo and redis. I think this is a bit easier to manage because it means those tests will be skipped unless the developer specifically switches them on via the environment variable.

Agree. Seems like a sure short method of ensuring that the developer actually wanted to run the tests.

[alimanfoo]

This seems like a useful thing to get merged soon to help with other PRs, so I'll merge later today if no objections.