Spec v3: fix conflicts with main and minor updates to match the current v3 spec #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The comment in the file suggest that the numpy version should be the latest one, it is not. This would help to make sure that there is no regression in numpy, and/or that the test suite would catch numpy deprecated functionality.
In case where 'shape' is passed as argument, this will avoid calling "contains_array" which does store queries, as well as avoid testign if the array contains a group as anyway this is what we were going to do in the else case. This does change behavior in edge case: - if shape is passed and the key is a group, we'll try to open as array instead of groups. - if contains_array or contains_group would have raise we might skip over those checks. Though both of the above case should not be found in user code and are bugs AFAICT, and would just fail differently.
* CLN : Use of super modified: zarr/indexing.py modified: zarr/n5.py modified: zarr/storage.py modified: zarr/tests/test_convenience.py * CLN : Remove coding cookies from file utf-8 is the default source code file encoding on python 3 * CLN : Remove u-string prefixes the string prefixes dont do anything on python 3 and are visual noise modified: zarr/tests/test_core.py modified: zarr/tests/test_creation.py modified: zarr/tests/test_filters.py modified: zarr/tests/test_hierarchy.py modified: zarr/util.py * CLN : Remove conditional The conditional always evaluates to True on Python 3 so the rest of the if/elif/else conditional has been removed modified: zarr/storage.py * DOC : Add an entry in release doc modified: docs/release.rst * CLN : Fix a pep8 issue remove the import entirely and use os.replace instead of replace modified: zarr/storage.py
use getitems with FSStore for concurrent reading
use `build` to build both a sdist and a platform independant wheel on tagging.
* Remove error utils function (not used anymore) Convert also one more err_ function into a proper exception that derives from ValueError. * turn more function into errors * use VindexInvalidSelectionError
Co-authored-by: jakirkham <[email protected]>
…arr-developers#715) * v2 spec: add optional dimension_separator (see zarr-developers#707) Various implementations allow for defining the separator between the dimension indexes when writing chunks: * n5-zarr defines a `dimensionSeparator` parameter; * zarr-python's NestedDirectoryStore does so by default * and FSStore provides a `key_separator` parameter; * tensorstore has a `key_encoding` parameter; and * jzarr is looking to add the same functionality. When writing an array, it is straight-forward to set this separator and have arrays properly configured. Consumers of such arrays, however, must either know *a priori* if their arrays use a non-default separator or must loop through all possible chunks keys searching for the right one. By defining adding an optional metadata key to the .zarray, we: * preserve the efficient configuration of arrays * while keeping the v2 spec backwards compatible. The primary downsides are that this will be the first optional metadata value in the v2 spec and therefore we don't have a strong understanding of how that will play out, and datasets which were previously written with non-default separators will need updating in order to enable the detection though that is no worse than the current situation. * Update dim. sep. description after feedback * Remove `MUST NOT` restriction for other keys
…developers#715) (zarr-developers#716) * Implement `dimension_separator` for Python storage classes (See zarr-developers#715) * All top-level storage classes now take an optional `dimension_separator` parameter which defaults to `None`, but can also be `.` or `/`. * A ValueError is raised at normalization time if this is not the case. * `None`s are normalized to the default of `.` in all except the NestedDirectoryStore case. * The value is stored as `self._dimension_separator` on participating classes so that array creation can lookup the value. * This value deprecates the `key_separator` value from FSStore. * Wrapper classes like LRUCacheStore and ConsolidatedMetadataStore *do not* follow this pattern and instead rely on the value in the underlying store. * Only store `dimension_separator` if not None All hexdigest tests were failing due to updated array metadata. In the case of NestedDirectoryStore and N5Store, this is necessary. If the dimension_separator key is excluded from the .zarray JSON when None, then most standard tests continue to pass. * Fix doctests with optional key * Add separator to missed LDBMStore * Fix linting issue * De-deprecate key_separator as public, non-null API * Add test for normalize_dim_sep to appease codecov * More tests for codecov * Remove key from n5 array metadata * Fix minor typo * Cleanup DIGESTS in test_core.py * Fix cut-n-paste error in test_utils.py * And hopefully on last codecov fix * Apply review changes * Add 2.8.0 release notes
Jupyterlab 3 changed how widgets are installed. Now both the frontend and backend are installed via pip/conda. If a user also then installs the front using the `jupyter labextension install` command then they can end up breaking things by trying to install an old version. Co-authored-by: jakirkham <[email protected]>
…ers#718)" (zarr-developers#729) This reverts commit 449ed26.
…arr-developers#730) * add section on rechunking to chunk optimisation section of tutorial * proofreading
Bumps [numpy](https://github.com/numpy/numpy) from 1.19.2 to 1.20.3. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/HOWTO_RELEASE.rst.txt) - [Commits](numpy/numpy@v1.19.2...v1.20.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pymongo](https://github.com/mongodb/mongo-python-driver) from 3.9.0 to 3.11.4. - [Release notes](https://github.com/mongodb/mongo-python-driver/releases) - [Changelog](https://github.com/mongodb/mongo-python-driver/blob/3.11.4/doc/changelog.rst) - [Commits](mongodb/mongo-python-driver@3.9.0...3.11.4) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ipytree](https://github.com/martinRenou/ipytree) from 0.1.3 to 0.2.1. - [Release notes](https://github.com/martinRenou/ipytree/releases) - [Changelog](https://github.com/martinRenou/ipytree/blob/master/RELEASE.md) - [Commits](https://github.com/martinRenou/ipytree/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [setuptools-scm](https://github.com/pypa/setuptools_scm) from 3.3.3 to 6.0.1. - [Release notes](https://github.com/pypa/setuptools_scm/releases) - [Changelog](https://github.com/pypa/setuptools_scm/blob/main/CHANGELOG.rst) - [Commits](pypa/setuptools-scm@v3.3.3...v6.0.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [redis](https://github.com/andymccurdy/redis-py) from 3.3.8 to 3.5.3. - [Release notes](https://github.com/andymccurdy/redis-py/releases) - [Changelog](https://github.com/andymccurdy/redis-py/blob/master/CHANGES) - [Commits](redis/redis-py@3.3.8...3.5.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [flake8](https://gitlab.com/pycqa/flake8) from 3.8.3 to 3.9.2. - [Release notes](https://gitlab.com/pycqa/flake8/tags) - [Commits](https://gitlab.com/pycqa/flake8/compare/3.8.3...3.9.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* FSStore: aflesh out incomplete test and add another. The first test passes after changes to FSStore. The second test fails. * pep8 fixes * TestNestedFSStore: tweak second assertion of test_numbered_groups * FSStore: change ggetitems to return dict with input keys * TestArrayWithFSStore: add key_separator kwarg FSStore constructor * TestArrayWithFSStore: add key_separator arg to store constructor in create_arrray. * revert changes. the logic I need to test actually lives in test_core, not test_storage. * kill some whitespace * add nested tfsstore tests * FSStore: fsstore.listdir now handles nested keys * FSStore: re-order conditional evaluation in listdir * FSStore: use self.fs.find in listdir * Add tests from zarr-developers#718 * Apply suggestion from @grlee77 * Update PartialRead hexdigest values * More hexdigest updates Co-authored-by: jmoore <[email protected]>
Bumps [fasteners](https://github.com/harlowja/fasteners) from 0.15 to 0.16. - [Release notes](https://github.com/harlowja/fasteners/releases) - [Changelog](https://github.com/harlowja/fasteners/blob/master/ChangeLog) - [Commits](https://github.com/harlowja/fasteners/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump s3fs from 0.5.1 to 2021.5.0 Bumps [s3fs](https://github.com/dask/s3fs) from 0.5.1 to 2021.5.0. - [Release notes](https://github.com/dask/s3fs/releases) - [Changelog](https://github.com/dask/s3fs/blob/main/release-procedure.md) - [Commits](fsspec/s3fs@0.5.1...2021.05.0) Signed-off-by: dependabot[bot] <[email protected]> * Bump fsspec as well Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: jmoore <[email protected]>
Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.7.1 to 2.12.0. - [Release notes](https://github.com/pytest-dev/pytest-cov/releases) - [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest-cov@v2.7.1...v2.12.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [tox](https://github.com/tox-dev/tox) from 3.14.0 to 3.23.1. - [Release notes](https://github.com/tox-dev/tox/releases) - [Changelog](https://github.com/tox-dev/tox/blob/master/docs/changelog.rst) - [Commits](tox-dev/tox@3.14.0...3.23.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…arr-developers#759) * initial attempt to update azure storage version in zarr * add getsize back and fixup listdir * overwrite existing blob in setitem * Updates for azure.storage.blob * deprecate old * fixup * doc * doc2 * linting * linting * fixup * avoid set * Rely on Zarr being present * skip for py36 * fix skip * flake8 * Fix coverage, deprecations * Fix contributing typo Co-authored-by: Joseph Hamman <[email protected]> Co-authored-by: Joe Hamman <[email protected]> Co-authored-by: jmoore <[email protected]> Co-authored-by: Josh Moore <[email protected]>
add dimension_separator to decode_array_metadata add dimension_separator to TestV3Adapter.create_store method add _dimension_separator to BaseV3Store class add dimension_separator to AsyncV3DirectoryStore, AsyncV3MemoryStore __init__ add dimension_separator to StoreComparer class
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I merged master, fixed the conflicts and then made some small changes for consistency with the current v3 spec draft (see commits starting from d56d416).
I opened this PR for visibility/awareness, but you don't necessarily need to review anything at this point.
TODO: