Use ManifestStore in Zarr reader #554
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Nice! |
Co-authored-by: Raphael Hagen <[email protected]>
for more information, see https://pre-commit.ci
norlandrhagen
added a commit
that referenced
this pull request
Apr 24, 2025
* wip toward zarr v2 reader * removed _ARRAY_DIMENSIONS and trimmed down attrs * WIP for zarr reader * adding in the key piece, the reader * virtual dataset is returned! Now to deal with fill_value * Update virtualizarr/readers/zarr.py Co-authored-by: Tom Nicholas <[email protected]> * replace fsspec ls with zarr.getsize * lint * wip test_zarr * removed pdb * zarr import in type checking * moved get_chunk_paths & get_chunk_size async funcs outside of construct_chunk_key_mapping func * added a few notes from PR review. * removed array encoding * v2 passing, v3 skipped for now * added missed staged files * missing return * add network * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * conftest fix * naming * comment out integration test for now * refactored test_dataset_from_zarr ZArray tests * adds zarr v3 req opt * zarr_v3 decorator * add more tests * wip * adds missing await * more tests * wip * wip on v3 * add note + xfail v3 * tmp run network * revert * update construct_virtual_array ordering * updated ABC after merge * wip * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * working for v2 and v3, but only local * cleanup test_zarr reader test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cleanup after zarr-python issue report * temp disabled validate_and_normalize_path_to_uri due to issue in zarr-python v3: zarr-developers/zarr-python#2554 * marked zarr integration test skipped b/c of zarr-v3 and kerchunk incompatability * fixes some async behavior, reading from s3 seems to work * lint + uri_fmt * adds to releases.rst * nit * cleanup, comments and nits * progress on mypy * make mypy happy * adds option for AsyncArray to _is_zarr_array * big async rewrite * fixes merge conflict * bit of restructure * nit * WIP on ChunkManifest.from_arrays * v2/v3 c chunk fix + build ChunkManifest from numpy arrays * removed method of creating ChunkManifests from dicts * cleanup * adds xfails to TestOpenVirtualDatasetZarr due to local filesystem zarr issue * some nits after merging w/ main * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates zarr v3 req * lint * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove build_chunk_manifest_from_dict_mapping function since manifest are build from np.ndarrays * tmp ignore lint * remove zarr fill_value skip * fixes network req import in test_integration * bump xarray to 2025.1.1 and icechunk to 0.1.0a10 in upstream * move zarr import into type checking * move zarr import in test_zarr * adding back in missing nbytes property * typing * tmp testing & removing old xfail * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adds back in validate_and_normalize_path_to_uri after upstream zarr fix & vendors concurrent map from zarr-python * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removing kerchunk from zarr integration test * removed zarr manifest + lint * wip on testing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert min-deps change * merge * revert environment.yaml * removed zarr manifest writing * cleanup and consolidation in zarr reader * typing * test_unsupported_zarr_python to zarr v3 * rel path issue? * revert accidental icechunk commit * wip on fixing codecs * cleaup of tests + codecs * renived test_zarr writer * bumping icechunk for now * typing lint * remove zarr writer test * adds Zarr V2 reader not supported exception * updates usage and releases and lints upstream.yaml * lint + clarified some todo/comments * quick nit, removed duplicated entry in ci * removed some comments and reverted pyproject * pyproj de-dup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * util fpaht * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding test to check zarr key format in manifest * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * switched Manifest creation back to dict * cleaned up zarr reader ArrayV3Metadata reading * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * vendor cleanup * merge w/ develop and update construct_virtual_dataset * added _zstd_codec check in get_codec_config to fix numcodecs complaint * mypy lint * mypy lint 2 * lint * typing * adds check for filepath * spelling nit + revert hdf int * removed virtualizarr.zarr + cleanup nits * cleanup + note * updates docs/faq.md data table * revert leading slash * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix bad merge commit * Use ManifestStore in Zarr reader (#554) * Use ManifestStore in Zarr reader * Update virtualizarr/readers/zarr.py Co-authored-by: Raphael Hagen <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Raphael Hagen <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * filepath slash nit * Update docs/faq.md Co-authored-by: Tom Nicholas <[email protected]> * Update virtualizarr/readers/zarr.py Co-authored-by: Tom Nicholas <[email protected]> * Update virtualizarr/readers/zarr.py Co-authored-by: Tom Nicholas <[email protected]> * Update virtualizarr/readers/zarr.py Co-authored-by: Tom Nicholas <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adds back in todo * adds wip test for scalar chunk testing * adds test for scalar zarr + modifies get_chunk_mapping_prefix to accomdate * update localstore to memorystore --------- Co-authored-by: Tom Nicholas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Max Jones <[email protected]> Co-authored-by: Tom Nicholas <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.