Unclear support status of Zarr V3 Specification in zarr-python 2.x

[candleindark]

What is the support status of Zarr V3 Specification in zarr-python 2.x? We hope to use zarr-python 2.x to open zarr objects that are in Zarr V3 Specification. (Due to some dependency restriction, we are not able to upgrade zarr-python to 3.x at the moment. See dandi/dandi-cli#1609 for details).

According to the documentation of zarr-python 2.18.5, zarr-python at a version >2.12,<3 includes an experimental implementation of Zarr V3 Specification, but through some experimentation, I have found that the experimental implementation of Zarr V3 Specification in zarr-python is quite different from the final Zarr V3 Specification in zarr-python 3.x.

For example, the following experiment shows that the latest zarr-python at 2.x, 2.18.7 at the moment, is not able to open a simple Zarr object created with the zarr-python at 3.x, 3.0.6 at the moment, that is of the final Zarr V3 Specification.

1. Given the following files.

create.py

# To be used for creating a Zarr object with Zarr V3 Specification using the
# zarr Python package at version 3 or later

import zarr
import numpy as np

z1 = zarr.create_array(
    store="data/example-1.zarr",
    shape=(10000, 10000),
    chunks=(1000, 1000),
    dtype="int32",
)


z1[:] = 42
z1[0, :] = np.arange(10000)
z1[:, 0] = np.arange(10000)

z2 = zarr.open("data/example-1.zarr", mode="r")
print(z2.info)


# Can be invoked with the following command:
# python3.11 -m venv .venv && source .venv/bin/activate && pip install zarr~=3.0 && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 create.py; deactivate; rm -rf .venv

read.py

# To be used for reading a Zarr object with Zarr V3 Specification using the
# zarr Python package at before version 3

import zarr

z1 = zarr.open("data/example-1.zarr", mode="r", zarr_version=3)

# Can be invoked with the following command:
# python3.11 -m venv .venv && source .venv/bin/activate && pip install "zarr<3.0" && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 read.py; deactivate; rm -rf .venv

2. Run the following to write a Zarr object in the final Zarr V3 Specification to the file system using the zarr-python at 3.x.

python3.11 -m venv .venv && source .venv/bin/activate && pip install zarr~=3.0 && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 create.py; deactivate; rm -rf .venv

which produces the following output.

...
zarr version: 3.0.6
Type               : Array
Zarr format        : 3
Data type          : DataType.int32
Shape              : (10000, 10000)
Chunk shape        : (1000, 1000)
Order              : C
Read-only          : True
Store type         : LocalStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : (ZstdCodec(level=0, checksum=False),)
No. bytes          : 400000000 (381.5M)

3. Run the following to read the written Zarr object using zarr-python 2.x, and an error will result due to incompatiblity.

python3.11 -m venv .venv && source .venv/bin/activate && pip install "zarr<3.0" && python -c "import zarr;print(f'zarr version: {zarr.__version__}')" && python3.11 read.py; deactivate; rm -rf .venv

which produces the following output.

...
zarr version: 2.18.7
Traceback (most recent call last):
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/read.py", line 6, in <module>
    z1 = zarr.open("data/example-1.zarr", mode="r", zarr_version=3)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/convenience.py", line 134, in open
    if contains_array(_store, path):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/storage.py", line 120, in contains_array
    key = _prefix_to_array_key(store, prefix)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 683, in _prefix_to_array_key
    sfx = _get_metadata_suffix(store)  # type: ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 592, in _get_metadata_suffix
    return _get_hierarchy_metadata(store)["metadata_key_suffix"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/_storage/store.py", line 587, in _get_hierarchy_metadata
    return store._metadata_class.decode_hierarchy_metadata(store["zarr.json"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/isaac/Developer/Dartmouth/workshop/zarr/.venv/lib/python3.11/site-packages/zarr/meta.py", line 393, in decode_hierarchy_metadata
    raise ValueError(f"Unexpected keys in metadata. meta={meta}")
ValueError: Unexpected keys in metadata. meta={'shape': [10000, 10000], 'data_type': 'int32', 'chunk_grid': {'name': 'regular', 'configuration': {'chunk_shape': [1000, 1000]}}, 'chunk_key_encoding': {'name': 'default', 'configuration': {'separator': '/'}}, 'fill_value': 0, 'codecs': [{'name': 'bytes', 'configuration': {'endian': 'little'}}, {'name': 'zstd', 'configuration': {'level': 0, 'checksum': False}}], 'attributes': {}, 'zarr_format': 3, 'node_type': 'array', 'storage_transformers': []}

Is there a plan to support the actual/final Zarr V3 Specification in zarr-python 2.x? If there is not, is it possible to have zarr-python output more informative error when opening an actual/final Zarr V3 Specification object, such as a NotImplementedError.

[joshmoore]

I'll leave others to respond more, but in the way of a fast response:

Is there a plan to support the actual/final Zarr V3 Specification in zarr-python 2.x?

No, but your point on a better error message is definitely taken.

You might also want to take a look at #2873 for some further background.

[rabernat]

Zarr Python 2 implemented a preliminary version of the V3 spec that is considerably different from what was eventually agreed upon. There are some users who wrote data using that format, so we have kept that code path around in order to support them.

If you want to write standards-compliant V3 data, you need to upgrade to Zarr Python 3. I'd say we should focus on that issue. What are the dependencies or other issues that are preventing you from upgrading and how can we help you overcome those issues?

[candleindark]

If you want to write standards-compliant V3 data, you need to upgrade to Zarr Python 3. I'd say we should focus on that issue. What are the dependencies or other issues that are preventing you from upgrading and how can we help you overcome those issues?

We are currently blocked by the dependency of hdmf-zarr, hdmf-dev/hdmf-zarr#202. The other issue is that zarr-python 3 doesn't support Python 3.9 and 3.10.

➜  zarr python3.10 -m venv venv && source venv/bin/activate && pip install "zarr~=3.0"
ERROR: Ignored the following versions that require a different python version: 2.18.4 Requires-Python >=3.11; 2.18.5 Requires-Python >=3.11; 2.18.6 Requires-Python >=3.11; 2.18.7 Requires-Python >=3.11; 3.0.0 Requires-Python >=3.11; 3.0.0a6 Requires-Python >=3.11; 3.0.0a7 Requires-Python >=3.11; 3.0.0b0 Requires-Python >=3.11; 3.0.0b1 Requires-Python >=3.11; 3.0.0b2 Requires-Python >=3.11; 3.0.0b3 Requires-Python >=3.11; 3.0.0rc1 Requires-Python >=3.11; 3.0.0rc2 Requires-Python >=3.11; 3.0.1 Requires-Python >=3.11; 3.0.2 Requires-Python >=3.11; 3.0.3 Requires-Python >=3.11; 3.0.4 Requires-Python >=3.11; 3.0.5 Requires-Python >=3.11; 3.0.6 Requires-Python >=3.11
ERROR: Could not find a version that satisfies the requirement zarr~=3.0 (from versions: 0.2.2, 0.2.3, 0.2.5, 0.2.6, 0.2.7, 0.3.0, 0.4.0, 1.0.0b3, 1.0.0b4, 1.0.0b6, 1.0.0, 1.1.0, 2.0.0a2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.5.0, 2.6.1, 2.7.0, 2.7.1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.10.0, 2.10.1, 2.10.2, 2.10.3, 2.11.0a1, 2.11.0a2, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.12.0a1, 2.12.0a2, 2.12.0, 2.13.0a1, 2.13.0a2, 2.13.0, 2.13.1, 2.13.2, 2.13.3, 2.13.6, 2.14.0, 2.14.1, 2.14.2, 2.15.0a1, 2.15.0a2, 2.15.0, 2.16.0, 2.16.1, 2.17.0, 2.17.1, 2.17.2, 2.18.0, 2.18.1, 2.18.2, 2.18.3, 3.0.0a0, 3.0.0a1, 3.0.0a2, 3.0.0a3, 3.0.0a4, 3.0.0a5)
ERROR: No matching distribution found for zarr~=3.0

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

Our package, https://github.com/dandi/dandi-cli, currently supports Python 3.9 and 3.10. I would be helpful if you add support for Python 3.9 and 3.10 as well.

[d-v-b]

Our package, https://github.com/dandi/dandi-cli, currently supports Python 3.9 and 3.10. I would be helpful if you add support for Python 3.9 and 3.10 as well.

I think this would be very difficult for us, as I'm sure we are using python features introduced in 3.11, and more broadly we are committed to following the SPEC 0 recommendations, which entails dropping support for python versions 3 years after release. In fact, we are likely dropping 3.11 before the end of the 2025.

[candleindark]

Our package, https://github.com/dandi/dandi-cli, currently supports Python 3.9 and 3.10. I would be helpful if you add support for Python 3.9 and 3.10 as well.

I think this would be very difficult for us, as I'm sure we are using python features introduced in 3.11, and more broadly we are committed to following the SPEC 0 recommendations, which entails dropping support for python versions 3 years after release. In fact, we are likely dropping 3.11 before the end of the 2025.

Thanks for the clarification.