Zarr file readable in zarr 2.x but not 3.0.x

[briantoby]

Zarr version

v3.0.4 and 3.0.5

Numcodecs version

0.15.1

Python Version

3.13

Operating System

Windows and Mac

Installation

conda

Description

The program MIDAS (@marinerhemant) writes files using the Zarr specification. These files can be read with older versions of Zarr (as late as 2.18.4) but not 3.0.x (tried with .4 and .5). I am not sure if this is a misreading of the zarr specification, or something is incomplete there, or an inadequacy in zarr 3. More information on what is going wrong is needed in any case.

Steps to reproduce

With this file:
Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysis.MIDAS.zip.caked.hdf.zarr.zip this statement is sufficient:
zarr.open('Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysis.MIDAS.zip.caked.hdf.zarr.zip',mode='r')

Additional output

The error message from zarr is Unable to find group: file:///{loc}/Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysis.MIDAS.zip.caked.hdf.zarr.zip

Full traceback:

>>> zarr.open('/Users/toby/shared/Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.\
h5.analysis.MIDAS.zip.caked.hdf.zarr.zip',mode='r')
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    zarr.open('/Users/toby/shared/Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysis.MIDAS.zip.caked.hdf.zarr.zip',mode='r')
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/api/synchronous.py", line 193, in open
    obj = sync(
        async_api.open(
    ...<7 lines>...
        )
    )
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/core/sync.py", line 163, in sync
    raise return_result
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/api/asynchronous.py", line 330, in open
    return await open_group(store=store_path, zarr_format=zarr_format, mode=mode, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toby/mamba310/envs/313_zarr3/lib/python3.13/site-packages/zarr/api/asynchronous.py", line 827, in open_group
    raise FileNotFoundError(f"Unable to find group: {store_path}")
FileNotFoundError: Unable to find group: file:///Users/toby/shared/Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysis.MIDAS.zip.caked.hdf.zarr.zip
>>> 

[jhamman]

Thanks for the report @briantoby. I think the issue here is that we haven't taught Zarr-Python 3 how to "auto discover" zip stores yet. We should fix that with a general solution that works across store types.

In the meantime, you can open the ZipStore manually:

In [78]: store = zarr.storage.ZipStore("/Users/jhamman/Downloads/Ceria_850mm_100x100_0p5s_att_1_aero_0_000580.h5.analysi
    ...: s.MIDAS.zip.caked.hdf.zarr.zip", mode="r")

In [79]: g = zarr.open_group(store, mode='r')

In [80]: g.tree()
Out[80]: 
/
├── InstrumentParameters
│   ├── Distance (1,) float64
│   ├── Lam (1,) float64
│   ├── Polariz (1,) float64
...

[briantoby]

@jhamman This was very helpful (along with the detour into -- new to me -- asyncio). With this I could adapt my code so that it could read the files with either version of the zarr module. Case closed AFAIC.

BTW, I don't think your edit works because zarr.open_group has a co-routine. I could not get things to work without an async routine and an await.

[marinerhemant]

Thanks @briantoby and @jhamman for solving this.