Order of keys created by `ArrayV2Metadata.to_dict` has changed between versions

[TomNicholas]

Zarr version

v3.1.0 vs v3.0.9

Numcodecs version

n/a

Python Version

3.12

Operating System

linux

Installation

uv pip

Description

In v3.0.9:

In [5]: ArrayV2Metadata(
   ...:             chunks=(10,),
   ...:             shape=(5,),
   ...:             dtype=np.dtype("int32"),
   ...:             order="C",
   ...:             fill_value=None,
   ...:         ).to_dict()
Out[5]: 
{'shape': (5,),
 'chunks': (10,),
 'fill_value': None,
 'order': 'C',
 'filters': None,
 'dimension_separator': '.',
 'compressor': None,
 'attributes': {},
 'zarr_format': 2,
 'dtype': '<i4'}

but in v3.1.0:

In [6]: ArrayV2Metadata(
   ...:             chunks=(10,),
   ...:             shape=(5,),
   ...:             dtype=parse_data_type(np.dtype("int32"), zarr_format=2),
   ...:             order="C",
   ...:             fill_value=None,
   ...:         ).to_dict()
Out[6]: 
{'shape': (5,),
 'chunks': (10,),
 'dtype': '<i4',
 'fill_value': None,
 'order': 'C',
 'filters': None,
 'dimension_separator': '.',
 'compressor': None,
 'attributes': {},
 'zarr_format': 2}

All the fields are the same, but their order is different.

I don't know if this is meant to be preserved between versions or not (it's private API, and unsure if anythig in zarr relies on this), but it was enough of a difference to break virtualizarr (one of a few problems in zarr-developers/VirtualiZarr#677). In our case we care about the order because we take the result and serialize it as json as part of kerchunk reference files, and we had tests checking against the result of older versions.

Maybe you could argue that if it still round-trips via JSON then this shouldn't matter, and VirtualiZarr / Kerchunk are in the wrong?

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr
# your reproducer code
# zarr.print_debug_info()

Additional output

No response

[d-v-b]

The array metadata classes model the zarr array metadata documents, which are JSON objects and thus unordered. Additionally, it's generally not a good idea to rely on key order for python dicts. Which tests or functionality are relying on the order of keys in a JSON document?

[d-v-b]

Since the set of keys is fixed, if the order matters then you can iterate over the keys in the correct order. But from the Zarr POV, the array metadata is unordered.

[TomNicholas]

Yes, I think this may be an artifact of Kerchunk / the way we are testing Kerchunk. I'll close this once I'm certain.

[TomNicholas]

See context here: zarr-developers/VirtualiZarr#679

I think this issue can probably be closed.