Zarr v3: TypeError on float fill_value "0.0" for uint8 arrays—compatibility with legacy datasets

[thewtex]

Zarr version

3.1.2

Numcodecs version

0.16.1

Python Version

3.11

Operating System

Linux

Installation

pixi / conda-forge

Description

Attempting to load legacy OME-Zarr datasets (e.g., https://s3.embl.de/i2k-2020/platy-raw.ome.zarr) with zarr-python v3.x using zarr.open_array results in a TypeError if the fill_value is encoded as 0.0 (float) while the dtype is uint8. This previously worked in zarr-python v2.x but fails in the latest versions, breaking compatibility with many published datasets.

Error message

TypeError: Invalid type: 0.0. Expected an integer.

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "requests",
#   "aiohttp",
#   "fsspec",
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr

url = "https://s3.embl.de/i2k-2020/platy-raw.ome.zarr"
component = "s0"  # Try s0, but other resolutions likely affected

# Attempt to open remote array
z = zarr.open_array(store=url, path=component)
print(z)

zarr.print_debug_info()

Additional output

In the metadata for affected arrays (e.g., https://s3.embl.de/i2k-2020/platy-raw.ome.zarr/s0/.zarray), the "fill_value" field is a float (0.0), but the "dtype" is "uint8". This is arguably non-standard, but many OME-Zarr arrays in the wild use this encoding.

Older zarr-python releases (e.g., v2.x) accepted this value. The latest (v3.x) fails, making many public datasets unreadable without editing their metadata.

Questions and Suggestions

• Should zarr-python accept 0.0 as a compatible fill_value for integer arrays and transparently cast it?

• Would it be possible to warn (not error) and interpret common cases (e.g., float-encoded zero for integer types)?

• If strict typing is desired, could an opt-in backward compatibility flag be provided?

CC @tischi @normanrz

[d-v-b]

Should zarr-python accept 0.0 as a compatible fill_value for integer arrays and transparently cast it?

Yes, we should do this. IMO there's no need for a warning or error here.

[d-v-b]

#3448 will fix this, until then here's a workaround:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "requests",
#   "aiohttp",
#   "fsspec",
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

from typing import Any
from numpy import unsignedinteger
from numpy._typing._nbit_base import _8Bit
import zarr
from zarr.core.common import ZarrFormat
from zarr.dtype import UInt8
from zarr.dtype import data_type_registry

class FixedUInt8(UInt8):
    def from_json_scalar(self, data: Any, *, zarr_format: ZarrFormat) -> Any:
        if isinstance(data, float):
           return super().from_json_scalar(int(data), zarr_format=zarr_format)
        return super().from_json_scalar(data, zarr_format=zarr_format)

data_type_registry.register('uint8', FixedUInt8)

url = "https://s3.embl.de/i2k-2020/platy-raw.ome.zarr"
component = "s0"  # Try s0, but other resolutions likely affected

# Attempt to open remote array
z = zarr.open_array(store=url, path=component)
print(z)

zarr.print_debug_info()

The logic here is that you override the json deseralization logic to convert floats to ints, then register that data type under its zarr v3 name (e.g., "uint8"). Repeat as needed for the other data types.

[d-v-b]

would it be helpful to include this kind of example in our documentation? I think overriding data type JSON deserialization behavior is very much in scope for our data type registry design.

[ianhi]

would it be helpful to include this kind of example in our documentation? I think overriding data type JSON deserialization behavior is very much in scope for our data type registry design.

Yes! and ideally the error messages related to this type of issue would point at those docs

[thewtex]

@d-v-b thank you for the fix!