From 0b3bf13d8b266c1b64009d1f9daeff13abb02db2 Mon Sep 17 00:00:00 2001 From: jmoore Date: Thu, 8 Apr 2021 09:50:05 +0200 Subject: [PATCH 1/3] v2 spec: add optional dimension_separator (see #707) Various implementations allow for defining the separator between the dimension indexes when writing chunks: * n5-zarr defines a `dimensionSeparator` parameter; * zarr-python's NestedDirectoryStore does so by default * and FSStore provides a `key_separator` parameter; * tensorstore has a `key_encoding` parameter; and * jzarr is looking to add the same functionality. When writing an array, it is straight-forward to set this separator and have arrays properly configured. Consumers of such arrays, however, must either know *a priori* if their arrays use a non-default separator or must loop through all possible chunks keys searching for the right one. By defining adding an optional metadata key to the .zarray, we: * preserve the efficient configuration of arrays * while keeping the v2 spec backwards compatible. The primary downsides are that this will be the first optional metadata value in the v2 spec and therefore we don't have a strong understanding of how that will play out, and datasets which were previously written with non-default separators will need updating in order to enable the detection though that is no worse than the current situation. --- docs/spec/v2.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/spec/v2.rst b/docs/spec/v2.rst index 7a58f64386..fa8a617865 100644 --- a/docs/spec/v2.rst +++ b/docs/spec/v2.rst @@ -78,6 +78,15 @@ filters filters are to be applied. Each codec configuration object MUST contain a ``"id"`` key identifying the codec to be used. +The following keys MAY be present within the object: + +dimension_separator + A string, optimally of length 1, which defines the separator placed + between the dimensions of a chunk. If the value is not set, then the + default MUST be assumed to ``"."``, leading to chunk keys of the form "0.0". + The most common non-default value is ``"/"``, which leads to nested keys of + the form "0/0" that in most implementations produces a directory structure. + Other keys MUST NOT be present within the metadata object. For example, the JSON object below defines a 2-dimensional array of 64-bit From f2871404db31ce4ffc50a5a5e5d8ab755dcce861 Mon Sep 17 00:00:00 2001 From: jmoore Date: Thu, 15 Apr 2021 07:14:09 +0200 Subject: [PATCH 2/3] Update dim. sep. description after feedback --- docs/spec/v2.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/spec/v2.rst b/docs/spec/v2.rst index fa8a617865..9f028398de 100644 --- a/docs/spec/v2.rst +++ b/docs/spec/v2.rst @@ -81,11 +81,12 @@ filters The following keys MAY be present within the object: dimension_separator - A string, optimally of length 1, which defines the separator placed + If present, either the string ``"."`` or ``"/""`` definining the separator placed between the dimensions of a chunk. If the value is not set, then the - default MUST be assumed to ``"."``, leading to chunk keys of the form "0.0". - The most common non-default value is ``"/"``, which leads to nested keys of - the form "0/0" that in most implementations produces a directory structure. + default MUST be assumed to be ``"."``, leading to chunk keys of the form "0.0". + Arrays defined with ``"/"`` as the dimension separator can be considered to have + nested, or hierarchical, keys of the form "0/0" that SHOULD where possible + produce a directory-like structure. Other keys MUST NOT be present within the metadata object. From 049e4af45a7c922e9c678fc58364f4c3cf124c91 Mon Sep 17 00:00:00 2001 From: jmoore Date: Fri, 16 Apr 2021 12:56:36 +0200 Subject: [PATCH 3/3] Remove `MUST NOT` restriction for other keys --- docs/spec/v2.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/spec/v2.rst b/docs/spec/v2.rst index 9f028398de..5de78ba78c 100644 --- a/docs/spec/v2.rst +++ b/docs/spec/v2.rst @@ -88,7 +88,8 @@ dimension_separator nested, or hierarchical, keys of the form "0/0" that SHOULD where possible produce a directory-like structure. -Other keys MUST NOT be present within the metadata object. +Other keys SHOULD NOT be present within the metadata object and SHOULD be +ignored by implementations. For example, the JSON object below defines a 2-dimensional array of 64-bit little-endian floating point numbers with 10000 rows and 10000 columns, divided