Interest in supporting image pyramid?

[mmorehea]

Zarr is ideal for operating on data with multiple reads and writes, but doesn't perfectly support visualization of massive datasets. This task usually requires some form of octree or image pyramid.

I could envision an optional function in the library that would take an existing Zarr directory and add a set of down-sampled blocks.

This would remove a lot of storage space requirements for processing then visualization. Without these additional layers, visualization software will need to create a identical copy of all of the high resolution data, followed by the down sampling, more than doubling the needed storage (1 copy in zarr, 1 copy + down sampled layers in another format).

With this optional set of blocks, the storage overhead is very small for 3D data, ~14% of the original high resolution blocks.

Of course this can be down relatively easily by anyone with numpy, but native support would enforce a standard which would be great.

[alimanfoo]

Hi @mmorehea, thanks a lot for raising this, I think there are many folks who would like to have a common convention for this. This issue goes beyond the Python implementation and relates to the underlying specifications, and we have a separate repo for spec-related discussions here: https://github.com/zarr-developers/zarr-specs. There is an existing issue there which covers this I think and has some relevant discussion: zarr-developers/zarr-specs#23. OK to close this issue in favour of zarr-developers/zarr-specs#23?

[joshmoore]

👍 for having your feedback on zarr-developers/zarr-specs#23, @mmorehea. Additionally, feel free to speak up on https://forum.image.sc/t/next-generation-file-formats-for-bioimaging/31361/9 -- from my side, the subresolutions are a great feature to start with.

[mmorehea]

Hi @joshmoore and @alimanfoo ,

Yes, that sounds great, thanks for pointing me in the right direction. Sorry, I tried searching around but did not check that repo -- will close this now and look there.

[rabernat]

Maybe we should add a mention of the spec discussion repo to the zarr-python issue template?

[joshmoore]

@mmorehea : please see zarr-developers/zarr-specs#50 for current thoughts on pyramids.