planned improvements

[martindurant]

• combine to support direct writing to remote
• support multiple filesystems in ReferenceFileSystem (will be great for multi-resolution, where we can put downsampled images somewhere other than originals) Allow ReferenceFS to refer to multiple FSs filesystem_spec#867
• more hooks for custom functions to manage zarr objects and/or post-process output JSON (Combine2 #122 )
• combine2 and multi-axis concat? ( parts of Add scratch code #83 Combine2 #122)
• consolidate reads in ReferenceFS, where possible, using code made for fsspec.parquet

--

• allow non-uniform chunks in zarr (just like dask.array) (cc @joshmoore , I may have mentioned this idea)
• entrypoints for numcodecs ( support entrypoints? zarr-developers/numcodecs#290 )

[joshmoore]

• I may have mentioned this idea

I definitely remember the discussion, I think from the Oct. 20th community call. One of the examples was chunking for 365 days versus 366 in leap years. I don't have a feeling for the implications though.

[martindurant]

I don't have a feeling for the implications though.

I'm pretty sure very light. In the case that the chunks is a list of lists (dask style), we use a copy of dask's indexing to figure out which slices of which chunks are required. If the chunks is a simple list (all existing data sets), we use the current code.

[rsignell-usgs]

@martindurant , this may already be covered in your list, but I'd like to be able to combine variables that are in different files, to avoid notebooks like this: https://nbviewer.org/gist/rsignell-usgs/1f8f74c3e4c7023d5b919444e42fd23a

[martindurant]

@rsignell-usgs , yes combine2 does this, but I'm not convinced I started in the right direction there. I am glad, though, that the JSONs are simple enough that post-processing can workaround shortcomings in the current code, however painful.