Zstd: ZSTD_getDecompressedSize is obsolete and used incorrectly.

[mkitti]

numcodecs currently treats a return value of 0 from ZSTD_getDecompressedSize as an input error. A value of zero could mean one of the following.

1. empty
2. unknown
3. error

numcodecs/numcodecs/zstd.pyx

Lines 151 to 153 in 366318f

	dest_size = ZSTD_getDecompressedSize(source_ptr, source_size)
	if dest_size == 0:
	raise RuntimeError('Zstd decompression error: invalid input data')

Rather numcodecs should use ZSTD_getFrameContentSize which the return value can be differentiated.

1. 0 means empty
2. 0xffffffffffffffff, ZSTD_CONTENTSIZE_UNKNOWN, means unknown
3. 0xfffffffffffffffe, ZSTD_CONTENTSIZE_ERROR, means error

See zstd.h or the manual for a reference.
https://github.com/facebook/zstd/blob/7cf62bc274105f5332bf2d28c57cb6e5669da4d8/lib/zstd.h#L195-L203
https://facebook.github.io/zstd/zstd_manual.html

This error arose during the implementation of Zstandard in n5-zarr:
saalfeldlab/n5-zarr#35

There the compressor was producing blocks which would return ZSTD_CONTENTSIZE_UNKNOWN. ZSTD_getDecompressedSize would return 0 and numcodecs would incorrectly interpret this as an error.

Handling ZSTD_CONTENTSIZE_UNKNOWN may be difficult.

1. If a dest buffer is provided, then perhaps that should we set as the expected decompressed size and an error should occur if the decompressed size is not that.
2. If a dest buffer is not provided, we may need to either use a default or use the streaming API to build an growing buffer until all the data is decompressed.

[mkitti]

We might borrow from python-zstandard here:
https://github.com/indygreg/python-zstandard/blob/1dc0596ccce5a3e84467183b35aaab96e445a047/c-ext/decompressor.c#L315-L325

[d-v-b]

what if we just added python-zstandard as a dependency

[mkitti]

We probably should. My suggestion is that we break numcodecs up into smaller packages focusing on individual upstream codecs and then combine them via some metapackage.

It would also be good if we had a mechanism to manage multiple implementations of the same codec.

[martindurant]

I have mentioned this before, but cramjam includes many of these standard compressors, and already compiles and builds everything in latest rust packages. Simply depending on them would simplify this repo a lot.

[mkitti]

cramjam does not do blosc though. I suppose we could just depend on https://github.com/Blosc/python-blosc2 .

I think we may need a new repository or Github organization for this.

Could someone sketch out an optional dependency package structure and how we make that work both in pip and conda or prefix?

[martindurant]

cramjam does not do blosc though

Correct, but it does deal with issues like this specific one. And indeed, I'd be happy to delegate blosc to someone else too ( or push on milesgranger/cramjam#110 ).

[joshmoore]

This is sounding a bit as if a WG like the zarr-python-refactoring WG might also be appropriate here.

[mkitti]

This was resolved in #519

Of the other Zstandard wrappers in Python, my current favorite is https://pypi.org/project/pyzstd/ .

Also, I think Zstandard is heading for inclusion in c-python itself:
https://peps.python.org/pep-0784/