zarr import fails with undefined numcodecs.Zstd

[mschubert]

Zarr version

v2.18.2

Numcodecs version

v0.12.1

zstd version

v1.5.6

Python Version

3.12

Operating System

Gentoo

Installation

custom ebuild

Description

After installing zarr, its import fails with:

>>> import zarr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/site-packages/zarr/__init__.py", line 2, in <module>
    from zarr.codecs import *
  File "/usr/lib/python3.12/site-packages/zarr/codecs.py", line 3, in <module>
    from numcodecs import get_codec, Blosc, Pickle, Zlib, Zstd, Delta, AsType, BZ2
ImportError: cannot import name 'Zstd' from 'numcodecs' (/usr/lib/python3.12/site-packages/numcodecs/__init__.py)

numcodecs does indeed not define Zstd, but either numcodecs.zstd or numcodecs.zstd.Zstd. If I understand correctly this symbol is created with:

# @numcodecs/__init__.py
with suppress(ImportError):
    from numcodecs import zstd
    from numcodecs.zstd import Zstd

    register_codec(Zstd)

However, if an error occurs here, the underlying message is never shown to the user. Would it make sense to be more transparent?

>>> from numcodecs import zstd
ImportError: /home/schubert/gentoo/usr/lib/python3.12/site-packages/numcodecs/zstd.cpython-312-x86_64-linux-gnu.so: undefined symbol: STD_compressBlock_btopt

Here the actual error is shown (an incompatibility with zstd=1.5.6? which should ideally be caught at compilation?)

Please note that this may be better addressed by numcodecs than zarr - if so, apologies for reporting to the wrong package.

Steps to reproduce

import zarr

Additional output

No response

[dstansby]

I'm guessing this was introduced by #1914, which added the Zstd import but didn't add zstd as a dependency for zarr.

[rabernat]

Good catch David. What do you think is the best way to resolve this?

[dstansby]

Actually, numcodecs vendors zstd instead of using the zstd package, so this is probably a bit more complex than what I suggested. I think what should happen is this line should be removed from numcodecs:

https://github.com/zarr-developers/numcodecs/blob/a7b8909f7f77a772cf2e4aa06b0547a52a7a72cf/numcodecs/__init__.py#L59

so that import errors are propagated, as import errors here are an issue with numcodecs itself and not just a missing dependency.

[dstansby]

Could you have a go with the latest version of numcodecs? A newer version should give a more helpful error message...

[mschubert]

The import error no longer occurs for me with a recent version of numcodecs, so I'd consider this solved (and the improved error message is a bonus)

Thank you!