Blosc silently fails with 2GB buffer

[jeromekelleher]

>>> import numcodecs
>>> import numpy as np                                                                                   
>>> a = np.ones(1024**3, dtype=np.int8)
>>> a.nbytes
1073741824
>>> codec = numcodecs.Blosc()
>>> x = codec.encode(a)
>>> len(x)
4505616
>>> a = np.ones(2 * 1024**3, dtype=np.int8)
>>> x = codec.encode(a)                                                                                  
Input buffer size cannot exceed 2147483631 bytes
>>> codec.decode(x)
# Freezes interpreter

It looks like Blosc is raising this error and it's not being caught here because csize is declared as size_t, and so cannot be negative.

Any thoughts @alimanfoo? Should be an easy fix, but it's not entirely clear how to test this. Do you know of any other ways we can provoke errors in blosc which don't involve mallocing 2GB?

[alimanfoo]

You could try corrupting a compressed buffer. I.e., create some compressed bytes via encode(), then remove some leading or trailing bytes then try to decode().