Intent to deprecate: 64-bit integer NIfTI images

[effigies]

64 bit integer images are easy to generate in nibabel due to numpy semantics, but almost universally incompatible with non-Python tools. We must balance breaking existing Python scripts, the Python library ecosystem, and the wider neuroimaging ecosystem that uses NIfTI as an interchange format.

Recognizing that we are currently contributing to breakage in the wider ecosystem, we have decided to deprecate 64-bit integer image creation except via explicit override. The following path is proposed in order to bring major Python libraries into alignment over a 6-month window through escalating warnings and errors.

NiBabel will take the following steps:

• In NiBabel 4.0.0 (March 2022):
  • Provide dtype= parameter to Nifti1Image() (and similar) to pass to set_data_dtype() after initialization (NF: Pass dtype to Analyze-like images at initialization/serialization, warn on creation of NIfTI images with 64-bit ints (API change) #1082)
  • Warn on Nifti1Image(data, affine) if data.dtype is int64 or uint64 unless dtype or header is provided (NF: Pass dtype to Analyze-like images at initialization/serialization, warn on creation of NIfTI images with 64-bit ints (API change) #1082)
  • Provide nib-cast (name may change) command-line tool to recast existing files to new data type (NF: nib-convert CLI tool #1113)
  • Provide dtype= parameter to Nifti1Image.to_filename() (and similar) to set_data_dtype() before serializing
    • If img.get_data_dtype() is (u)int64 and dtype is not passed, raise a warning, to be promoted to error in 5.0
    • Provide "mask" (uint8), "label" (int16 or int32 depending on values) aliases (ENH: Add static and dynamic dtype aliases to NIfTI images #1096)
  • For both __init__ and to_filename():
    • Raise TypeError if dtype is int or "int" (np.dtype(int) == np.dtype("int") == np.dtype("int64")) so users need to explicitly request np.int64.
  • Add documentation page to link in warnings/errors
• In NiBabel 5.0.0 (September 2022):
  • img.to_filename(dtype=None) to raise exception if img.get_data_dtype() is (u)int64

cc @matthew-brett @jbteves @neurolabusc @mrneont @jeromedockes @NicolasGensollen

Please add anything I've missed or let me know if anything needs adjustment.

[jbteves]

Thanks for writing this up! I would add to the dtype parameter that you should explicitly test to make sure that if somebody passes the int alias, it errors out as invalid so that we avoid the int32/int64 ambiguity when discussing "nifti" int versus the "numpy" int. I'm not sure exactly how the type resolution might work so I wanted to make sure that was down.

[neurolabusc]

@effigies thanks for the concise summary.

I like @jbteves idea, but I doubt there is a way to distinguish an explicit np.int64 from the common way people define int

>>> A = np.array([1,123,0], np.int64)
>>> A.dtype
dtype('int64')
>>> B = np.array([1,123,0], int)
>>> B.dtype
dtype('int64')

[jbteves]

I know this is hacky but you can actually use the __str__ method to disambiguate:

>>> ambiguous = int
>>> explicit = np.int64
>>> print(str(ambiguous))
<class 'int'>
>>> print(str(explicit))
<class 'numpy.int64'>

So you can do something like this:

def check_explicit(x):
    return str(x) != "<class 'int'>"
print(check_explicit(int))
print(check_explicit(np.int64))

which results in

False
True

[jbteves]

Note that the Python int type will actually expand to meet its precision needs, while np.int64 will overflow. So while there are some rules that make this complicated, they are ultimately different types. Most pertinently, we can force the user to be explicit in this case.

[neurolabusc]

@jbteves wow! I did not know that. That is a very nice trick. It does provide a future mechanism for a user to explicitly demand int64 if they really want it.

[matthew-brett]

@jbteves - ah - but it's subtle, because dtype=int, as interpreted by Numpy, doesn't give the infinite size Python int, but gives one of np.int64 or np.int32, depending on your platform. So because:

[ins] In [3]: np.zeros(4, dtype=int).dtype
Out[3]: dtype('int64')

the user might expect the same for dtype=int when saving in Nibabel.

[jbteves]

Yes, I agree! I more meant for the purposes of this particular saving function, if we want a user to be very explicit we can use the string casting to check if they used int alias or provided a fixed precision like np.int64 for us to use, avoiding the ambiguity referenced on our Zoom call. Unless you're aware of a way this could be circumvented? I had assumed the function would look like this:

def to_file(self, fname, dtype=None, ...):
    # do stuff
    if not dtype:
          # use the header
    else:
         if isinstance(dtype, str): # do stuff, complain about 'int'
         elif isinstance(dtype, type):
              if str(dtype) == "<class 'int'>" raise ValueError("Please provide a fixed precision")
              # we should be golden now for integers, maybe we want to force the same for floats??

[matthew-brett]

Yes - sure - or maybe

if issubclass(dtype, int):
    raise ValueError("Please ... etc")

?

[jbteves]

That's much less hacky, good idea! I always forget about Python's inheritance features.

[effigies]

Just to make it concrete, I was thinking something quite simple: 46bf959. (This is on the __init__ side. Would do a similar thing on the save side.