Discussion: Support conversion of Arrow `Int8` and `Int16` to `PrimitiveType::Int`

[gruuya]

Presently only Int32 is converted into the corresponding Iceberg type Int

iceberg-rust/crates/iceberg/src/arrow/schema.rs

Line 370 in 42aff04

DataType::Int32 => Ok(Type::Primitive(PrimitiveType::Int)),

I'm wondering is there any downside in just adding Int8 and Int16 in there as well?

The spec suggests that it's fine to promote a smaller into a larger type (i.e. int to long), so this might be fine?

For context I'm working with a pyiceberg table, created from a parquet file with fields such as

  optional int32 cchar (INTEGER(8,true)) = 3;
  optional int32 cint2 (INTEGER(16,true)) = 4;

and thus the corresponding Arrow schema contains the above-mentioned types.

Related: apache/iceberg#2054

[Fokko]

@gruuya I think that would be fine to cast those into an int 👍

In PyIceberg we do the same: https://github.com/apache/iceberg-python/blob/547d881948dfe17c92bdde9e5b63a94d095a110d/pyiceberg/io/pyarrow.py#L1081-L1089

[gruuya]

Nice, thanks!

I've opened a corresponding PR (also took the liberty of adding support for the new Utf8View as well).