[Variant] Casting errors behavior support in `variant_get`

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736

We introduced the basic variant_get kernel in and now we need to expand its feature set

• [Variant] Add variant_get and Shredded VariantArray #8021

However, it doesn't really handle the case when the variant contains a int column but the output calls for a float, for example

Describe the solution you'd like

I would like to handle this case within the variant_get kernel

Describe alternatives you've considered

I personally suggest we follow the rules in the existing arrow cast kernel, likely simply calling that kernel for shredded values, and implementing the equivalent logic for unshredded variants.

@scovich proposed https://github.com/apache/arrow-rs/pull/8021/files#r2257527874

I guess we could pursue three potential levels of casting:

Widening casts (based on type). E.g. if the type is int32, we can also handle int8, int16 and even decimal4 (if scale=0). The conversion is guaranteed to succeed losslessly.
Lossless narrowing casts (based on value). E.g. if the type is int32, we can still handle a small int64 value like 42. TBD what should happen if any one value failed to cast?
Converting casts. These are the lossy ones, e.g. converting 3.14f64 to 3i32, or 3i32 to string. I'm guessing we don't want to mess with these?

Additional context

[scovich]

I personally suggest we follow the rules in the existing arrow cast kernel, likely simply calling that kernel for shredded values, and implementing the equivalent logic for unshredded variants.

Thinking out loud here --

The engine and/or user may have very different opinions than arrow about casting semantics, which is why I've been concerned about "just" doing a cast internally. Even with cast options I don't think it gives enough control.

However -- I think the caller can just provide a schema where all leaf columns are VARIANT, if they want control over casting semantics? The resulting VariantArray would come back unchanged in its original form (shredded or unshredded). At worst, we might have to extract a subset of bytes if some requested leaf path penetrated an unshredded field.

[alamb]

The engine and/or user may have very different opinions than arrow about casting semantics, which is why I've been concerned about "just" doing a cast internally. Even with cast options I don't think it gives enough control.

In my opinion since this is all in the context of parquet-variant-compute -- namely manipulating variants using Arrow arrays -- I think it is actually fine to only provide arrow casting semantics.

The resulting VariantArray would come back unchanged in its original form (shredded or unshredded). At worst, we might have to extract a subset of bytes if some requested leaf path penetrated an unshredded field.

Yes, this is my ideal outcome -- that the library is flexible enough people can implement whatever semantics if what is provided by variant_get is insufficient