implement `cast_to_variant` kernel to cast native types to `VariantArray` #8044

alamb · 2025-08-05T16:01:46Z

Which issue does this PR close?

We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.

Closes [Variant] Implement cast_to_variant kernel #8043

Rationale for this change

As @Samyak2 suggested on https://github.com/apache/arrow-rs/pull/8021/files#r2249926579, having the ability to convert FROM a typed value to a VariantArray will be important

For example, in SQL it could be used to cast columns to variant like in some_column::variant

What changes are included in this PR?

Add cast_to_variant kernel to cast native types to VariantArray
Tests

Are these changes tested?

yes

Are there any user-facing changes?

New kernel

Samyak2

This is nice!

Samyak2 · 2025-08-05T18:23:34Z

parquet-variant-compute/src/cast_to_variant.rs

+    Ok(builder.build())
+}
+
+// TODO add cast_with_options that allow specifying


I'm assuming this will go in a separate issue/PR

Yeah, that is probably a good idea -- I can file tickets for it tomorrow hopefully

I thought more about this, and since there is (currently) no way for a conversion from Variant --> Array to fail, that means the cast options are less obviously relevant (and thus I can't file a ticket as I don't know what it should do)

I updated the comment to reflect this

alamb · 2025-08-05T16:33:39Z

parquet-variant-compute/src/cast_to_variant.rs

+            primtive_conversion!(Float64Type, input, builder);
+        }
+        dt => {
+            return Err(ArrowError::CastError(format!(


There are a bunch of follow on tickets we can perhaps file as follow on work:

Support for timestamps

Support for string / binary arrays

Support for various timestamps types

alamb · 2025-08-05T20:54:56Z

parquet-variant-compute/src/cast_to_variant.rs

+    Ok(builder.build())
+}
+
+// TODO add cast_with_options that allow specifying


Yeah, that is probably a good idea -- I can file tickets for it tomorrow hopefully

scovich

Running out of time for today, so will flush what little I have so far.

Tomorrow is another day.

scovich · 2025-08-05T20:56:56Z

parquet-variant-compute/src/cast_to_variant.rs

+
+/// Convert the input array of a specific primitive type to a `VariantArray`
+/// row by row
+macro_rules! primtive_conversion {


Suggested change

macro_rules! primtive_conversion {

macro_rules! primitive_conversion {

scovich · 2025-08-05T21:27:12Z

parquet-variant-compute/src/cast_to_variant.rs

+    let input_type = input.data_type();
+    // todo: use `downcast_primitive` to avoid the boilerplate and match more types
+    match input_type {
+        DataType::Int8 => {
+            primtive_conversion!(Int8Type, input, builder);
+        }
+        DataType::Int16 => {
+            primtive_conversion!(Int16Type, input, builder);
+        }
+        DataType::Int32 => {
+            primtive_conversion!(Int32Type, input, builder);
+        }
+        DataType::Int64 => {
+            primtive_conversion!(Int64Type, input, builder);
+        }
+        DataType::UInt8 => {
+            primtive_conversion!(UInt8Type, input, builder);
+        }
+        DataType::UInt16 => {
+            primtive_conversion!(UInt16Type, input, builder);
+        }
+        DataType::UInt32 => {
+            primtive_conversion!(UInt32Type, input, builder);
+        }
+        DataType::UInt64 => {
+            primtive_conversion!(UInt64Type, input, builder);
+        }
+        DataType::Float32 => {
+            primtive_conversion!(Float32Type, input, builder);
+        }
+        DataType::Float64 => {
+            primtive_conversion!(Float64Type, input, builder);
+        }


Not sure what downcast_primitive is, but if we used downcast_primitive_array! macro, I think the code simplifies to just:

Suggested change

let input_type = input.data_type();

// todo: use `downcast_primitive` to avoid the boilerplate and match more types

match input_type {

DataType::Int8 => {

primtive_conversion!(Int8Type, input, builder);

}

DataType::Int16 => {

primtive_conversion!(Int16Type, input, builder);

}

DataType::Int32 => {

primtive_conversion!(Int32Type, input, builder);

}

DataType::Int64 => {

primtive_conversion!(Int64Type, input, builder);

}

DataType::UInt8 => {

primtive_conversion!(UInt8Type, input, builder);

}

DataType::UInt16 => {

primtive_conversion!(UInt16Type, input, builder);

}

DataType::UInt32 => {

primtive_conversion!(UInt32Type, input, builder);

}

DataType::UInt64 => {

primtive_conversion!(UInt64Type, input, builder);

}

DataType::Float32 => {

primtive_conversion!(Float32Type, input, builder);

}

DataType::Float64 => {

primtive_conversion!(Float64Type, input, builder);

}

downcast_primitive_array! {

input => input.iter().for_each(|v| match v {

Some(value) => builder.append_variant(Variant::from(value)),

None => builder.append_null(),

}),

Indeed it does. However, when I make this change, I get the following errorrs (because we need to implement conversion to/from the other types (like Decimal128 --> i128)

I think we may have to handle those cases specially. I will file a follow on ticket to do so.

error[E0277]: the trait bound `parquet_variant::Variant<'_, '_>: std::convert::From<half::binary16::f16>` is not satisfied --> parquet-variant-compute/src/cast_to_variant.rs:72:52 | 72 | Some(value) => builder.append_variant(Variant::from(value)), | ^^^^^^^ the trait `std::convert::From<half::binary16::f16>` is not implemented for `parquet_variant::Variant<'_, '_>` | = help: the following other types implement trait `std::convert::From<T>`: `parquet_variant::Variant<'_, '_>` implements `std::convert::From<&[u8]>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<&str>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<()>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<bool>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<chrono::datetime::DateTime<chrono::offset::utc::Utc>>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<chrono::naive::date::NaiveDate>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<chrono::naive::datetime::NaiveDateTime>` `parquet_variant::Variant<'_, '_>` implements `std::convert::From<f32>` and 12 others

Ah, makes sense.

alamb · 2025-08-06T11:12:15Z

I am going to merge this PR in as @Samyak2 has approved it and I want to file follow on tickets that refer to it. While going through the list it turns out there were quite a few!

alamb · 2025-08-06T11:12:42Z

@scovich and anyone else, I am happy to make follow on PRs to address anything else that might come up with review of this one.

alamb · 2025-08-06T11:22:32Z

scovich

LGTM. Couple nits/thoughts to consider, but nothing significant.

scovich · 2025-08-06T13:54:53Z

parquet-variant/src/variant.rs

+        if let Ok(value) = i8::try_from(value) {
+            Variant::Int8(value)
+        } else {
+            Variant::Int16(value as i16)


value.into() makes clear that the conversion is infallible?

(more below)

it is a good idea -- done in

[Variant] Minor: use From impl to make conversion infallable #8068

scovich · 2025-08-06T13:56:48Z

parquet-variant/src/variant.rs

+        if let Ok(value) = i16::try_from(value) {
+            Variant::Int16(value)
+        } else {
+            Variant::Int32(value as i32)
+        }
+    }


Could also consider doing:

i16::try_from(value).map_or_else(|| Variant::Int32(value.into()), Variant::Int16)

tho it's a bit up in the air whether the compactness/readability trade-off is a net win

I agree it is a tossup -- I personally prefer the if let Ok(...) syntax

Thank you for the comment

@scovich

# Which issue does this PR close? - Follow on to #8044 # Rationale for this change @scovich had a good suggestion here https://github.com/apache/arrow-rs/pull/8044/files#r2257256848: > value.into() makes clear that the conversion is infallible? # What changes are included in this PR? Use `From` impl to make it clear the conversion is infallible and can not lose precision # Are these changes tested? Covered by existing tests # Are there any user-facing changes? No

Add cast_to_variant kernel

23910cd

alamb mentioned this pull request Aug 5, 2025

[Variant] Add variant_get and Shredded VariantArray #8021

Merged

alamb added 2 commits August 5, 2025 12:34

Use normal From conversions

52e461d

cleanup

a28f4bc

github-actions bot added the parquet Changes to the parquet crate label Aug 5, 2025

Samyak2 approved these changes Aug 5, 2025

View reviewed changes

alamb commented Aug 5, 2025

View reviewed changes

scovich reviewed Aug 5, 2025

View reviewed changes

alamb added 3 commits August 6, 2025 06:38

fix typo

bc4f34d

update comments

330a68d

Fix docs

764c027

alamb merged commit c25c5a7 into apache:main Aug 6, 2025
12 checks passed

alamb deleted the alamb/cast_to_variant branch August 6, 2025 11:12

alamb mentioned this pull request Aug 6, 2025

[Variant]: Implement DataType::RunEndEncoded support for cast_to_variant kernel #8064

Closed

scovich reviewed Aug 6, 2025

View reviewed changes

alamb mentioned this pull request Aug 6, 2025

[Variant] Minor: use From impl to make conversion infallable #8068

Merged

This was referenced Aug 20, 2025

[Variant]: Implement DataType::Duration support for cast_to_variant kernel #8194

Closed

[Variant]: Implement DataType::Union support for cast_to_variant kernel #8195

Closed

alamb mentioned this pull request Aug 21, 2025

[Variant] Implement cast_to_variant kernel #8043

Closed

liamzwbao mentioned this pull request Aug 28, 2025

[Variant]: Implement DataType::ListView/LargeListView support for cast_to_variant kernel #8236

Closed

liamzwbao mentioned this pull request Sep 4, 2025

[Variant]: Implement DataType::FixedSizeList support for cast_to_variant kernel #8281

Closed

	macro_rules! primtive_conversion {
	macro_rules! primitive_conversion {

implement cast_to_variant kernel to cast native types to VariantArray #8044

implement cast_to_variant kernel to cast native types to VariantArray #8044

Uh oh!

Conversation

alamb commented Aug 5, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Samyak2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Aug 6, 2025

Uh oh!

alamb commented Aug 6, 2025

Uh oh!

Uh oh!

alamb commented Aug 6, 2025

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

implement `cast_to_variant` kernel to cast native types to `VariantArray` #8044

implement `cast_to_variant` kernel to cast native types to `VariantArray` #8044