Convert RunEndEncoded to Parquet

[vegarsti]

Which issue does this PR close?

• Closes Support converting RunEndEncodedType to parquet #8016.

[vegarsti]

This is handled in #7713

[albertlockett]

I have a feeling this could be because we're not handling REE when handling the arrow type hint here:

arrow-rs/parquet/src/arrow/schema/primitive.rs

Line 40 in 0931768

fn apply_hint(parquet: DataType, hint: DataType) -> DataType {

Because there's not a direct 1:1 mapping for arrow type to parquet type, the parquet writer will write the encoded arrow schema to the parquet file metadata, and will use this to convert the parquet types back to the arrow types when it reads.

We might have to add something like in that schema::primitive::apply_hint functino:

        (_, DataType::RunEndEncoded(_, value)) => {
            let hinted = apply_hint(parquet, value.data_type().clone());
            match &hinted == value.data_type() {
                true => hint,
                false => hinted,
            }
        },

[albertlockett]

Also, if this has been added ^^^ I suspect it might cause an error. If i'm not mistaken, once the arrow type is known, when we construct the reader we'll end up in this builder code:

arrow-rs/parquet/src/arrow/array_reader/builder.rs

Lines 381 to 394 in 1dacecb

	PhysicalType::BYTE_ARRAY => match arrow_type {
	Some(DataType::Dictionary(_, _)) => {
	make_byte_array_dictionary_reader(page_iterator, column_desc, arrow_type)?
	}
	Some(DataType::Utf8View | DataType::BinaryView) => {
	make_byte_view_array_reader(page_iterator, column_desc, arrow_type)?
	}
	_ => make_byte_array_reader(page_iterator, column_desc, arrow_type)?,
	},
	PhysicalType::FIXED_LEN_BYTE_ARRAY => match arrow_type {
	Some(DataType::Dictionary(_, _)) => {
	make_byte_array_dictionary_reader(page_iterator, column_desc, arrow_type)?
	}
	_ => make_fixed_len_byte_array_reader(page_iterator, column_desc, arrow_type)?,

and because there's no match for the arrow datatype DataType::RunEndEncoded, we'll fall into this branch which may error

_ => make_byte_array_reader(page_iterator, column_desc, arrow_type)?,

The quick fix would be to add something like this, which will still cause the array to read back as plan encoded array.

                Some(DataType::RunEndEncoded(_, val)) => {
                    make_byte_array_reader(page_iterator, column_desc, Some(val.data_type().clone()))?
                }

The more involved (and probably more correct) thing to do would be to add a reader implementation for REE. Similarly to what's been done for dictionary arrays
https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/array_reader/byte_array_dictionary.rs

[vegarsti]

You're absolutely right on both counts! By adding the hint, the schema is now REE, and it then fails with invalid data type for byte array reader - RunEndEncoded, but adding the quick fix gets past that. I will commit that now so we at least have that.

I agree that adding a reader implementation for REE sounds more correct.

[vegarsti]

I've pushed an update with your suggestions now -- thank you! It makes sense to add the reader implementation in this PR as well, right?

[albertlockett]

It makes sense to add the reader implementation in this PR as well, right?

Sure, if you have time!

If you don't have time to add in this PR maybe we could maybe add it as a followup (I'd be happy to help with this as well, if needed).

The temporary workaround to get the Parquet column to decode to the correct Arrow type would be to decode the column to the native Arrow type, and then convert to a RunArray by casting. This is less efficient because we materialize a full length array before encoding it to REE.

For non-byte types (numberic types), this would already be happening. We follow this branch when creating the reader:

arrow-rs/parquet/src/arrow/array_reader/builder.rs

Lines 354 to 358 in 1dacecb

	Box::new(PrimitiveArrayReader::<Int32Type>::new(
	page_iterator,
	column_desc,
	arrow_type,
	)?) as _

And then in the PrimitiveArrayReader we do the cast here:

arrow-rs/parquet/src/arrow/array_reader/primitive_array.rs

Line 484 in 1dacecb

_ => arrow_cast::cast(&array, target_type)?,

For byte types (String, Binary, etc), we'd need to do something like this:
albertlockett@f3d072f

[vegarsti]

Yeah, let's do the reader in another PR after all!

I've applied your diff -- thank you! The test I added (arrow_writer_run_end_encoded_string) is now failing with Casting from Utf8 to RunEndEncoded here in cast_with_options when the source type is Utf8:

arrow-rs/arrow-cast/src/cast/mod.rs

Lines 1248 to 1250 in 4506998

	_ => Err(ArrowError::CastError(format!(
	"Casting from {from_type:?} to {to_type:?} not supported",
	))),

Similarly, added another test arrow_writer_run_end_encoded_int which also fails on the cast in cast_with_options.

I know there's work on casting in progress here: #7713 (comment)

[albertlockett]

Yeah @vegarsti when I was testing this, I had rebased the changes off of #7713 . Once that PR lands, the cast should stop failing

[vegarsti]

Sweet!

[albertlockett]

@alamb related to your comment here: #3520 (comment)

In the discussion above, we were tentatively thinking that the shortest path to getting REE support would be to decode the column to the native arrow array using the existing readers, and then cast it to a RunArray. Then, maybe adding the dedicated REE reader as a followup. Does that sound like a workable approach?

[alamb]

Yes, I think that sounds like a very good approach -- especially if we focus on the tests for reading/writing REE arrays first so when we go to implement the optimized version we already have good test coverage

[vegarsti]

Thanks so much for your advice @albertlockett! I hope to address this soon