Skip to content

Reading parquet with (pre-release) arrow fails with "out of order projection is not supported" #2543

Closed
@alamb

Description

@alamb

Describe the bug

When testing using a pre-release version of arrow-rs with apache/arrow-rs#1682 from @tustvold which added stricter error checking two datafusion tests fail with "out of order projection is not supported"

---- physical_plan::file_format::parquet::tests::evolved_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_filter' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:968:14

---- physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:819:14

To Reproduce
See #2530

Expected behavior
Tests should succeed

Additional context
@tustvold suggests there is some interplay of the parquet schema adapter logic that was silently masking a bug in arrow-rs. Now that arrow-rs has stricter error checking, the query is erroring #2530 (comment)

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdatafusionChanges in the datafusion crate

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions