Filter rows directly from pa.RecordBatch

[gabeiglio]

This PR from Apache Arrow was merged to allow to filter with a boolean expression directly on pa.RecordBatch.

I believe pyiceberg is currently using pyarrow version 19.0.0.
Filtering from pa.RecordBatch was introduced in python in version 17.0.0

I have not run integration tests for some reason my docker setup is messed up. I believe this test should check this change:

iceberg-python/tests/integration/test_deletes.py

Line 314 in dfbee4b

def test_read_multiple_batches_in_task_with_position_deletes(spark: SparkSession, session_catalog: RestCatalog) -> None:

Closes #1050

[Fokko]

Thanks for fixing this @gabeiglio

In addition, I think we also need to bump the minimal version of Arrow here:

iceberg-python/pyproject.toml

Line 64 in dfbee4b

pyarrow = { version = ">=14.0.0,<20.0.0", optional = true }

[kevinjqliu]

thanks for following up on that comment 😄

if we're bumping minimum pyarrow version to 17, we might want to address this comment as well
https://github.com/apache/iceberg-python/pull/1621/files#diff-8d5e63f2a87ead8cebe2fd8ac5dcf2198d229f01e16bb9e06e21f7277c328abdR1335-R1338

[gabeiglio]

@kevinjqliu IIUC removing the schema casting will allow pyarrow scanner to infer by itself if it needs or not large types? So it is basically a matter of changing the assertions in tests to the types of the result of the scan?

[kevinjqliu]

I believe so. We can also do this in a follow up PR! I just saw that comment during code review

[kevinjqliu]

Looks like theres an issue in CI tests

[gabeiglio]

Yes, I think it would be better to split these changes in separate PRs since there are a lot of changes to be made to tests specially. (If thats okay ill open the other PR for schema casting @kevinjqliu @Fokko)

[Fokko]

One minor comment, looks great, and so much cleaner :)

[Fokko]

nit, I think we can drop the batch:

Suggested change

	for batch in batches:
	for current_batch in batches:

[kevinjqliu]

looks like CIs broken on poetry

[kevinjqliu]

is this logically equivalent? feels like there was a reason to write it the other way.

cc @sungwy do you have context on this?

[gabeiglio]

Oh I wasn't planning on pushing this change 🤦. I'll revert it in the next commit if we want

[kevinjqliu]

@gabeiglio when you get a chance could you rebase and also fix the change with next_index

[kevinjqliu]

there a conflict with poetry.lock, heres something i found that worked

git fetch origin main
git merge origin/main -X theirs
poetry lock

This will ignore the conflict and regenerate the poetry lock file

[kevinjqliu]

LGTM! do you mind double checking if we have sufficient test coverage for the changes in this PR. Since its part of the read path, i want to be extra careful here

[gabeiglio]

Sounds good, Let me check!

[gabeiglio]

• 32 tests in tests_reads.py test filtering using case sensitive, and insensitive, multiple fields, one field, empty fields, nulls, etc.
• 4 tests test filtering partitioned tables in test_partitions.py that also tests deletes files (this makes sure that filter is not being skipped when applying deletes)
• 4 tests in test_pyarrow that also tests deletes + filtering

There could me more, but Im thinking this is enough, wdyt @kevinjqliu?

[kevinjqliu]

LGTM! Thanks @gabeiglio for the contribution and double checking the tests. The logic here is fairly straight forward after reverting the current_index change.
Thanks @Fokko for the review!

[gabeiglio]

Thanks for the help! @kevinjqliu