Manual deduction of partitions

[afiodorov]

I want to

a) be able to add files that a partitioned by the filename convention, e.g. s3://bucket/table/year=2025/month=12
b) add files even if they have extra columns without having to migrate the table

This comes from a common pattern of having existing hive tables and the need to migrate them to iceberg.

I propose we can achieve this by doing

pattern = re.compile(r"([^/]+)=([^/]+)")

def deduct_partition(path: str) -> Record:
    return Record(**dict(pattern.findall(path))

table.add_files(['s3://bucket/table/year=2025/month=12/file.parquet'], check_schema=False, partition_deductor=deduct_partition)

[kevinjqliu]

Hi @afiodorov thanks for the PR!

For adding hive partitioned files to Iceberg, there's a specific way we can do so using column projections, https://iceberg.apache.org/spec/#column-projection. We've implemented the read side in #1443. We'd want to implement the write side as well by overridding the partition field in data_file object in the manifest.

I think we need to define an API that does not involve regex. The example above is confusing as a user

pattern = re.compile(r"([^/]+)=([^/]+)")

def deduct_partition(path: str) -> Record:
    return Record(**dict(pattern.findall(path))

table.add_files(['s3://bucket/table/year=2025/month=12/file.parquet'], check_schema=False, partition_deductor=deduct_partition)

IMO the add_files API should not infer the hive-style partition scheme. Perhaps we can create a different API to "migrate" hive-style partition scheme to an Iceberg table. As long as we can create the proper DataFile object with the right partition field set, we can register these to the Iceberg Table.

pyarrow has a helper function pyarrow.dataset.HivePartitioning which can parse the hive-style partition scheme without relying in regex

Please let me know what you think!

[afiodorov]

Hey! The regex was just an example, it's not part of the proposed API though - the partition deduction function is. Thanks for showing me the HivePartioning function though - makes sense to switch to that when possible.

The issue I am having at a work is that our pipelines keep writing hive-partitioned parquet files to s3 and we have 100s of tables and many of the pipelines that I don't maintain. However we need a quick conversion (and upkeep) of those tables to iceberg.

add_files is almost what we need for both initial migration and the subsequent upkeep however it assumes the partition columns are written in the parquet files - which isn't the case. We don't want to rewrite all parquet files nor touch pipelines at all, we just need a quick hassle-free inplace migration of as many hive-partitioned tables to iceberg as possible. Current idea is to just add the add_files step to our scheduler to register new partitions after regular runs.

If you add another API that deals with DataFile directly as opposed to add_files that assumes certain things that'd work for us. Until then I might just have to deploy my fork to solve our problem.

[Fokko]

Thanks for the PR @afiodorov, I think there is room to make this more flexible, however, I left some concerns in the comments.

[Fokko]

At Iceberg, we're pretty concerned at making sure that everything is compatible at write time. Instead, we could also change the _check_pyarrow_schema_compatible to allow for additional columns in the Parquet column.

It is okay to skip optional columns but not required ones.

[Fokko]

While you can add keys to the Record, it is looked up by position, based on the Schema that belongs to it (in this case, the one of the active PartitionSpec.