Add files to add existing Parquet files to a table

[ZENOTME]

In #345, we support writing new data files and appending them to the table. But we haven't support appending existing data files which need to support reading existing data files and generating corresponding metadata DataFile.

[jonathanc-n]

I would like to try working on this.

[ZENOTME]

I would like to try working on this.

Thanks @jonathanc-n! Feel free to send the PR for this.

[jonathanc-n]

@ZENOTME When appending existing data files, should the system load file metadata by reading the current snapshot’s manifest lists from an existing Iceberg table, or would you prefer to specify a file path from which the system scans and infers metadata? I'm looking to just perform a TableScan based the answer and have it just add the DataFiles with the add_data_file.

[ZENOTME]

@ZENOTME When appending existing data files, should the system load file metadata by reading the current snapshot’s manifest lists from an existing Iceberg table, or would you prefer to specify a file path from which the system scans and infers metadata? I'm looking to just perform a TableScan based the answer and have it just add the DataFiles with the add_data_file.

Hi @jonathanc-n, I think we can refer the implementation of pyiceberg: https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L669C9-L669C18.

should the system load file metadata by reading the current snapshot’s manifest lists from an existing Iceberg table, or would you prefer to specify a file path from which the system scans and infers metadata?

I think the user will add file using transaction API so we can know which table it will be append and related metadata.

[mkarbo]

@liurenjie1024 @jonathanc-n should this be closed now that #960 is in?

[jonathanc-n]

Don't believe so, there a bunch of follow up prs that should be done before this is closed