Tracking issues of iceberg-rust v0.2

[Xuanwo]

• proposal: proposal: Roadmap of iceberg-rust v0.2 #9

Tasks

• specs: Add Schema
  • Visitor pattern
    • Binding visitor to bind an UnboundPredicate
  • Expressions:
    • Term trait
    • Unbound trait
    • Expression trait
    • Bound trait
• specs: Add TableMetadata
  • Support for both V1 and V2.
  • When read, set V2 fields (schemas, current-schema-id, partition-specs, default-spec-id, last-partition-id, sort-orders, default-sort-order-id, refs) so we can just assume that it is a V2 table.
• specs: Add Manifest
  • ManifestList and Manifest
• specs: Transforms
  • Iceberg has a set of transforms that need to be implemented
  • For the bucket transforms, the values need to be hashed.
• io: How to handle IO? Support for GCS, S3, ADLS, etc. Do we want something similar to the FileIO in (Py)Iceberg.
• arrow: Reading data from iceberg to arrow record batches
• arrow: Writing unpartitioned data into iceberg from arrow record batches
• arrow: Writing partitioned data into iceberg from arrow record batches
• api: Table API design
• api: Catalog API design #23
• rest catalog

Future tasks

• PartitionEvaluator: Able to prune files on the ManifestList and Manifest level by taking the partition predicate into account.
• MetricsEvaluator: Able to prune DataFiles based on the column metrics on the Manifest

[Xuanwo]

I'm not sure whether those tasks splitted correctly, please feel free to leave comments.

cc @liurenjie1024, @JanKaul

[Xuanwo]

For io: How to handle IO? Support for GCS, S3, ADLS, etc.:

we can use opendal: An Apache project maintained by me and has already supported all those mentioned services. We can expose a similiar trait as FileIO and keep opendal underhood as an implementation.

I'm willing to take those part of tasks.

[JanKaul]

There is also the question whether to support transactions for writing arrow in v0.2. Which leads to further questions regarding filesystem tables and a catalog interface(required for metastore transactions).

Should we support filesystem tables? (I don't really need them)

How should a catalog interface(trait) look like?

[Xuanwo]

Should we support filesystem tables? (I don't really need them)

I'm guessing we will support writing with rest catalog first (@liurenjie1024 may need this feature). And catalog trait could be added in later versions.

[Fokko]

Should we support filesystem tables? (I don't really need them)

In PyIceberg we don't support them, maybe in the future, but since Iceberg is object store native, we try to stay away from FileSystems. For read support, you could always load them using a StaticTable.

How should a catalog interface(trait) look like?

This is what we went with in PyIceberg.

[JanKaul]

I think the catalog interface fundamentally influences the design of the transactions and in my opinion it is going to save time defining the interface early.

[liurenjie1024]

I think the catalog interface fundamentally influences the design of the transactions and in my opinion it is going to save time defining the interface early.

Agree.

[liurenjie1024]

Should we support filesystem tables? (I don't really need them)

I'm guessing we will support writing with rest catalog first (@liurenjie1024 may need this feature). And catalog trait could be added in later versions.

I need it, but I want to implement it after discussion.

[liurenjie1024]

cc @Xuanwo Is it possible to make the todo items in this tracking issue as issues?

[Xuanwo]

cc @Xuanwo Is it possible to make the todo items in this tracking issue as issues?

Seems need higher permission, cc @Fokko

[liurenjie1024]

Close this issue as 0.2 has been release. Feel free to reopen when necessary.