Support Appends with TimeTransform Partitions #703

sungwy · 2024-05-06T16:22:32Z

Support writes with TimeTransforms.

TimeTransforms are can be supported natively within pyarrow as pyarrow compute functions, without requiring conversions back and forth between Arrow and Python data types.

…he#702)

Bumps [mkdocs-section-index](https://github.com/oprypin/mkdocs-section-index) from 0.3.8 to 0.3.9. - [Release notes](https://github.com/oprypin/mkdocs-section-index/releases) - [Commits](oprypin/mkdocs-section-index@v0.3.8...v0.3.9) --- updated-dependencies: - dependency-name: mkdocs-section-index dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [cython](https://github.com/cython/cython) from 3.0.8 to 3.0.10. - [Release notes](https://github.com/cython/cython/releases) - [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst) - [Commits](cython/cython@3.0.8...3.0.10) --- updated-dependencies: - dependency-name: cython dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

HonahX

Looks Great! Thanks @syun64

Makefile

pyiceberg/transforms.py

HonahX · 2024-05-07T05:28:30Z

pyiceberg/transforms.py

+    def pyarrow_transform(self, source: IcebergType) -> "Callable[[pa.Array], pa.Array]": ...
+
+    @property
+    def supports_pyarrow_transform(self) -> bool:


Instead of adding a public method, how about we maintaining a private list of transform that supports pyarrow transform for the check at the beginning of append. For transforms that does not yet support pyarrow transform, we could throw NotImplementedError in pyarrow_transform

WDYT?

I was following the existing convention for can_transform and preserves_order that are specified as class properties. I'm in favor of keeping it consistent with these

Thanks for sharing the context. My initial concern was that unlike can_transform and preserve_oder which behave differently across transforms, in the end the support_pyarrow_transform would return True for all the transforms, making it a little bit redundant. But on second thought, since we will likely not support pyarrow transform for other transforms in 0.7.0 release, this support_pyarrow_transform can be a useful property in the upcoming release.

We could deprecate this property after we support pyarrow for all transforms later. WDYT?

Yeah I'm on the same page! To add to that - I'm not yet sure if we will be able to support the other Transform classes in support_pyarrow_transform. The key idea here is that we want to use native pyarrow compute functions to apply the equivalent transform without converting the values back and forth between Arrow and Python data types. I'm not yet sure if we'll be able to do the same for BucketTransform for instance, with the existing range of pyarrow compute functions.

Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.66.2 to 4.66.3. - [Release notes](https://github.com/tqdm/tqdm/releases) - [Commits](tqdm/tqdm@v4.66.2...v4.66.3) --- updated-dependencies: - dependency-name: tqdm dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.0.1...3.0.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

benihildebrand · 2024-05-08T20:20:40Z

pyiceberg/transforms.py

+        elif isinstance(source, TimestamptzType):
+            epoch = datetime.EPOCH_TIMESTAMPTZ
+        else:
+            raise ValueError(f"Cannot apply month transform for type: {source}")


"Cannot apply month transform for type" should be referring to 'hour', not 'month'.

Thank you for the check - appreciate it!

Bumps [tenacity](https://github.com/jd/tenacity) from 8.2.3 to 8.3.0. - [Release notes](https://github.com/jd/tenacity/releases) - [Commits](jd/tenacity@8.2.3...8.3.0) --- updated-dependencies: - dependency-name: tenacity dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) from 0.25.0 to 0.25.1. - [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases) - [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md) - [Commits](mkdocstrings/mkdocstrings@0.25.0...0.25.1) --- updated-dependencies: - dependency-name: mkdocstrings dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.0 to 7.5.1. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.5.0...7.5.1) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.29 to 2.0.30. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [flask-cors](https://github.com/corydolphin/flask-cors) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/corydolphin/flask-cors/releases) - [Changelog](https://github.com/corydolphin/flask-cors/blob/main/CHANGELOG.md) - [Commits](corydolphin/flask-cors@4.0.0...4.0.1) --- updated-dependencies: - dependency-name: flask-cors dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.5.2 to 7.5.3. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.5.2...7.5.3) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pydantic](https://github.com/pydantic/pydantic) from 2.7.1 to 2.7.2. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](pydantic/pydantic@v2.7.1...v2.7.2) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [requests](https://github.com/psf/requests) from 2.32.2 to 2.32.3. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.2...v2.32.3) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [getdaft](https://github.com/Eventual-Inc/Daft) from 0.2.24 to 0.2.25. - [Release notes](https://github.com/Eventual-Inc/Daft/releases) - [Commits](Eventual-Inc/Daft@v0.2.24...v0.2.25) --- updated-dependencies: - dependency-name: getdaft dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

First step towards apache#579

Bumps [moto](https://github.com/getmoto/moto) from 5.0.8 to 5.0.9. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.0.8...5.0.9) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fokko

This looks good @syun64 Thanks for working on this, could you rebase to resolve the merge conflicts?

* Update fsspec.py to respect s3.signer.uri property * Add S3_SIGNER_URI constant, add docs --------- Co-authored-by: Fokko Driesprong <[email protected]>

…4/iceberg-python into transform-partition-writes

Fokko · 2024-05-31T19:04:44Z

@syun64 It looks like something is off. I see a lot of unrelated changes 🤔

sungwy · 2024-05-31T19:12:22Z

@Fokko hmmm yeah I've seen this happen some times and I don't quite understand it - when you do a diff between the two branches, it shows the right changes: main...syun64:iceberg-python:transform-partition-writes

sungwy · 2024-05-31T19:13:23Z

I'll close this PR and reopen it and see if that helps

sungwy added 8 commits May 4, 2024 02:20

checkpoint

0cad231

checkpoint2

96e5533

todo: sort with pyarrow_transform vals

ddfa9ac

checkpoint

1a5327a

checkpoint

e067a28

fix

069f3bd

tests

615d5e3

more tests

c0a0f32

sungwy requested a review from Fokko May 6, 2024 17:06

felixscherz and others added 3 commits May 6, 2024 10:32

Remove trailing slash from table location when creating a table (apac…

d872245

…he#702)

HonahX reviewed May 7, 2024

View reviewed changes

dependabot bot and others added 5 commits May 6, 2024 22:39

adopt review feedback

6a39eda

Make add_files to support snapshot_properties argument (apache#695)

990ce80

HonahX approved these changes May 7, 2024

View reviewed changes

Add support for categorical type (apache#693)

0508667

benihildebrand reviewed May 8, 2024

View reviewed changes

dependabot bot and others added 9 commits May 8, 2024 15:10

comment

d14e137

Build: Bump mkdocs-material from 9.5.20 to 9.5.21 (apache#719)

4de207d

Build: Bump getdaft from 0.2.23 to 0.2.24 (apache#721)

d02d7a1

Test, write subset of schema (apache#704)

aa361d1

dependabot bot and others added 10 commits May 29, 2024 06:54

Remove record_fields from the Record class (apache#580)

d3ad61c

First step towards apache#579

Unify to double quotes using Ruff (apache#781)

cf3bf8a

Support CreateTableTransaction for SqlCatalog (apache#684)

0339e7f

Support CreateTableTransaction for HiveCatalog (apache#683)

84a2c04

Support viewfs scheme along side with hdfs (apache#777)

8d79664

Fokko approved these changes May 31, 2024

View reviewed changes

c-thiel and others added 13 commits May 31, 2024 17:19

Update fsspec.pyto respect s3.signer.uri property (apache#741)

20f6afd

* Update fsspec.py to respect s3.signer.uri property * Add S3_SIGNER_URI constant, add docs --------- Co-authored-by: Fokko Driesprong <[email protected]>

checkpoint

5dd846d

checkpoint2

6357193

todo: sort with pyarrow_transform vals

c30a57c

checkpoint

541655f

checkpoint

afe83b1

fix

00ca5f0

tests

511e988

more tests

3b784ab

adopt review feedback

3711b1b

comment

f16d778

Merge branch 'transform-partition-writes' of https://github.com/syun6…

80d5064

…4/iceberg-python into transform-partition-writes

rebase

9f0a92b

sungwy closed this May 31, 2024

sungwy mentioned this pull request May 31, 2024

Support Appends with TimeTransform Partitions #784

Merged

sungwy deleted the transform-partition-writes branch May 31, 2024 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Appends with TimeTransform Partitions #703

Support Appends with TimeTransform Partitions #703

sungwy commented May 6, 2024

HonahX left a comment

HonahX May 7, 2024 •

edited

Loading

sungwy May 7, 2024

HonahX May 7, 2024 •

edited

Loading

sungwy May 7, 2024

benihildebrand May 8, 2024

sungwy May 8, 2024

Fokko left a comment

Fokko commented May 31, 2024

sungwy commented May 31, 2024

sungwy commented May 31, 2024

Support Appends with TimeTransform Partitions #703

Support Appends with TimeTransform Partitions #703

Conversation

sungwy commented May 6, 2024

HonahX left a comment

Choose a reason for hiding this comment

HonahX May 7, 2024 • edited Loading

Choose a reason for hiding this comment

sungwy May 7, 2024

Choose a reason for hiding this comment

HonahX May 7, 2024 • edited Loading

Choose a reason for hiding this comment

sungwy May 7, 2024

Choose a reason for hiding this comment

benihildebrand May 8, 2024

Choose a reason for hiding this comment

sungwy May 8, 2024

Choose a reason for hiding this comment

Fokko left a comment

Choose a reason for hiding this comment

Fokko commented May 31, 2024

sungwy commented May 31, 2024

sungwy commented May 31, 2024

HonahX May 7, 2024 •

edited

Loading

HonahX May 7, 2024 •

edited

Loading