Skip to content

[infra] nightly pypi build for pyiceberg_core #948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 21, 2025

Conversation

kevinjqliu
Copy link
Contributor

@kevinjqliu kevinjqliu commented Feb 6, 2025

Inspired by pyiceberg#1601 & https://github.com/apache/iceberg-python/blob/main/.github/workflows/nightly-pypi-build.yml

This PR adds nightly build of pyiceberg_core to testpypi.

I have successfully tested the github action on my fork by pushing to my own package,

I've also verified library versions locally

(3.12.8) ➜  python git:(kevinjqliu/nightly-pyiceberg-core) pip install -i https://test.pypi.org/simple/ pyiceberg-core-kevinliu --prefer-binary --force
Looking in indexes: https://test.pypi.org/simple/
Collecting pyiceberg-core-kevinliu
  Using cached https://test-files.pythonhosted.org/packages/4b/c7/771857185d3d96e4ce0b59293540d0e1df9a8a39dbc938e0770328504c25/pyiceberg_core_kevinliu-0.4.0.dev20250218012743-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.metadata (1.7 kB)
Using cached https://test-files.pythonhosted.org/packages/4b/c7/771857185d3d96e4ce0b59293540d0e1df9a8a39dbc938e0770328504c25/pyiceberg_core_kevinliu-0.4.0.dev20250218012743-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (1.3 MB)
Installing collected packages: pyiceberg-core-kevinliu
  Attempting uninstall: pyiceberg-core-kevinliu
    Found existing installation: pyiceberg_core-kevinliu 0.4.0.dev20250218012743
    Uninstalling pyiceberg_core-kevinliu-0.4.0.dev20250218012743:
      Successfully uninstalled pyiceberg_core-kevinliu-0.4.0.dev20250218012743
Successfully installed pyiceberg-core-kevinliu-0.4.0.dev20250218012743

pip show pyiceberg-core-kevinliu

(3.12.8) ➜  python git:(kevinjqliu/nightly-pyiceberg-core) pip show pyiceberg-core-kevinliu
Name: pyiceberg_core-kevinliu
Version: 0.4.0.dev20250218012743
Summary: 
Home-page: https://rust.iceberg.apache.org
Author: 
Author-email: 
License: Apache-2.0
Location: /Users/kevinliu/.pyenv/versions/3.12.8/lib/python3.12/site-packages
Requires: 
Required-by: 

@kevinjqliu kevinjqliu force-pushed the kevinjqliu/nightly-pyiceberg-core branch 2 times, most recently from 9c4dd8d to 5c24fc0 Compare February 18, 2025 01:48
@kevinjqliu kevinjqliu marked this pull request as ready for review February 18, 2025 01:56
@kevinjqliu kevinjqliu force-pushed the kevinjqliu/nightly-pyiceberg-core branch from 72fef36 to c854549 Compare February 18, 2025 17:38
@kevinjqliu
Copy link
Contributor Author

Rebased and addressed @Xuanwo 's concern with building wheels.

I'll set up trusted publishing for pyiceberg_core in testpypi when I get the proper permission :)

@kevinjqliu kevinjqliu requested review from Xuanwo and Fokko February 18, 2025 17:39
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me. Let see how @Fokko and @liurenjie1024 think about this.

Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kevinjqliu for this pr, LGTM!

@liurenjie1024
Copy link
Contributor

Let's wait for a moment for @Fokko to take a look at this as he's more familiar with python.

@kevinjqliu
Copy link
Contributor Author

Added trusted published to testpypi's pyiceberg-core (Thanks sung)

Screenshot 2025-02-20 at 8 33 11 AM

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks @kevinjqliu for working on this, and @liurenjie1024 and @Xuanwo for the review 🚀

@Fokko Fokko merged commit e4b2810 into apache:main Feb 21, 2025
24 checks passed
@kevinjqliu kevinjqliu deleted the kevinjqliu/nightly-pyiceberg-core branch February 21, 2025 15:59
kevinjqliu added a commit that referenced this pull request Feb 21, 2025
Follow up to #948 
I found a bug in the nightly pypi build where the version timestamp
changes within the jobs, causing a single run to push multiple versions.
This PR fixes the issue by setting the version timestamp once for the
entire workflow and then pass it down to individual jobs. This should
ensure a consistency version

Workflow run:
https://github.com/apache/iceberg-rust/actions/runs/13460497360
Testpypi: https://test.pypi.org/project/pyiceberg-core-kevinliu/#history

Tested on my fork: 
Worfklow run:
https://github.com/kevinjqliu/iceberg-rust/actions/runs/13461643730
Testpypi on my fork:
https://test.pypi.org/project/pyiceberg-core-kevinliu/0.4.0.dev20250221170617/#files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants