Skip to content

Port PyPy's new unicode name db format to optimize binary CPython size #96954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
isidentical opened this issue Sep 20, 2022 · 5 comments
Closed

Port PyPy's new unicode name db format to optimize binary CPython size #96954

isidentical opened this issue Sep 20, 2022 · 5 comments
Labels
type-feature A feature request or enhancement

Comments

@isidentical
Copy link
Member

isidentical commented Sep 20, 2022 •
edited by bedevere-app bot
Loading

Feature or enhancement

PyPy has a new algorithm that might help us reduce the unicode name db size of our binaries (which would be helpful on our way forward to better WASM compatibility [through reduced download sizes]). For more details, you can see the new implementation in PyPy side by @cfbolz.

CC: @ambv @cfbolz

We'll probably start with missing tests, and then build out a prototype to see how feasible it is and will share numbers on how much it would help before going forward with the implementation.

Linked PRs
@isidentical isidentical added the type-feature A feature request or enhancement label Sep 20, 2022
isidentical added a commit to isidentical/cpython that referenced this issue Sep 20, 2022
@isidentical @cfbolz
pythongh-96954: Add tests for unicodedata.name/lookup
911d341 
They were undertested, and since python#96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
isidentical added a commit to isidentical/cpython that referenced this issue Sep 20, 2022
@isidentical @cfbolz
pythongh-96954: Add tests for unicodedata.name/lookup
b1c54f3 
They were undertested, and since python#96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
@isidentical isidentical mentioned this issue Sep 20, 2022
Merged
ambv pushed a commit that referenced this issue Sep 21, 2022
@isidentical @cfbolz
gh-96954: Add tests for unicodedata.name/lookup (#96955)
5a32eec 
They were undertested, and since #96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
ambv pushed a commit to ambv/cpython that referenced this issue Sep 21, 2022
@isidentical @ambv
[3.11] pythongh-96954: Add tests for unicodedata.name/lookup (pythonG…
5803cdb 
…H-96955)

They were undertested, and since pythonGH-96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
(cherry picked from commit 5a32eec)

Co-authored-by: Batuhan Taskaya <[email protected]>
@ambv ambv mentioned this issue Sep 21, 2022
Merged
ambv pushed a commit to ambv/cpython that referenced this issue Sep 21, 2022
@isidentical @ambv
[3.10] pythongh-96954: Add tests for unicodedata.name/lookup (pythonG…
d416137 
…H-96955)

They were undertested, and since pythonGH-96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
(cherry picked from commit 5a32eec)

Co-authored-by: Batuhan Taskaya <[email protected]>
@ambv ambv mentioned this issue Sep 21, 2022
Merged
ambv pushed a commit that referenced this issue Sep 21, 2022
@isidentical @ambv
[3.11] gh-96954: Add tests for unicodedata.name/lookup (GH-96955)
c428ff2 
They were undertested, and since GH-96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
(cherry picked from commit 5a32eec)

Co-authored-by: Batuhan Taskaya <[email protected]>
ambv added a commit that referenced this issue Sep 21, 2022
@ambv @isidentical
[3.10] gh-96954: Add tests for unicodedata.name/lookup (GH-96955) (GH…
c112489 
…-96992)

They were undertested, and since GH-96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
(cherry picked from commit 5a32eec)

Co-authored-by: Batuhan Taskaya <[email protected]>
ambv added a commit that referenced this issue Sep 21, 2022
@ambv @isidentical
[3.11] gh-96954: Add tests for unicodedata.name/lookup (GH-96955) (GH…
d35b9d0 
…-96991)

They were undertested, and since GH-96954 might involve a
rewrite of this part of the code we want to ensure that
there won't be any behavioral change.

Co-authored-by: Carl Friedrich Bolz-Tereick <[email protected]>
(cherry picked from commit 5a32eec)

Co-authored-by: Batuhan Taskaya <[email protected]>
@cfbolz cfbolz mentioned this issue Oct 5, 2022
Merged
ambv added a commit that referenced this issue Nov 4, 2023
@cfbolz @ambv
@eendebakpt @sweeneyde
gh-96954: use a directed acyclic word graph for storing the unicodeda…
9573d14 
…ta codepoint names (#97906)

Co-authored-by: Łukasz Langa <[email protected]>
Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: Dennis Sweeney <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Nov 6, 2023
Merged
ambv pushed a commit that referenced this issue Nov 6, 2023
@sweeneyde
gh-96954: Use skip_if_missing in test_makeunicodedata (GH-111764)
5e5762a 
skip_if_missing in test_makeunicodedata
hugovk pushed a commit to hugovk/cpython that referenced this issue Nov 8, 2023
@sweeneyde @hugovk
pythongh-96954: Use skip_if_missing in test_makeunicodedata (pythonGH…
6c73b36 
…-111764)

skip_if_missing in test_makeunicodedata
@hroncok
Copy link
Contributor

hroncok commented Nov 15, 2023

I've noticed make regen-unicodedata fails in out-of-tree build:

python3.13 Tools/unicode/makeunicodedata.py
python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
make: *** [Makefile:1498: regen-unicodedata] Error 2

I think I know how to fix this and will submit a PR once I test my assumption.

@bedevere-app bedevere-app bot mentioned this issue Nov 15, 2023
Merged
@hroncok
Copy link
Contributor

hroncok commented Nov 15, 2023

#112118

@hroncok
Copy link
Contributor

hroncok commented Nov 15, 2023

I've also noticed it is now impossible to call make regen-all when offline.

Is it necessary to include regen-unicodedata in regen-all?

vstinner pushed a commit that referenced this issue Nov 15, 2023
@hroncok
gh-96954: Fix make regen-unicodedata in out-of-tree builds (#112118)
ec05507 
This avoids:

    python3.13 Tools/unicode/makeunicodedata.py
    python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
    make: *** [Makefile:1498: regen-unicodedata] Error 2

Re-run `make regen-unicodedata` to update the script path in generated files.
@vstinner
Copy link
Member

I've also noticed it is now impossible to call make regen-all when offline.

regen-configure pulls a docker container image, it's not the only "regen" command using Internet. But regen-configure is excluded from make regen-all.

Maybe we can exclude regen-unicodedata from regen-all since it's rare to have to update these data, whereas it's common to run regen-all to update more frequently changed files.

vstinner added a commit to vstinner/cpython that referenced this issue Nov 15, 2023
@vstinner
pythongh-96954: Don't run regen-unicodedata in regen-all
466d256 
The "make regen-unicodedata" should now be run manually. By the
default, it requires an Internet connection, which is not always the
case. Some Linux distributions build Linux packages in isolated
environment (without network).
@bedevere-app bedevere-app bot mentioned this issue Nov 15, 2023
Merged
vstinner added a commit that referenced this issue Nov 15, 2023
@vstinner
gh-96954: Don't run regen-unicodedata in regen-all (#112120)
e5dfcc2 
The "make regen-unicodedata" should now be run manually. By the
default, it requires an Internet connection, which is not always the
case. Some Linux distributions build Linux packages in isolated
environment (without network).
@vstinner
Copy link
Member

The initial feature request is now implemented and known issues have been fixed. I close the issue.

Thanks @cfbolz for this cool optimization!

@terryjreedy terryjreedy mentioned this issue Dec 17, 2023
Open
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@cfbolz @ambv
@eendebakpt @sweeneyde @aisk
pythongh-96954: use a directed acyclic word graph for storing the uni…
973c537 
…codedata codepoint names (python#97906)

Co-authored-by: Łukasz Langa <[email protected]>
Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: Dennis Sweeney <[email protected]>
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@sweeneyde @aisk
pythongh-96954: Use skip_if_missing in test_makeunicodedata (pythonGH…
71cacdf 
…-111764)

skip_if_missing in test_makeunicodedata
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@hroncok @aisk
pythongh-96954: Fix make regen-unicodedata in out-of-tree builds (p…
21e8501 
…ython#112118)

This avoids:

    python3.13 Tools/unicode/makeunicodedata.py
    python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
    make: *** [Makefile:1498: regen-unicodedata] Error 2

Re-run `make regen-unicodedata` to update the script path in generated files.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@vstinner @aisk
pythongh-96954: Don't run regen-unicodedata in regen-all (python#112120)
7da5e0e 
The "make regen-unicodedata" should now be run manually. By the
default, it requires an Internet connection, which is not always the
case. Some Linux distributions build Linux packages in isolated
environment (without network).
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@cfbolz @ambv
@eendebakpt @sweeneyde @Glyphack
pythongh-96954: use a directed acyclic word graph for storing the uni…
c234d2f 
…codedata codepoint names (python#97906)

Co-authored-by: Łukasz Langa <[email protected]>
Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: Dennis Sweeney <[email protected]>
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@sweeneyde @Glyphack
pythongh-96954: Use skip_if_missing in test_makeunicodedata (pythonGH…
84be68a 
…-111764)

skip_if_missing in test_makeunicodedata
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@hroncok @Glyphack
pythongh-96954: Fix make regen-unicodedata in out-of-tree builds (p…
e158f79 
…ython#112118)

This avoids:

    python3.13 Tools/unicode/makeunicodedata.py
    python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
    make: *** [Makefile:1498: regen-unicodedata] Error 2

Re-run `make regen-unicodedata` to update the script path in generated files.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-96954: Don't run regen-unicodedata in regen-all (python#112120)
30aa724 
The "make regen-unicodedata" should now be run manually. By the
default, it requires an Internet connection, which is not always the
case. Some Linux distributions build Linux packages in isolated
environment (without network).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vstinner @hroncok @isidentical