Restructure sage.*.all for modularization, replace relative by absolute imports #37900

saraedum · 2024-04-29T11:50:38Z

This PR recreates #36676 by @mkoeppe after it was reverted, see #37796.

We restructure the all.py files for modularization.

Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, #35095 defines distribution packages (pip-installable packages) sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}.

When a namespace package such as sage.misc is filled by modules from several distribution packages, we create modules named:

src/sage/misc/all__sagemath_environment.py
src/sage/misc/all__sagemath_objects.py
src/sage/misc/all__sagemath_repl.py

Import statements are moved from src/sage/misc/all.py to these files as appropriate, and src/sage/misc/all.py imports * from there.

Also some imports are replaced by lazy imports.

The new files provide the top level namespaces for the modularized distribution packages, thus enabling modularized testing.

This design was introduced in #29865 (merged in Jan 2022, early in the Sage 9.6 development cycle).

Moreover, applied a one-line command to replace relative by absolute imports, thus complementing #36666, which does not touch .all* files.

The changes to other files in sage.modules etc. come from the PRs #36597, #36572, #36588, #36589 merged here and do not need review.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on Replace relative imports by absolute ones in modules #36597 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in categories #36572 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in rings #36588 (merged here to resolve merge conflict)
Depends on Replace relative imports by absolute ones in a few packages #36589 (merged here to resolve merge conflict)
Depends on Autogenerate flint headers #36449 (merged here to resolve merge conflict)
Depends on add simple function to count irreducible polynomials over 𝔽q #36390 (merged here to resolve merge conflict)
Depends on Relative to absolute imports, del lazy_import in .all files #38030 (split out from here)
Depends on all*.py files: Use 'del lazy_import', 'del install_doc' #38086 (split out from here)
Depends on src/sage/algebras/steenrod/all.py: Use lazy_import, remove deprecated global import #38095 (split out from here)
Depends on sage.combinat, sage.games: Use more lazy_import in all.py files #38096 (split out from here)

(cd src && for d in $(find sage -name __pycache__ -prune -o -type d -print); do sed -i.bak "/from *[.].*import/s/from /from $(echo $d | sed 's,/,.,g')/;s/[.] import / import /;" $d/all*.py; done)

…zy imports

…th#36566

…te-known-test-failures

…o if ! grep -q 'del lazy_import' $a; then echo 'del lazy_import' >> $a; fi; done

…l*.py

…*/all*.py'

…ctree

…o if ! grep -q 'del install_doc' $a; then echo 'del install_doc' >> $a; fi; done

The chain of immediate ancestors of this commit was created by cherry-picking the relevant commits that comprise sagemath#36676. Replaying all these commits (and the necessary conflict resolution) still leaves a non-empty diff when comparing 10.4.beta2 to the previous sagemath#36676 and 10.4.beta3 to that cherry-picked branch. This commit makes sure that the diff is trivial. The changes introduced here were likely introduced in conflict resolution when merging develop into sagemath#36676. I did not replay these conflict resolutions, so I do this here manually.

saraedum · 2024-04-29T11:56:12Z

Last count of votes from #36676:

👍 @mkoeppe @kwankyu @culler @jhpalmieri @kiwifb @NathanDunfield @GMS103 @roed314
👎 @tobiasdiez @dimpase @tornaria @mantepse @orlitzky

(Technically, I am the "author" of this PR but I just recreated it. I did not vote on this PR.)

mkoeppe · 2024-05-02T01:10:14Z

separate sagemath-foobar submodules of sagelib have very little value by themselves

@dimpase The deliberate design (https://groups.google.com/g/sage-devel/c/kiB32zP3xD4) is that one combines several distributions at runtime, to avoid a combinatorial explosion. I explained this clearly again a week ago in the sage-devel thread https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/SYQWB27hAAAJ -- after you declared the distributions "useless" the first time (https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/_iD6dkbdAAAJ). Oh, and in https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/aGsSdQ7GBAAJ you emphasized the importance of making binary wheels (until I pointed out that I'm making them), only to turn around in https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/5x8gjw33AAAJ to say that you "don't see the point of distributing such pieces, as Python wheels, as something useful on their own."
"On their own" - that's a strawman. And nothing is gained by repeating it once more here in this PR.

kwankyu · 2024-05-02T06:50:31Z

In a way, RJF, a UC Berkeley professor of CS, was not incorrect in his scepticism.

Sage with "a huge collection of bugs" had not even existed by now if @williamstein blindly believed his ("rjf") skepticism.

dimpase · 2024-05-02T07:25:11Z

In a way, RJF, a UC Berkeley professor of CS, was not incorrect in his scepticism.

Sage with "a huge collection of bugs" had not even existed by now if @williamstein blindly believed his ("rjf") skepticism.

please recall that Sage hasn't achieved its original mission. In that sense RJF has been 100% correct.

It's a good indication of the state of the project that we can't even agree on a new mission statement :-)

kwankyu · 2024-05-02T10:45:17Z

... we can't even agree on a new mission statement

By the way, in #36827, people spent much time and energy proposing new statements. Their time was only wasted because you didn't even try to come up with some candidates for which people can vote for on sage-devel.

dimpase · 2024-05-02T12:02:41Z

... we can't even agree on a new mission statement

By the way, in #36827, people spent much time and energy proposing new statements. Their time was only wasted because you didn't even try to come up with some candidates for which people can vote for on sage-devel.

I saw no way to reconcile the opinions there.

SageMath version 10.4.beta5, Release Date: 2024-05-02

sagemathgh-37993: `sage.calculus.expr`: Split out from `sage.calculus.all`       Cherry-picked from sagemath#36676/sagemath#37900. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#37993 Reported by: Matthias Köppe Reviewer(s): John H. Palmieri, Matthias Köppe

sagemathgh-38030: Relative to absolute imports, `del lazy_import` in `.all` files       - Split out from sagemath#37900 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38030 Reported by: Matthias Köppe Reviewer(s):

sagemathgh-38086: `all*.py` files: Use 'del lazy_import', 'del install_doc'       - Cherry-picked from sagemath#37900 Includes style fixes from there as well. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38086 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

sagemathgh-38095: `src/sage/algebras/steenrod/all.py`: Use lazy_import, remove deprecated global import       - The change to use `lazy_import` was cherry-picked from sagemath#37900 - The global import was deprecated in sagemath#32647 (2021) ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38095 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

sagemathgh-38086: `all*.py` files: Use 'del lazy_import', 'del install_doc'       - Cherry-picked from sagemath#37900 Includes style fixes from there as well. ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38086 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

sagemathgh-38095: `src/sage/algebras/steenrod/all.py`: Use lazy_import, remove deprecated global import       - The change to use `lazy_import` was cherry-picked from sagemath#37900 - The global import was deprecated in sagemath#32647 (2021) ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38095 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

…m global bindings

sagemathgh-38096: `sage.combinat`, `sage.games`: Use more `lazy_import` in all.py files       This helps with the modularization, but is also desirable for reducing the start-up time of Sage. - Cherry-picked from sagemath#37900 Related: sagemath#35520 ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#38096 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee

Matthias Koeppe and others added 24 commits April 14, 2024 21:56

Set up all__sage_categories.py

e4b376b

sage.*.all*: Update from sagemath#35095

a960393

relative to absolute imports

2a71fe1

(cd src && for d in $(find sage -name __pycache__ -prune -o -type d -print); do sed -i.bak "/from *[.].*import/s/from /from $(echo $d | sed 's,/,.,g')/;s/[.] import / import /;" $d/all*.py; done)

Fixups in .all*

0b7f892

Reverts

6a8f404

Restores

c791f78

src/sage/misc/dev_tools.py: Update doctest output

366035e

src/sage/misc/sageinspect.py: Update doctests for increased use of la…

a96f570

…zy imports

Defer some changes to all__sagemath_{objects,categories} until sagema…

c749f35

…th#36566

sage -fixdoctests --no-test --distribution sagemath_categories --upda…

563a3f2

…te-known-test-failures

Defer some changes to all__sagemath_repl until sagemath#36566

22395ae

src/sage/calculus/all.py: Move 'symbolic_expression' to new module .expr

7de8387

for a in $(git --no-pager grep -l lazy_import src/sage/**/all*.py); d…

e411118

…o if ! grep -q 'del lazy_import' $a; then echo 'del lazy_import' >> $a; fi; done

Manual fixes re 'del lazy_import'

7a07f69

autopep8 --in-place --ignore E402 --max-line-length=88 src/sage/**/al…

dfc6cca

…l*.py

sed -i.bak 's,http:,https:,g' src/sage/**/all*.py

a7fdfc9

Fix errors from 'sage -tox -e pycodestyle -- --ignore E402 src/sage/*…

e630e90

…*/all*.py'

src/doc/en/reference/calculus/index.rst: Add sage.calculus.expr to to…

bcc7137

…ctree

src/sage/calculus/expr.py: Move imports from method to module

80d33ee

for a in $(git --no-pager grep -l install_doc src/sage/**/all*.py); d…

49a508d

…o if ! grep -q 'del install_doc' $a; then echo 'del install_doc' >> $a; fi; done

src/sage/combinat/all.py: Also del install_dict

91ad68f

src/sage/rings/all__sagemath_categories.py: Add import of lazy_import

e9f86a0

Merge remote-tracking branch 'origin/develop' into 36676

f8cda25

saraedum added s: needs review disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ labels Apr 29, 2024

This was referenced Apr 29, 2024

Add # sage_setup: distribution directives to all files #37901

Open

Revert PR with disputed dependencies #37796

Merged

Restructure sage.*.all for modularization, replace relative by absolute imports #36676

Closed

This comment was marked as abuse.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Merge tag '10.4.beta5' into 36676

ac36cbc

SageMath version 10.4.beta5, Release Date: 2024-05-02

github-actions bot added the v: large label May 12, 2024

Merge remote-tracking branch 'upstream/develop' into 36676

45cc4bd

Merge remote-tracking branch 'upstream/develop' into 36676

0a27566

Matthias Koeppe added 4 commits June 1, 2024 16:44

sage.combinat, sage.games: Use more lazy_import in all.py files

20e6cda

src/sage/structure/parent.pyx: In doctest, use SymmetricFunctions fro…

a34bd71

…m global bindings

src/sage/categories/weyl_groups.py: Fix import

327fb5c

Merge branch 'lazy_import_combinat' into 36676

af1ff33

vbraun force-pushed the develop branch from eba5e19 to e5f42fa Compare June 3, 2024 22:15

Matthias Koeppe added 2 commits June 9, 2024 11:01

Merge remote-tracking branch 'upstream/develop' into 36676

fd5ba9a

src/sage/matrix/args.pyx: late-import MatrixSpace in all code paths

8865085

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Uh oh!

saraedum commented Apr 29, 2024 •

edited by mkoeppe

Loading

Uh oh!

saraedum commented Apr 29, 2024

Uh oh!

mkoeppe commented May 2, 2024

Uh oh!

This comment was marked as abuse.

kwankyu commented May 2, 2024 •

edited

Loading

Uh oh!

dimpase commented May 2, 2024

Uh oh!

kwankyu commented May 2, 2024

Uh oh!

dimpase commented May 2, 2024

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Are you sure you want to change the base?

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Uh oh!

Conversation

saraedum commented Apr 29, 2024 • edited by mkoeppe Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

⌛ Dependencies

Uh oh!

saraedum commented Apr 29, 2024

Uh oh!

mkoeppe commented May 2, 2024

Uh oh!

This comment was marked as abuse.

kwankyu commented May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimpase commented May 2, 2024

Uh oh!

kwankyu commented May 2, 2024

Uh oh!

dimpase commented May 2, 2024

Uh oh!

This comment was marked as off-topic.

Uh oh!

saraedum commented Apr 29, 2024 •

edited by mkoeppe

Loading

kwankyu commented May 2, 2024 •

edited

Loading