Skip to content

libregrtest: Reduce number of imports at startup #111881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vstinner opened this issue Nov 9, 2023 · 3 comments
Closed

libregrtest: Reduce number of imports at startup #111881

vstinner opened this issue Nov 9, 2023 · 3 comments
Labels
tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

vstinner commented Nov 9, 2023

Currently, when a test is run by ./python -m test (...), the test starts with 170 modules already imported. The problem is that imports can have side effects and I would prefer to start tests in a "minimal environment".

Running a test with ./python -m test (...) only imports 164 modules.

Example by adding Lib/test/test_x.py:

import sys
print(len(sys.modules))

Output:

vstinner@mona$ ./python -m test test_x
170

vstinner@mona$ ./python -m test -j1 test_x
164

I propose to try to make more imports lazy: only import the module the first time that it's needed, tests are rarely "performance sensitive". It may improve the "startup time".

Linked PRs

@vstinner vstinner added the tests Tests in the Lib/test dir label Nov 9, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
In most cases, doctest is not needed. So don't always import it at
startup.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
The linecache module is only used by StackSummary and
FrameSummary.line property. Import the linecache module lazily to
speedup Python imports and reduce the number of imports when the
traceback module is imported.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
The linecache module is only used by StackSummary and
FrameSummary.line property. Import the linecache module lazily to
speedup Python imports and reduce the number of imports when the
traceback module is imported.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
The linecache module is only used by StackSummary and
FrameSummary.line property. Import the linecache module lazily to
speedup Python imports and reduce the number of imports when the
traceback module is imported.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
Use lazy imports for argparse, difflib and fnmatch imports in
unittest to speedup Python startup time when running tests, and the
number of imported modules when running tests.
@vstinner vstinner changed the title libregrtest: Reduce number of imports libregrtest: Reduce number of imports at startup Nov 9, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
Use lazy imports for argparse, difflib and fnmatch imports in
unittest to speedup Python startup time when running tests, and the
number of imported modules when running tests.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
Use lazy imports for argparse, difflib and fnmatch imports in
unittest to speedup Python startup time when running tests, and the
number of imported modules when running tests.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
Use lazy import for difflib and linecache imports in doctest to
speedup Python startup and reduce the number of imports when running
tests.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
The random module now imports the _sha2 module laziliy in
Random.seed() method for str, bytes and bytearray seeds. Lazy import
makes Python startup faster and reduce the number of imported modules
at startup.
vstinner added a commit that referenced this issue Nov 9, 2023
* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.

(cherry picked from commit 0372e3b)
vstinner added a commit to vstinner/cpython that referenced this issue Nov 9, 2023
The random module now imports the _sha2 module lazily in the
Random.seed() method for str, bytes and bytearray seeds. It also
imports lazily the warnings module in the _randbelow() method for
classes without getrandbits(). Lazy import makes Python startup
faster and reduces the number of imported modules at startup.
vstinner added a commit that referenced this issue Nov 9, 2023
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Nov 9, 2023
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
(cherry picked from commit 6f09f69)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Nov 9, 2023
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
(cherry picked from commit 6f09f69)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Nov 9, 2023
gh-111881: Use lazy import in test.support (#111885)

* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.

(cherry picked from commit 0372e3b)
vstinner added a commit that referenced this issue Nov 9, 2023
…111894)

gh-111881: Import doctest lazily in libregrtest (GH-111884)

In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
(cherry picked from commit 6f09f69)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Nov 9, 2023
…111893)

gh-111881: Import doctest lazily in libregrtest (GH-111884)

In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
(cherry picked from commit 6f09f69)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Nov 9, 2023
…#111902)

[3.12] gh-111881: Use lazy import in test.support (#111885) (#111890)

gh-111881: Use lazy import in test.support (#111885)

* Import lazily getpass in test.support

Backport to 3.11: test.support.os_helper is unchanged.

(cherry picked from commit 0372e3b)
(cherry picked from commit e983ca8)
vstinner added a commit that referenced this issue Nov 9, 2023
The random module now imports the _sha2 module lazily in the
Random.seed() method for str, bytes and bytearray seeds. It also
imports lazily the warnings module in the _randbelow() method for
classes without getrandbits(). Lazy import makes Python startup
faster and reduces the number of imported modules at startup.
@arneja-arnav
Copy link

May I work on this issue? would love to take a crack at it.

@vstinner
Copy link
Member Author

May I work on this issue? would love to take a crack at it.

Yes, you can review proposed PRs and help to get them merged.

vstinner added a commit to vstinner/cpython that referenced this issue Nov 16, 2023
It allows running the test suite when the zlib extension is missing.
vstinner added a commit that referenced this issue Nov 16, 2023
It allows running the test suite when the zlib extension is missing.
@vstinner
Copy link
Member Author

I lost track of this change, so I just close it.

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
The random module now imports the _sha2 module lazily in the
Random.seed() method for str, bytes and bytearray seeds. It also
imports lazily the warnings module in the _randbelow() method for
classes without getrandbits(). Lazy import makes Python startup
faster and reduces the number of imported modules at startup.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
…on#112172)

It allows running the test suite when the zlib extension is missing.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
* Import lazily getpass in test.support
* Only import ctypes on Windows in test.support.os_helper.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
The random module now imports the _sha2 module lazily in the
Random.seed() method for str, bytes and bytearray seeds. It also
imports lazily the warnings module in the _randbelow() method for
classes without getrandbits(). Lazy import makes Python startup
faster and reduces the number of imported modules at startup.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
…on#112172)

It allows running the test suite when the zlib extension is missing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Tests in the Lib/test dir
Projects
None yet
Development

No branches or pull requests

2 participants