[WIP] Make incremental not propagate staleness if the public interface of a module is unchanged #2002
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michael0x2a
changed the title
Michael0x2a
force-pushed
the
incremental-interface
branch
from
63dc1b2 to
331db38
Compare
This commit modifies the incremental tests so that changes that don't change the "public interface" of the file will not trigger a recheck in any dependees.
"modinterface.py" is a new stage of the parsing process that ideally happens relatively early, right after each file is parsed. It performs a lightweight, best-effort-basis crawl of the parse tree to try and obtain approximate type signatures of a given module to determine what its "public interface" is. If it cannot deduce the type signature of a given symbol, it resorts to using a serialized form of the entire expression so that absolutely any changes to how that symbol is derived is reflected.
Previously, if the file timestamp or size differed from the data stored in the meta files, we treated it as immediately stale. In preparation for the interface changes, this commit extracts that logic into a separate flag.
This commit modifies the file parsing logic to also compute and store the public interface for a module. The public interface is not yet used anywhere.
This commit finally enables the build process to throttle back incremental mode when the public interface of a method is not changed.
Previously, we always extracted the public interface, even though there's really no need to do so when not running incremental mode.
Michael0x2a
force-pushed
the
incremental-interface
branch
from
331db38 to
71dd658
Compare
|
#2014 is a better and more accurate version of this pull request. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request modifies incremental mode so that if a module is modified in a way such that the types for any of its publicly accessible symbols are unchanged, we recheck the file but don't queue up the importers to be rechecked. For example, if we change the contents of a certain function but not its type signature, we would ideally like to have to re-check only that file and not all the dependees.
Or, more precisely, if module A imports module B and module B's public interface is unchanged, then we consider module B to be internally stale but not externally stale and consider module A to still be fresh.
It does so by adding a new visitor named
InterfaceExtractorwithinmodinterface.pythat makes a best-effort attempt at extracting the "public interface" of a module once its been parsed. Since this extraction happens at an early stage in the pipeline (inbuild.load_graph, which callsparse_file) it's not necessarily always possible to extract the actual types, so the extractor falls back to using a serialized form of the underlying symbol node in those cases.The public interface of a module consists of al imports, all top-level variables/functions/classes, and all methods/class attributes/nested class definitions.
This is a work-in-progress. TODO: