bpart: Fully switch to partitioned semantics #57253

Keno · 2025-02-04T07:30:35Z

This is the final PR in the binding partitions series (modulo bugs and tweaks), i.e. it closes #54654 and thus closes #40399, which was the original design sketch.

This thus activates the full designed semantics for binding partitions, in particular allowing safe replacement of const bindings. It in particular allows struct redefinitions. This thus closes timholy/Revise.jl#18 and also closes #38584.

The biggest semantic change here is probably that this gets rid of the notion of "resolvedness" of a binding. Previously, a lot of the behavior of our implementation depended on when bindings were "resolved", which could happen at basically an arbitrary point (in the compiler, in REPL completion, in a different thread), making a lot of the semantics around bindings ill- or at least implementation-defined. There are several related issues in the bugtracker, so this closes #14055 closes #44604 closes #46354 closes #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus closes #53958 closes #54733 - however, this is not activated yet for performance reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some hopefully more well-defined semantics. I will describe the semantics below, but before I do I will make two notes:

There are a number of cases where these semantics will behave slightly differently than the old semantics absent some other task going around resolving random bindings.
The new behavior (except for the replacement stuff) was generally permissible under the old semantics if the bindings happened to be resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

Implicit Bindings: Anything implicitly obtained from using Mod, "no binding", plus slightly more exotic corner cases around conflicts
Weakly declared bindings: Declared using global sym and nothing else
Strongly declared bindings: Declared using global sym::T, const sym=val, import Mod: sym, using Mod: sym or as an implicit strong global declaration in sym=val, where sym is known to be global (either by being at toplevle or as global sym=val inside a function).

In general, you always allowed to syntactically replace a weaker binding by a stronger one (although the runtime permits arbitrary binding deletion now, this is just a syntactic constraint to catch errors). Second, any implicit binding can be replaced by other implicit bindings as the result of changing the using'ed module. And lastly, any constants may be replaced by any other constants (irrespective of type).

We do not currently allow replacing globals, but may consider changing that in 1.13.

This is mostly how things used to work, as well in the absence of any stray external binding resolutions. The most prominent difference is probably this one:

set_foo!() = global foo = 1

In the above terminology, this now always declares a "strongly declared binding", whereas before it declared a "weakly declared binding" that would become strongly declared on first write to the global (unless of course somebody had created a different strongly declared global in the meantime). To see the difference, this is now disallowed:

julia> set_foo!() = global foo = 1
set_foo! (generic function with 1 method)

julia> const foo = 1
ERROR: cannot declare Main.foo constant; it was already declared global
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1

Before it would depend on the order of binding resolution (although it just crashes on current master for some reason - whoops, probably my fault).

Another major change is the ambiguousness of imports. In:

module M1; export x; x=1; end
module M2; export x; x=2; end
using .M1, .M2

the binding Main.x is now always ambiguous and will throw on access. Before which binding you get, would depend on resolution order. To choose one, use an explicit import (which was the behavior you would previously get if neither binding was resolved before both imports).

Keno · 2025-02-05T11:04:15Z

This should hopefully be passing tests now, modulo the revise tests, which need JuliaDebug/JuliaInterpreter.jl#658. There are some additional performance optimizations to (mostly to call invalidation less often), as well as a missing invalidation when somebody changes the export set that a precompile'd package sees. However, those don't affect the semantics and this PR is already large, so I'm planning to do those in a follow-up.

This is the final PR in the binding partitions series (modulo bugs and tweaks), i.e. it closes #54654 and thus closes #40399, which was the original design sketch. This thus activates the full designed semantics for binding partitions, in particular allowing safe replacement of const bindings. It in particular allows struct redefinitions. This thus closes timholy/Revise.jl#18 and also closes #38584. The biggest semantic change here is probably that this gets rid of the notion of "resolvedness" of a binding. Previously, a lot of the behavior of our implementation depended on when bindings were "resolved", which could happen at basically an arbitrary point (in the compiler, in REPL completion, in a different thread), making a lot of the semantics around bindings ill- or at least implementation-defined. There are several related issues in the bugtracker, so this closes #14055 #44604 #46354 #30277 It is also the last step to close #24569. It also supports bindings for undef->defined transitions and thus closes #53958 #54733 - however, this is not activated yet for performance reasons and may need some further optimization. Since resolvedness no longer exists, we need to replace it with some hopefully more well-defined semantics. I will describe the semantics below, but before I do I will make two notes: 1. There are a number of cases where these semantics will behave slightly differently than the old semantics absent some other task going around resolving random bindings. 2. The new behavior (except for the replacement stuff) was generally permissible under the old semantics if the bindings happened to be resolved at the right time. With all that said, there are essentially three "strengths" of bindings: 1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no binding", plus slightly more exotic corner cases around conflicts 2. Weakly declared bindings: Declared using `global sym` and nothing else 3. Strongly declared bindings: Declared using `global sym::T`, `const sym=val`, `import Mod: sym`, `using Mod: sym` or as an implicit strong global declaration in `sym=val`, where `sym` is known to be global (either by being at toplevle or as `global sym=val` inside a function). In general, you always allowed to syntactically replace a weaker binding by a stronger one (although the runtime permits arbitrary binding deletion now, this is just a syntactic constraint to catch errors). Second, any implicit binding can be replaced by other implicit bindings as the result of changing the `using`'ed module. And lastly, any constants may be replaced by any other constants (irrespective of type). We do not currently allow replacing globals, but may consider changing that in 1.13. This is mostly how things used to work, as well in the absence of any stray external binding resolutions. The most prominent difference is probably this one: ``` set_foo!() = global foo = 1 ``` In the above terminology, this now always declares a "strongly declared binding", whereas before it declared a "weakly declared binding" that would become strongly declared on first write to the global (unless of course somebody had created a different strongly declared global in the meantime). To see the difference, this is now disallowed: ``` julia> set_foo!() = global foo = 1 set_foo! (generic function with 1 method) julia> const foo = 1 ERROR: cannot declare Main.foo constant; it was already declared global Stacktrace: [1] top-level scope @ REPL[2]:1 ``` Before it would depend on the order of binding resolution (although it just crashes on current master for some reason - whoops, probably my fault). Another major change is the ambiguousness of imports. In: ``` module M1; export x; x=1; end module M2; export x; x=2; end using .M1, .M2 ``` the binding `Main.x` is now always ambiguous and will throw on access. Before which binding you get, would depend on resolution order. To choose one, use an explicit import (which was the behavior you would previously get if neither binding was resolved before both imports).

This is the final PR in the binding partitions series (modulo bugs and tweaks), i.e. it closes #54654 and thus closes #40399, which was the original design sketch. This thus activates the full designed semantics for binding partitions, in particular allowing safe replacement of const bindings. It in particular allows struct redefinitions. This thus closes timholy/Revise.jl#18 and also closes #38584. The biggest semantic change here is probably that this gets rid of the notion of "resolvedness" of a binding. Previously, a lot of the behavior of our implementation depended on when bindings were "resolved", which could happen at basically an arbitrary point (in the compiler, in REPL completion, in a different thread), making a lot of the semantics around bindings ill- or at least implementation-defined. There are several related issues in the bugtracker, so this closes #14055 closes #44604 closes #46354 closes #30277 It is also the last step to close #24569. It also supports bindings for undef->defined transitions and thus closes #53958 closes #54733 - however, this is not activated yet for performance reasons and may need some further optimization. Since resolvedness no longer exists, we need to replace it with some hopefully more well-defined semantics. I will describe the semantics below, but before I do I will make two notes: 1. There are a number of cases where these semantics will behave slightly differently than the old semantics absent some other task going around resolving random bindings. 2. The new behavior (except for the replacement stuff) was generally permissible under the old semantics if the bindings happened to be resolved at the right time. With all that said, there are essentially three "strengths" of bindings: 1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no binding", plus slightly more exotic corner cases around conflicts 2. Weakly declared bindings: Declared using `global sym` and nothing else 3. Strongly declared bindings: Declared using `global sym::T`, `const sym=val`, `import Mod: sym`, `using Mod: sym` or as an implicit strong global declaration in `sym=val`, where `sym` is known to be global (either by being at toplevle or as `global sym=val` inside a function). In general, you always allowed to syntactically replace a weaker binding by a stronger one (although the runtime permits arbitrary binding deletion now, this is just a syntactic constraint to catch errors). Second, any implicit binding can be replaced by other implicit bindings as the result of changing the `using`'ed module. And lastly, any constants may be replaced by any other constants (irrespective of type). We do not currently allow replacing globals, but may consider changing that in 1.13. This is mostly how things used to work, as well in the absence of any stray external binding resolutions. The most prominent difference is probably this one: ``` set_foo!() = global foo = 1 ``` In the above terminology, this now always declares a "strongly declared binding", whereas before it declared a "weakly declared binding" that would become strongly declared on first write to the global (unless of course somebody had created a different strongly declared global in the meantime). To see the difference, this is now disallowed: ``` julia> set_foo!() = global foo = 1 set_foo! (generic function with 1 method) julia> const foo = 1 ERROR: cannot declare Main.foo constant; it was already declared global Stacktrace: [1] top-level scope @ REPL[2]:1 ``` Before it would depend on the order of binding resolution (although it just crashes on current master for some reason - whoops, probably my fault). Another major change is the ambiguousness of imports. In: ``` module M1; export x; x=1; end module M2; export x; x=2; end using .M1, .M2 ``` the binding `Main.x` is now always ambiguous and will throw on access. Before which binding you get, would depend on resolution order. To choose one, use an explicit import (which was the behavior you would previously get if neither binding was resolved before both imports). (cherry picked from commit 888cf03)

inkydragon · 2025-02-07T15:42:50Z

Seems this pr broken x86_64-linux-gnummtk build.

https://buildkite.com/julialang/julia-master/builds?branch=master

Backported PRs: - [x] #57142  - [x] #57241  - [x] #57249  - [x] #57211  - [x] #57262  - [x] #57226  - [x] #57253  - [x] #57273  - [x] #57280  - [x] #57229  - [x] #57298  - [x] #57248  - [x] #57312  - [x] #57289  - [x] #57310  - [x] #57311  - [x] #57338  - [x] #57293  - [x] #57349  - [x] #57344  - [x] #57348  - [x] #55040  - [x] #57299