Add builtin functions Core._import, Core._using; implement import/using logic in Julia

[xal-0]

Overview

Currently, import and using statements are compiled into calls to
jl_toplevel_eval with the Expr as an argument. It can be useful in an
interactive environment to do import/using programmatically, for which
eval(Expr(:import, ...)) or a macro is the only option at the moment (see for
example InteractiveUtils.@activate).

This PR adds two builtins, _module_import and _module_using, with these
signatures, and removes Expr(:import/:using, ...) from the lowered IR:

_module_import(explicit::Bool, to::Module, ctx::Union{Expr, Nothing}, names::Expr...)
_module_using(to::Module, from::Expr{Symbol})

Lowering example

import statements and using X: statements are lowered to _module_import
like so:

import A             => _module_import(true,  Main, nothing,          Expr(:., :A))
import A.b           => _module_import(true,  Main, nothing,          Expr(:., :A, :b))
import A.b as c      => _module_import(true,  Main, nothing,          Expr(:as, Expr(:., :A, :b), :c))
import A.B: C.d, e   => _module_import(true,  Main, Expr(:., :A, :B), Expr(:., :C, :d), Expr(:., :e))
import A.B: C.d as e => _module_import(true,  Main, Expr(:., :A, :B), Expr(:as, Expr(:., :C, :d), :e))
using  A.B: C.d, e   => _module_import(false, Main, Expr(:., :A, :B), Expr(:., :C, :d), Expr(:., :e))

Plain using statements become _module_using:

using A.B            => _module_using(Main, Expr(:., :A, :B))

Multiple comma-separated using or import paths are lowered to multiple calls to the appropriate builtin:

julia> Meta.@lower using A.B, C
:($(Expr(:thunk, CodeInfo(
1 ─       builtin Core._module_using(Main, $(QuoteNode(:($(Expr(:., :A, :B))))))
│         builtin Core._module_using(Main, $(QuoteNode(:($(Expr(:., :C))))))
│       $(Expr(:latestworld))
└──     return nothing
))))

Alternative designs

Some alternatives to these builtins are worth considering.

The lowering pass could do even less work by inserting a call to
_module_import or _module_using that takes a single Expr argument for the
entire statement.

Another option is to do even more in lowering, introducing very low-level
builtins that wrap jl_module_import and jl_module_using directly. The
lowering pass would expand the import path into direct calls to Base.require,
Base.getproperty, etc. For example, import A.B: C.d, e could become
something like:

(block
  (= (ssavalue 1)
     (call (top getproperty)
           (call (top require) (thismodule) 'A)
           'B))
  (call (core module_import)
        (thismodule)
        (call (top getproperty) (ssavalue 1) 'C)
        'd)
  (call (core module_import) (thismodule) (ssavalue 1) 'd)
  (latestworld))

I like the idea of supporting calls to _module_import with a module for
ctx instead of an Expr, but this could be added to the proposed design too.

[Keno]

Generally positive on the direction.

For this particular implementation, I'll need to think about it in a bit more depth, but my initial impression is that this feels too "syntax oriented" for a builtin. I mean this in two senses. The first is that these builtins are taking Expr, which is a syntax data structure and not really touched anywhere else by the runtime system. The second is that it conflates the resolution of the name to be imported with the establishment of the link between them. I'd have expected something closer to:

_module_import(::Binding, ::Binding, imported::Bool)
_module_using(::Module, ::Module)

From this, there could then be higher level functions or macros that implement the syntax parts. The primary downside I see is that this approach would prevent batching of the world age update. However, I think that is possibly better handled with a generic world agree compression optimization in the runtime rather than trying to design the syntax around it.

[vtjnash]

Agreed as to our eventual plan here. I think there are few phases, one (here) is keeping most of the existing code structure, but introducing a function form for it. Secondly we can move a lot of this C code into julia using some simpler builtins for the primitive bits. I think the tricky part of deciding on that form is dealing with the fact this is very much just interpreting the syntax, so it is hard to see what that julia function's arguments should be other than exactly the AST.

[xal-0]

c638398 is a more-or-less direct translation of _module_import and _module_using into Julia, renamed to _eval_import and _eval_using because we already have jl_module_import and jl_import_module and it was a little confusing. The low-level builtins are now:

_import(to::Module, from::Module, asname::Symbol, [sym::Symbol, imported::Bool])

With all five arguments, imports sym from module from into to with name
asname. imported is true for bindings created with import (set it to
false for using A: ...).

With only the first three arguments, creates a binding for the module from
with name asname in to.

_using(to::Module, from::Module)

Add from to the usings list of to.

Core._using is a direct wrapper around jl_module_using. Core._import either calls jl_module_import (first form),or jl_import_module (second form, installs a PARTITION_KIND_CONST_IMPORT). If we could remove PARTITION_KIND_CONST_IMPORT it could be deleted.

[KristofferC]

Can use Core.Compiler.@zone now perhaps.

[oscardssmith]

is Core._using necessary to have as a builtin? Couldn't it have just been a wrapper over Core._import?

[xal-0]

is Core._using necessary to have as a builtin? Couldn't it have just been a wrapper over Core._import?

I thought it was nicest to keep them separate, and this distinction already existing in module.c: Core._using adds to the destination module's usings list, while Core._import always creates a single binding in the destination module (the two forms import a binding from the source module or create a const binding to the source module itself).

[c42f]

Drive by comment: this is great, it's exactly what I needed to fix some janky use of eval() in JuliaLowering.jl, thank you!

(It also breaks JuliaLowering, but that's a price I'm very much willing to pay 😆)