make closure over comptime var a compile error; comptime vars become immutable when they go out of scope

[SpexGuy]

Update

---

This issue is heavily inspired by #5895 and #5578.

As it stands right now, the compiler allows you to create a closure over mutable comptime state. This allows some unique abilities, like building a borrow checker and lazily creating a global list based on what functions or types get compiled.

But it also causes a lot of problems:

• it wreaks havoc with the comptime memoization system.
• it makes comptime execution order observable.
• it prevents incremental compilation.

The aim of this issue is to provide a solid bedrock for how comptime var should behave, without introducing new features. There may be extensions that add new features in future proposals.

After discussing in the design meeting, this is how we believe comptime var (and comptime mutable memory in general) should work:

• closure over comptime var is a compile error.
• closure over const pointer to mutable memory (e.g. comptime const x: *u32 = ...) is allowed
• when mutable pointers are written at comptime, the compiler ensures that the underlying memory is still mutable
• the memory underlying comptime vars is marked immutable when the vars go out of scope.
• if the compiler attempts to generate runtime code which dereferences a mutable pointer to a comptime var, that’s also a compile error.
• for the purposes of memoization, any values which are closed over are treated like additional parameters.

---

This means that the example given in #5578 can be implemented like this:

const GiveCtState = struct {
    times: usize,
};
fn Give(comptime T: type, comptime state: *GiveCtState) type {
    return struct {
        val: T,
        pub fn init(_val: T) @This() {
            return Self{ .val = _val };
        }

        pub fn give(self: *@This()) T {
            comptime {
                if (state.times == 0)
                    @compileError("You can only 'give' 2 times");

                state.times -= 1;
            }
            return self.val;
        }
    };
}

test "Test 1" {
    // create comptime state in this scope
    comptime var track_a = GiveTracker{ .times = 2 };
    // create a type that closes over that state
    var a = Give(usize, &track_a).init(1);
    _ = a.give();
    _ = a.give(); // not memoized because it closes over &track_a
    _ = a.give(); // <-- Should Fail

    comptime var track_b = GiveTracker{ .times = 2 };
    var b = Give(usize, &track_b).init(1); // not memoized because it accepts &track_b
    _ = b.give();
    _ = b.give();
    _ = b.give(); // <-- Should Fail
}

---

The other use case, building a global index lazily based on what gets compiled, is not officially supported by Zig. The recommended alternative way to approach this problem is to use comptime code to create an index explicitly, either from a hardcoded list, or by calling functions in submodules that will return the needed parts of the index for each submodule, and aggregating those results into a single global index.

[ghost]

#5895 is now appropriately divided, as per recommendation.

[SpexGuy]

Adding links to some related issues:
#1470: interactions between comptime var and runtime branches
#2710: interactions between comptime var and runtime function calls
#7948: constant deduplication for memoization

[topolarity]

Out of curiosity, what does this change mean for examples like this?

test {
    comptime var i: usize = 0;
    const f = struct {
        fn foo(x: usize) bool {
            return x == i;
        }
    }.foo;
    try expect(f(0)); // Currently fails because of `i=1` on the next line
    i = 1;
}

Would this count as closure (by reference) over i, as it does today? Or will it capture by value?

[ghost]

@topolarity That will fail to compile. No closure, no capture, error. That's the point.

[topolarity]

@topolarity That will fail to compile. No closure, no capture, error. That's the point.

My question is whether it will be capture by value, so that it is not an error and the test passes. Or if it will be captured by reference, so that this code does not compile.

[ghost]

My understanding is that your example should compile and pass:

test {
    comptime var i: usize = 0;
    const f = struct {
        fn foo(x: usize) bool {
            return x == i;
        }
    }.foo;
    try expect(f(0)); // f(0) desugars to f(i,0) as per rule 6
    i = 1; // no problems
} // i becomes const, if it is still referenced somewhere (rule 4)

[ghost]

@topolarity @zzyxyzz Think of it like this: a function in function scope is a new scope, which cannot see any locals from the enclosing function. Since i is local to test, it is not visible to foo; in effect, the name is undefined inside it (though the reported error is different). If foo wants to use i, it needs to take it as an argument (by value or by reference), or test needs to assign its value to a const (comptime consts being the only locals visible to inner functions).

The point of this change is to avoid exactly this kind of question; however this works in a particular case is explicitly written. This is a problem for the compiler too, as currently it can evaluate things out of order and produce confusing results (as in your example). I was in the design meeting where this was decided and it was my proposal that was eventually shaped into this, so you can trust what I tell you about it.

[ghost]

@EleanorNB
Could you comment on why closing over const pointers to mutable comptime variables is still permitted? It seems to me that this proposal could be simplified considerably if there was a blanket ban on referring to outer-scope comptime mutable state in any way within local functions.

[ghost]

@zzyxyzz On the contrary: detecting that a pointer is to mutable state and banning specifically that is another layer of complication and another rule exception to remember. It's more complicated. (You might think this is detectable by the signature alone, but regular functions can be evaluated at comptime; whether a pointer is to comptime state or runtime data is only fully determinable by looking at the surrounding context.)

[ghost]

Zig's comptime semantics are an incredible mess.

[ghost]

Yep. That's why this exists.

[topolarity]

Banning closure (by-reference) over a comptime var is not essential to this proposal, right?

You can still achieve the same thing by closing over a ptr to its contents:

test {
    comptime var i: usize = 0;
    const ptr: *const usize = &i;
    const f = struct {
        fn foo(x: usize) bool {
            return x == ptr.*;
        }
    }.foo;
    try expect(f(0)); // Should this pass?
    i = 1;
    try expect(f(1)); // Should this fail?
}

If I understand the proposal correctly, I believe the first expect would pass, and the second would fail.

That is, unless the first was commented out, in which case the second would suddenly pass.

This strange behavior is due to #7948, of course.

[ghost]

@topolarity You sort of have it. That kind of closure by reference will be allowed, but it would play out like this:

test {
    comptime var i: usize = 0; // in scope throughout `test`
    const ptr: *const usize = &i; // can be closed over as it is constant
    const f = struct {
        fn foo(x: usize) bool { // cannot see `i`, but can see `ptr`
            return x == ptr.*; // references the current value of `i`
        }
    }.foo;
    try expect(f(0)); // will pass as `i` is 0 here
    i = 1;
    try expect(f(1)); // will also pass as `i` is now 1,
                      // and `ptr` tracks the current value just as at runtime
}

comptime vars continue to be mutable in the scope they're defined in, and any dynamically enclosed scope, regardless of depth or whether they're referenced or mutated in those inner scopes. Dereferencing a pointer, even an immutable pointer, to a comptime var returns the current value of it, regardless of where it's dereferenced, at arbitrary depth or even outside the defining scope. The only restriction is that a mutable pointer can't mutate a value that's out of scope; it can still be dereferenced, just not assigned. So here, since i is mutated between calls to f, the dereference of ptr within the function will produce different values, just as a runtime pointer dereference within a function will produce different values on different invocations.

(Btw: attempting out-of-scope mutation is a compile error, not a silent failure, just like attempting to mutate an immutable value. Or, it will be, once this is implemented. I'm not familiar with #7948, but from a glance it is a bug and will be fixed.)

[topolarity]

@EleanorNB Great, thanks for the clear explanation.

I agree that behavior would be correct. However, I'm not sure we've achieved it yet

The gist of #7948 is that pointers are compared shallow-ly for the purposes of comptime memoization. In the original issue, that meant functions were re-analyzed too often, despite the referenced value being the same between invocations with different pointers. In this case, functions are re-analyzed too little, even when the referenced value has changed.

As your explanation demonstrates, this is unsound with respect to the runtime semantics - it means that f(0) reports the same value throughout the entire program.

I think we might need to fix memoization before this will work properly

[mlugg]

Some parts of this proposal have been implemented, but in other ways, we've diverged from it. This proposal has been effectively un-accepted in its current form, although the language rules aren't finalized, so we could revisit parts of it. Here's the current status.

---

Closure over a comptime var's value is allowed, and records the value at the point of type declaration:

test "closure over comptime var" {
    comptime var x: u32 = 123;
    const S = struct {
        const val = x;
    };
    x = 456;
    try std.testing.expect(S.val == 123);
}
const std = @import("std");

I think it might be worth changing this and making it a compile error for simplicity/clarity; however, doing so when these rules were originally implemented caused a few std regressions, so I chose to kick that can down the road. I'm open to discussion on what's best here.

---

Closure over a pointer to a comptime var (regardless of whether the pointer type is const-qualified) is always a compile error (this includes closing over a comptime var as an lvalue):

test "closure over mutable pointer to comptime var" {
    comptime var x: u32 = 123;
    const ptr: *u32 = &x;
    _ = struct {
        const val = ptr;
    };
}

test "closure over const pointer to comptime var" {
    comptime var x: u32 = 123;
    const ptr: *const u32 = &x;
    _ = struct {
        const val = ptr;
    };
}

test "closure over comptime var as lvalue" {
    comptime var x: u32 = 123;
    _ = struct {
        fn foo() void {
            x += 1;
        }
    };
}

$ zig test foo.zig
foo.zig:4:9: error: captured value contains reference to comptime var
    _ = struct {
        ^~~~~~
foo.zig:2:27: note: 'ptr' points to comptime var declared here
    comptime var x: u32 = 123;
                          ^~~
foo.zig:12:9: error: captured value contains reference to comptime var
    _ = struct {
        ^~~~~~
foo.zig:10:27: note: 'ptr' points to comptime var declared here
    comptime var x: u32 = 123;
                          ^~~
foo.zig:19:9: error: captured value contains reference to comptime var
    _ = struct {
        ^~~~~~
foo.zig:18:27: note: 'x' points to comptime var declared here
    comptime var x: u32 = 123;
                          ^~~

I think this rule is completely reasonable, because it makes it impossible to ever "get back to" a pointer to comptime-mutable memory. It simplifies the compiler implementation, and that isn't just a property of our implementation; this design means that comptime vars can be completely local to "units" of semantic analysis rather than keeping them around forever with rules about changing mutability.

It is also a compile error to:

• make a pointer to comptime var runtime-known in any way (e.g. store it to a runtime pointer, pass it as a runtime function argument, perform runtime pointer arithmetic on it)
• pass a pointer to a comptime var as an argument to a non-inline non-comptime function call, even if the parameter is comptime
  • because accesses to comptime vars need to be well-ordered at comptime, passing around pointers to comptime vars only makes sense for inline calls; other calls have ill-defined analysis order, and the callee is only analyzed once for N calls
• have the resolved value of a const/var declaration contain a pointer to a comptime var

The above rules also all apply to aggregates containing these pointers, including even through other pointers:

const global = val: {
    var x: u32 = undefined;
    x = 123;

    const bad_ptr = &x; // can we sneak this into `global` somehow?

    const some_array: [2]*const u32 = .{ bad_ptr, undefined };
    const some_pointer: *const *const u32 = &some_array[1];
    // Let's even type-erase the inner pointer!
    const final: *const usize = @ptrCast(some_pointer);
    break :val final;
};
comptime {
    _ = global;
}

$ zig build-obj foo.zig
foo.zig:1:21: error: global variable contains reference to comptime var
const global = val: {
               ~~~~~^
foo.zig:1:21: note: 'global' points to '@as(*const usize, @ptrCast(&v0[1])).*', where
foo.zig:2:18: note: 'v0[0]' points to comptime var declared here
    var x: u32 = undefined;
                 ^~~~~~~~~

---

When a comptime var goes out of scope, it is still mutable. Comptime-mutable memory is tied to the "unit" of semantic analysis it is a part of (i.e. the declaration, function body, or similar). For instance:

comptime {
    const ptr: *u32 = ptr: {
        var x: u32 = undefined;
        break :ptr &x;
    };
    ptr.* = 123;
    @compileLog(ptr.*); // @as(u32, 123)
}

This restriction could be tightened (per this proposal), but I don't think there is any real need to. As a nice bonus, the current system also allows for a lovely comptime allocator implementation:

const comptime_allocator: std.mem.Allocator = .{
    .ptr = undefined,
    .vtable = &.{
        .alloc = &comptimeAlloc,
        .resize = &comptimeResize,
        .free = &comptimeFree,
    },
};

fn comptimeAlloc(_: *anyopaque, len: usize, ptr_align: u8, _: usize) ?[*]u8 {
    // these panics are necessary because we do emit runtime versions of these functions (our "is X
    // only used at comptime" analysis breaks here). so not having the panics leads to compile errors
    if (!@inComptime()) @panic("comptime_allocator used at runtime");
    var buf: [len]u8 align(1 << ptr_align) = undefined;
    return &buf;
}
fn comptimeResize(_: *anyopaque, buf: []u8, buf_align: u8, new_len: usize, _: usize) bool {
    if (!@inComptime()) @panic("comptime_allocator used at runtime");
    _ = buf_align;
    return new_len <= buf.len;
}
fn comptimeFree(_: *anyopaque, buf: []u8, buf_align: u8, _: usize) void {
    if (!@inComptime()) @panic("comptime_allocator used at runtime");
    _ = buf;
    _ = buf_align;
}

comptime {
    const foo = comptime_allocator.alloc(u32, 4) catch unreachable;
    @memset(foo, 0);
    foo[2] = 2;
    @compileLog(foo[0..].*);
}

const std = @import("std");

$ zig build-obj foo.zig
foo.zig:30:5: error: found compile log statement
    @compileLog(foo[0..].*);
    ^~~~~~~~~~~~~~~~~~~~~~~

Compile Log Output:
@as([4]u32, .{ 0, 0, 2, 0 })