spec: initialization ordering does not consider references in interface methods

[karalabe]

Go version

go version go1.22.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/karalabe/Library/Caches/go-build'
GOENV='/Users/karalabe/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/karalabe/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/karalabe/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/karalabe/work/sources/github.com/karalabe/go-init-bug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/bc/q6rslcrn5tngrfvr9w8s45vr0000gn/T/go-build2401450664=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I hit a global variable init order that depends on the file build order (i.e. lexicographic order). I haven not used init() methods that could be problematic according to the spec, rather only defined variables, types and called methods on the types.

Repro needs multiple files, please see: https://github.com/karalabe/go-init-bug

What did you see happen?

Depending on the file order, the Go compiler accesses not-yet-initialized variables.

What did you expect to see?

I expect the Go compiler to correctly order my type definitions and variable declarations.

[gabyhelp]

Related Issues and Documentation

• cmd/compile: global variable initialization done in unexpected order #51913 (closed)
• cmd/compile: package-level variable initialization with constant dependencies doesn't match order specified in Go spec #66575 (closed)
• cmd/compile, runtime: initialization order within a file appears to have changed #49150
• cmd/compile: declaring a variable before a particular type definition causes compiler crash #50276 (closed)
• cmd/compile: declaration order of types affects cycle detection #68025
• cmd/compile: package-level variable initialization order looks not correct #66585 (closed)
• cmd/compile: type checking succeeds/fails depending on declaration order #52698 (closed)
• cmd/compile: unexpected behavior difference between global and in-function variable definition #19482 (closed)
• cmd/compile: the execution order of init functions from different packages is incorrect #65033 (closed)
• cmd/go: Errors sometimes reported out of source error #26452 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[karalabe]

Been digging into this a bit more, I can trigger it within a single file too:

https://go.dev/play/p/ArZIYeg1g-L

package main

import (
	"fmt"
)

type X struct{}
type Y struct{}

type Sizer interface {
	Size() int
}
func Size(obj Sizer) int { return obj.Size() }

var sizeBad = 2 * Size(new(Y)) // Change this to new(Y).Size() and it works correctly

func (*X) Size() int { return 100 }

var sizeY = Size(new(X)) + 10
func (*Y) Size() int { return sizeY }

func main() {
	fmt.Println(sizeBad)
}

Seems the root cause is that the interface call is not detected as an initialization dependency.

Edit: Which is still wrong, because nowhere in the code can any of the sizes be 0 since they are all initialized on declaration.

[karalabe]

Further simplified: https://go.dev/play/p/uGhQfR0tUKH

package main

import (
	"fmt"
)

// A boring interface and a method operating on it
// -----------------------------------------------
type Sizer interface {
	Size() int
}

func Size(obj Sizer) int {
	return obj.Size()
}
// -----------------------------------------------

var sizeBad = 2 * Size(new(S))

func Size1() int { return 1 }
var size2 = Size1() + 1

type S struct{}
func (*S) Size() int { return size2 }

func main() {
	fmt.Println(sizeBad)
}

Outputs 0, expected 4.

[ianlancetaylor]

CC @griesemer

This suggests that when initialization code refers to a type, we should look through all methods of the type for any initialization. However, I would not be surprised if introducing that introduces initialization loop errors for some existing code.

Rearranging the code does change the behavior, which is unfortunate.

[karalabe]

@ianlancetaylor The interface part in there is important too for some reason. If I were to replace Size(new(S)) with new(S).Size() then it works correctly, so it does consider references in methods, just it seems to break when crossing an interface boundary.

[cuonglm]

I think this behavior is allowed by spec, since interface methods are not considered for dependency references.

Dependency analysis is performed per package; only references referring to variables, functions, and (non-interface) methods declared in the current package are considered. If other, hidden, data dependencies exists between variables, the initialization order between those variables is unspecified.

[karalabe]

FWIW, the example from the spec does indeed produce the same behavior https://go.dev/play/p/yVeZx2xrsGO

That said, seems like something that would be better off fixed then left as a footgun?

[karalabe]

Perhaps just a description of my use case.

I have a code generator that can generate encoder/decoders between Go types and a specific binary format. As part of that format, static objects have fixed sizes and I'd like to precompute and store these sizes in some global package-private variable for each type so that I don't have to iterate over the fields of the types over and over and over all the time.

Since I can have nested objects, the size of an outer object naturally depends on the size of the internal ones (i.e. fields), so when pre-computing the outer size, I need the ensure the inner size is already pre-computed. Now, since this is a code generator that users can just run on their own types, it's out of my control as to what file order they'll end up having, and thus what init order they'll end up with.

But if this bug persists, then my outer type's size calculation might precede an inner type's size calculation, thus breaking everything. As this is not even a compilation error, the user will be none the wiser and this will result in a runtime error (or rather panic in my case).

Unfortunately I haven't figured out a way to force Go to actually forcefully run a size calculation on the type before calling my interface method. This could be highlighted in https://go.dev/play/p/-QEeLdw5kxx with the example from the specs, where I added a y = a declaration as the first one, but the compiler still delays evaluating this and rather does the evaluation of the line afterwards it which uses an uninitialised a. This kind of highlights that I can't even generate some trick into my code that would ensure proper ordering of events.

[cuonglm]

How about:

• Always generate code that involve interface methods call at the end of your file?
• Or do not use interface methods call at all?

[karalabe]

Always generate code that involve interface methods call at the end of your file?

I am generating one file per type, so unfortunately end or beginning is the same because the ordering issue arises across files. Sure, I could merge everything together an reorder them but then it becomes an unmaintainable mess that would be very hard to find issues in.

Or do not use interface methods call at all?

Unfortunately my size calculation is non trivial, so inlining all that code for every type just feels the wrong solution.

[cuonglm]

Unfortunately my size calculation is non trivial, so inlining all that code for every type just feels the wrong solution.

What do you mean when saying "inlining all that code for every type"?

I mean instead of generating Size(new(Y)), you could just generate new(Y).Size() which will work.

You can still do var _ Sizer = (*Y)(nil) to ensure Y satisfy your Sizer.

[ianlancetaylor]

Apologies for missing that interface methods are involved.

I don't see how we can handle initialization ordering correctly (or perhaps I should say "correctly") when interface methods are involved. In the general case we aren't going to be able to figure out the dynamic type of an interface value, so we aren't going to be able to know which method was called, so we aren't going to be able to ensure that all references from that method was fully initialized before the interface conversion occurs.

So I don't see what change we can make here that will keep initialization simple and reliable.