cmd/go: prefers compiled pkg in second GOPATH root to rebuilding source in first

[calmh]

I'm doing GOPATH-based vendoring, where my GOPATH is basically "$MyVendorDir:$HOME". This works perfectly fine most of the time. However in some cases, when the version of a package directly in GOPATH differs significantly from the vendored version, it doesn't. I think this highlights a race condition or other issue in the "go" tool when doing paralell compilation that doesn't show up when the vendored version of a package matches the "default" version.

These are my paths; there is a vendored github.com/calmh/xdr package, and I remove the pkg dirs in both GOPATH components before building:

jb@syno:~/s/g/s/syncthing $ go version
go version go1.5.2 darwin/amd64
jb@syno:~/s/g/s/syncthing $ echo $GOPATH
/Users/jb/src/github.com/syncthing/syncthing/Godeps/_workspace:/Users/jb
jb@syno:~/s/g/s/syncthing $ pwd
/Users/jb/src/github.com/syncthing/syncthing
jb@syno:~/s/g/s/syncthing $ ls -l Godeps/_workspace/src/github.com/calmh/xdr
total 112
-rw-r--r--  1 jb  staff  1059 Feb  1 08:17 LICENSE
-rw-r--r--  1 jb  staff   660 Feb  1 08:17 README.md
...
-rw-r--r--  1 jb  staff   675 Feb  1 08:17 writer_ipdr.go
-rw-r--r--  1 jb  staff   365 Feb  1 08:17 writer_xdr.go
jb@syno:~/s/g/s/syncthing $ rm -r Godeps/_workspace/pkg ~/pkg

When I then build, I hit this:

jb@syno:~/s/g/s/syncthing $ go install ./cmd/...
# github.com/syncthing/syncthing/cmd/stfinddevice
github.com/syncthing/syncthing/lib/discover.Relay.EncodeXDR: type.github.com/calmh/xdr.Writer: not defined
github.com/syncthing/syncthing/lib/discover.Relay.AppendXDR: type.github.com/calmh/xdr.AppendWriter: not defined
github.com/syncthing/syncthing/lib/discover.Relay.AppendXDR: type.github.com/calmh/xdr.Writer: not defined
github.com/syncthing/syncthing/lib/discover.Relay.AppendXDR: type.*github.com/calmh/xdr.AppendWriter: not defined
github.com/syncthing/syncthing/lib/discover.Relay.EncodeXDRInto: github.com/calmh/xdr.(*Writer).WriteString: not defined
github.com/syncthing/syncthing/lib/discover.Relay.EncodeXDRInto: github.com/calmh/xdr.(*Writer).WriteUint32: not defined
github.com/syncthing/syncthing/lib/discover.Relay.EncodeXDRInto: type.github.com/calmh/xdr.XDRError: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).DecodeXDR: type.github.com/calmh/xdr.Reader: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).UnmarshalXDR: type.github.com/calmh/xdr.Reader: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).DecodeXDRFrom: github.com/calmh/xdr.(*Reader).ReadStringMax: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).DecodeXDRFrom: github.com/calmh/xdr.(*Reader).ReadUint32: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).DecodeXDRFrom: type.github.com/calmh/xdr.XDRError: not defined
type.func(*github.com/calmh/xdr.Reader) error: type.*github.com/calmh/xdr.Reader: not defined
type.func(*github.com/calmh/xdr.Writer) (int, error): type.*github.com/calmh/xdr.Writer: not defined
type.func(*github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Reader) error: type.*github.com/calmh/xdr.Reader: not defined
type.func(*github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Writer) (int, error): type.*github.com/calmh/xdr.Writer: not defined
type.func(github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Writer) (int, error): type.*github.com/calmh/xdr.Writer: not defined
panic: runtime error: index out of range

goroutine 1 [running]:
cmd/link/internal/ld.defgotype(0xc821bfb180, 0x5)
    /usr/local/go/src/cmd/link/internal/ld/dwarf.go:984 +0x4b42
cmd/link/internal/ld.defdwsymb(0x0, 0xc821c20ec0, 0x38, 0x61, 0x0, 0x0, 0x0, 0xc821bfb180)
    /usr/local/go/src/cmd/link/internal/ld/dwarf.go:1447 +0x4be
cmd/link/internal/ld.genasmsym(0x2c85c0)
    /usr/local/go/src/cmd/link/internal/ld/lib.go:1858 +0x6d6
cmd/link/internal/ld.Dwarfemitdebugsections()
    /usr/local/go/src/cmd/link/internal/ld/dwarf.go:2179 +0xa46
cmd/link/internal/amd64.asmb()
    /usr/local/go/src/cmd/link/internal/amd64/asm.go:665 +0x8ba
cmd/link/internal/ld.Ldmain()
    /usr/local/go/src/cmd/link/internal/ld/pobj.go:246 +0x1ed6
cmd/link/internal/amd64.Main()
    /usr/local/go/src/cmd/link/internal/amd64/obj.go:44 +0x19
main.main()
    /usr/local/go/src/cmd/link/main.go:26 +0x189
jb@syno:~/s/g/s/syncthing $

There's two strange things here. The first is the compile errors complaining about types not being defined in github.com/calmh/xdr. This is because it's using the version from ~/src (the second component of GOPATH), not the vendored one (the first component). The second is the panic, which doesn't look good but also doesn't always happen...

Looking at what it actually produced before erroring we can see that it built both the XDR package at the top level GOPATH and the vendored version (and they differ, in this case) - notice the timestamps on xdr.a and the fact that I nuked both pkg dirs before building:

jb@syno:~/syno/g/s/syncthing $ ls -l Godeps/_workspace/pkg/darwin_amd64/github.com/calmh
total 336
-rw-r--r--  1 jb  staff    5182 Feb  1 08:34 du.a
-rw-r--r--  1 jb  staff   33230 Feb  1 08:34 luhn.a
-rw-r--r--  1 jb  staff  123500 Feb  1 08:34 xdr.a
jb@syno:~/s/g/s/syncthing $ ls -l ~/pkg/
total 0
drwxr-xr-x  3 jb  staff  102 Feb  1 08:34 darwin_amd64/
jb@syno:~/s/g/s/syncthing $ ls -l ~/pkg/darwin_amd64/github.com/calmh/
total 152
-rw-r--r--  1 jb  wheel  77688 Feb  1 08:34 xdr.a
jb@syno:~/s/g/s/syncthing $ 

The XDR package is used by many things in "./cmd/..." - is there maybe some race condition around that when the build parallellizes things?

This all happens "now and then", and I finally solve it by touching source files, repeatedly running go install and so on to try to narrow it down ... and then it compiles just fine again... When writing this up, it started working when I used "go install -v -x" to find out what exact command builds the "wrong" version of the XDR package, without actually changing anything on disk, and now I cant reproduce it again. So I don't have a minimal reproducer, but there's something weird going on here. :(

[rsc]

Thanks for the report. Definitely sounds like something we need to fix. If you can make this happen on demand (even rarely), it would help to see the output of adding -x to the failing "go install ./cmd/...".

[rsc]

I see now that you said you couldn't reproduce this with -x.

I spent a while looking at the source code for cmd/go. All the package loading runs in a single goroutine. There are no races here. And I see no way that a reference to github.com/calmh/xdr could be resolved using the second GOPATH entry. Do you perhaps have some kind of cron script or other background builds running? The errors you got from the linker would make sense if maybe something was reinstalling the package files underfoot.

[calmh]

You're probably right. I don't have anything periodic building things, but my editor does, and I probably had the XDR package open in the editor at the time. The above error (minus the panic) is trivially reproducible by having an xdr.a file in ~/pkg as it then won't try to build the package from Godeps/_workspace even though that's earlier in GOPATH. I think this is a known limitation with this method of building with dependencies. :/

[rsc]

Got one! This does look like a bug to me. Will investigate. Thanks.

$ rm -r Godeps/_workspace/pkg /tmp/gopath/pkg
$ mkdir -p /tmp/gopath/pkg/darwin_amd64/github.com/calmh
$ echo malformed! > /tmp/gopath/pkg/darwin_amd64/github.com/calmh/xdr.a
$ GOPATH=$(pwd)/Godeps/_workspace:/tmp/gopath go install ./cmd/...
# github.com/syncthing/syncthing/cmd/stfinddevice
/tmp/gopath/pkg/darwin_amd64/github.com/calmh/xdr.a: not an object file
github.com/syncthing/syncthing/lib/discover.Relay.MarshalXDRInto: github.com/calmh/xdr.ElementSizeExceeded: not defined
github.com/syncthing/syncthing/lib/discover.Relay.MarshalXDRInto: github.com/calmh/xdr.(*Marshaller).MarshalString: not defined
github.com/syncthing/syncthing/lib/discover.Relay.MarshalXDRInto: github.com/calmh/xdr.(*Marshaller).MarshalUint32: not defined
github.com/syncthing/syncthing/lib/discover.(*Relay).UnmarshalXDRFrom: github.com/calmh/xdr.(*Unmarshaller).UnmarshalStringMax: not defined
github.com/syncthing/syncthing/lib/discover.init: github.com/calmh/xdr.init: not defined
github.com/syncthing/syncthing/lib/protocol.init: github.com/calmh/xdr.init: not defined
type.func(*github.com/calmh/xdr.Marshaller) error: type.*github.com/calmh/xdr.Marshaller: not defined
type.func(*github.com/calmh/xdr.Unmarshaller) error: type.*github.com/calmh/xdr.Unmarshaller: not defined
type.func(*github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Marshaller) error: type.*github.com/calmh/xdr.Marshaller: not defined
type.func(*github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Unmarshaller) error: type.*github.com/calmh/xdr.Unmarshaller: not defined
type.func(github.com/syncthing/syncthing/lib/discover.Relay, *github.com/calmh/xdr.Marshaller) error: type.*github.com/calmh/xdr.Marshaller: not defined
panic: runtime error: index out of range

goroutine 1 [running]:
cmd/link/internal/ld.defgotype(0xc8219c0b40, 0x5)
    /Users/rsc/go/src/cmd/link/internal/ld/dwarf.go:970 +0x4b33
cmd/link/internal/ld.defdwsymb(0x0, 0xc8219d5440, 0x38, 0x61, 0x30, 0x0, 0x0, 0xc8219c0b40)
    /Users/rsc/go/src/cmd/link/internal/ld/dwarf.go:1403 +0x4be
cmd/link/internal/ld.genasmsym(0x2dc258)
    /Users/rsc/go/src/cmd/link/internal/ld/lib.go:1990 +0x675
cmd/link/internal/ld.Dwarfemitdebugsections()
    /Users/rsc/go/src/cmd/link/internal/ld/dwarf.go:2102 +0xa3f
cmd/link/internal/amd64.asmb()
    /Users/rsc/go/src/cmd/link/internal/amd64/asm.go:663 +0x8bb
cmd/link/internal/ld.Ldmain()
    /Users/rsc/go/src/cmd/link/internal/ld/pobj.go:248 +0x1f5d
cmd/link/internal/amd64.Main()
    /Users/rsc/go/src/cmd/link/internal/amd64/obj.go:44 +0x19
main.main()
    /Users/rsc/go/src/cmd/link/main.go:27 +0x36f
$ 

[gopherbot]

CL https://golang.org/cl/19385 mentions this issue.