x/tools/go/packages: Load returns different results depending on build cache state #33687

hajimehoshi · 2019-08-16T14:32:40Z

What version of Go are you using (`go version`)?

$ go version
go version go1.12.5 darwin/amd64

Does this issue reproduce with the latest release?

Maybe yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/hajimehoshi/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/hajimehoshi/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ht/ky_bwgzs4bd5z1hh02k34x_h0000gn/T/go-build727446393=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Clear the cache by

go clean -cache

and run the below code:

package main

import (
        "fmt"

        "golang.org/x/tools/go/packages"
)

func main() {
        cfg := &packages.Config{
                Mode: packages.NeedName | packages.NeedFiles | packages.NeedCompiledGoFiles |
                        packages.NeedImports | packages.NeedDeps |
                        packages.NeedTypes | packages.NeedSyntax | packages.NeedTypesInfo,
        }
        allPkg, err := packages.Load(cfg, "golang.org/x/mobile/bind/testdata/testpkg/javapkg")
        if err != nil {
                panic(err)
        }
        fmt.Println(allPkg)
}

What did you expect to see?

[golang.org/x/mobile/bind/testdata/testpkg/javapkg]

What did you see instead?

[Java/java/beans]

In the second time with cache, the result became expected one.

CC @hyangah

The text was updated successfully, but these errors were encountered:

hajimehoshi · 2019-08-16T14:37:30Z

CC @matloob

hyangah · 2019-08-16T15:45:52Z

The "Java/..." packages are special package paths introduced to support for reverse-binding (See #16876, and also x/mobile/bind/genclasses.go files).

@eliasnaur Is it possible to generate those packages early? I hope we are not facing a chicken-and-egg problem.
@matloob Is it possible to teach package.Load to ignore some of the not-yet-existing packages?

hajimehoshi · 2019-08-16T16:40:18Z

I think this is not a mobile-specific issue, but an issue with unknown package.

For example,

// $GOPATH/go/src/github.com/hajimehoshi/testpkg/foo.go
package foo

import (
        "Fooooooo"
)

can cause the same issue:

// main.go
package main

import (
        "fmt"

        "golang.org/x/tools/go/packages"
)

func main() {
        cfg := &packages.Config{
                Mode: packages.NeedName | packages.NeedFiles | packages.NeedCompiledGoFiles |
                        packages.NeedImports | packages.NeedDeps |
                        packages.NeedTypes | packages.NeedSyntax | packages.NeedTypesInfo,
        }
        allPkg, err := packages.Load(cfg, "github.com/hajimehoshi/testpkg")
        if err != nil {
                panic(err)
        }
        fmt.Println(allPkg)
}

and run:

go clean -cache && env GO111MODULE=off go run . && env GO111MODULE=off go run .

The result is:

[Fooooooo]
[github.com/hajimehoshi/testpkg]

eliasnaur · 2019-08-16T21:05:01Z

The "Java/..." packages are special package paths introduced to support for reverse-binding (See #16876, and also x/mobile/bind/genclasses.go files).

@eliasnaur Is it possible to generate those packages early?

I'm not sure what you mean by "early". The Java/... and ObjC/... packages are generated as a first step before the actual generating of bindings.

I hope we are not facing a chicken-and-egg problem.

The reverse bindings work similarly to Cgo, where the AST of Go source files are analyzed for their imports matching Java/... or ObjC/... and for references using those imports. In a sense this is a chicken-and-egg problem because Go source that imports the magic packages are not buildable yet. However, since the analysis only needs the AST the chicken-and-egg problem didn't apply when using go/build. I don't know about go/packages.

hyangah · 2019-08-16T21:39:58Z

@hajimehoshi Indeed! Thanks for the small example for reproducing the issue.

 go clean -cache | env GO111MODULE=off GOPACKAGESDEBUG=true  go run . && env GO111MODULE=off GOPACKAGESDEBUG=true  go run .
2019/08/16 17:33:53 26.264922ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "env" "GOMOD", stderr: <<>>
2019/08/16 17:33:53 26.415485ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "list" "-m" "-json" "all", stderr: <>
2019/08/16 17:33:54 25.019105ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "env" "GOPATH", stderr: <<>>
2019/08/16 17:33:54 52.135849ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "list" "-e" "-json" "-compiled=true" "-test=false" "-export=false" "-deps=true" "-find=false" "--" "foo.com/cool", stderr: <>
[Foo]
2019/08/16 17:33:54 26.189039ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "env" "GOMOD", stderr: <<>>
2019/08/16 17:33:54 26.442476ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "list" "-m" "-json" "all", stderr: <>
2019/08/16 17:33:54 37.745538ms for GOROOT= GOPATH= GO111MODULE=off PWD=/Users/hakim/go/src/bar go "list" "-e" "-json" "-compiled=true" "-test=false" "-export=false" "-deps=true" "-find=false" "--" "foo.com/cool", stderr: <<>>
[foo.com/cool]

I will move this to x/tools/go/packages category.

hyangah · 2019-08-16T21:52:18Z

@hajimehoshi is it possible to work around the issue temporarily while we figure out what's going on with go/packages, by running Load twice to warm the cache?

gopherbot · 2019-08-17T05:47:29Z

Change https://golang.org/cl/190479 mentions this issue: cmd/gobind: fix the try bot failures.

hajimehoshi · 2019-08-17T07:53:00Z

I thought yes, but the trybots are still unhappy: https://storage.googleapis.com/go-build-log/c485506b/linux-amd64-androidemu_0afe1c3b.log

Apparently the error message comes from the Go compiler (the message "type-checking package ..." is defined at go/internal/srcimporter/srcimporter.go) so Go compiler version might matter. Unfortunately I could not reproduce this error with my local machine with Go 1.12 and Go 1.13 beta 1.

I'm fine reverting my change to use go/packages once, if we cannot find a good solution.

hyangah · 2019-08-19T16:57:58Z

@hajimehoshi I ran trybot after reverting your change (https://golang.org/cl/190837), and still got the same error https://storage.googleapis.com/go-build-log/0dd120df/linux-amd64-androidemu_d1e7d4ed.log

hajimehoshi · 2019-08-19T17:49:48Z

@hyangah That's interesting. Then I'd like to move on the CL https://go-review.googlesource.com/c/mobile/+/190479, that should make the situation better. Could you take a look? Thanks!

* Correctly format build tags to pass into go/packages * Removes CGO_ENABLED=0 from a packages.Load configuration * Calls go/packages.Load twice to work around a build cache * staleness issue These bugs were introduced by CL 189597. Updates golang/go#27234. Updates golang/go#33687. Change-Id: I3ae6737bf53bbecda0c7e25885b9c6aea5779332 Reviewed-on: https://go-review.googlesource.com/c/mobile/+/190479 Reviewed-by: Hyang-Ah Hana Kim <[email protected]> Run-TryBot: Emmanuel Odeke <[email protected]>

matloob · 2019-10-30T00:24:04Z

Is this still an issue in Go 1.13? I can't reproduce it now

$ go version
go version go1.13 darwin/amd64
$ go clean -cache
$ cat code.go
package main

import (
        "fmt"

        "golang.org/x/tools/go/packages"
)

func main() {
        cfg := &packages.Config{
                Mode: packages.NeedName | packages.NeedFiles | packages.NeedCompiledGoFiles |
                        packages.NeedImports | packages.NeedDeps |
                        packages.NeedTypes | packages.NeedSyntax | packages.NeedTypesInfo,
        }
        allPkg, err := packages.Load(cfg, "golang.org/x/mobile/bind/testdata/testpkg/javapkg")
        if err != nil {
                panic(err)
        }
        fmt.Println(allPkg)
}
$ go run code.go
go: finding golang.org/x/tools latest
[golang.org/x/mobile/bind/testdata/testpkg/javapkg]
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/matloob/Library/Caches/go-build"
GOENV="/Users/matloob/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/matloob/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f4/9p58ddnj40x58zchdb36p7lr004_sl/T/go-build859295338=/tmp/go-build -gno-record-gcc-switches -fno-common"

I'm also wondering whether this is a tools issue or a cmd/go issue if it's still a problem.

hajimehoshi · 2019-10-30T02:11:14Z

Me neither: I could not reproduce this with Go 1.13.

matloob · 2019-10-30T14:48:41Z

Okay, I'll mark this as fixed? Please reopen if this shows up again.

gopherbot added this to the Unreleased milestone Aug 16, 2019

hyangah changed the title ~~x/tools/go/package: Load returns weird results with a package including gomobile reverse bindings~~ x/mobile: x/tools/go/packages.Load is unhappy about not-yet-existing packages for reverse binding Aug 16, 2019

gopherbot added the mobile Android, iOS, and x/mobile label Aug 16, 2019

hyangah changed the title ~~x/mobile: x/tools/go/packages.Load is unhappy about not-yet-existing packages for reverse binding~~ x/tools/go/packages Load returns different results depending on build cache state Aug 16, 2019

bcmills changed the title ~~x/tools/go/packages Load returns different results depending on build cache state~~ x/tools/go/packages: Load returns different results depending on build cache state Aug 19, 2019

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 19, 2019

gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019

matloob closed this as completed Oct 30, 2019

golang locked and limited conversation to collaborators Oct 29, 2020

gopherbot added the FrozenDueToAge label Oct 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/go/packages: Load returns different results depending on build cache state #33687

x/tools/go/packages: Load returns different results depending on build cache state #33687

hajimehoshi commented Aug 16, 2019 •

edited

Loading

hajimehoshi commented Aug 16, 2019

hyangah commented Aug 16, 2019

hajimehoshi commented Aug 16, 2019 •

edited

Loading

eliasnaur commented Aug 16, 2019

hyangah commented Aug 16, 2019

hyangah commented Aug 16, 2019

gopherbot commented Aug 17, 2019

hajimehoshi commented Aug 17, 2019 •

edited

Loading

hyangah commented Aug 19, 2019

hajimehoshi commented Aug 19, 2019

matloob commented Oct 30, 2019

hajimehoshi commented Oct 30, 2019

matloob commented Oct 30, 2019

x/tools/go/packages: Load returns different results depending on build cache state #33687

x/tools/go/packages: Load returns different results depending on build cache state #33687

Comments

hajimehoshi commented Aug 16, 2019 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

hajimehoshi commented Aug 16, 2019

hyangah commented Aug 16, 2019

hajimehoshi commented Aug 16, 2019 • edited Loading

eliasnaur commented Aug 16, 2019

hyangah commented Aug 16, 2019

hyangah commented Aug 16, 2019

gopherbot commented Aug 17, 2019

hajimehoshi commented Aug 17, 2019 • edited Loading

hyangah commented Aug 19, 2019

hajimehoshi commented Aug 19, 2019

matloob commented Oct 30, 2019

hajimehoshi commented Oct 30, 2019

matloob commented Oct 30, 2019

hajimehoshi commented Aug 16, 2019 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

hajimehoshi commented Aug 16, 2019 •

edited

Loading

hajimehoshi commented Aug 17, 2019 •

edited

Loading