proxy.golang.org: '400 bad request' for pseudo-versions that refer to commits that also have canonical version tags

[ghost]

What version of Go are you using (go version)?

$ go version
go version go1.13beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOHOSTARCH="amd64"
GOHOSTOS="linux"

Ubuntu Disco Dingo, go1.13beta1 downloaded from the Golang site.

What did you do?

go get -d github.com/future-architect/vuls
cd ~/go/src/github.com/future-architect/vuls
  # see that it has a `go.mod` already, so ..
go install

What did you expect to see?

Either a successful build or some kind of build failure

What did you see instead?

A bad request from proxy.golang.org:

go: github.com/tomoyamachi/[email protected]: reading https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.mod: 400 Bad Request

I repeated the command, to rule out transient glitch. I then ran GOPROXY=direct go install which continued past that point to get to a different inscrutable error but one which is more likely tied to the project itself. Limiting this issue to the observed proxy error.

[bcmills]

to get to a different inscrutable error but one which is more likely tied to the project itself

That suggests that the proxy probably should have instead returned a 404 or 410 to prompt the go command to fall back to direct.

CC @hyangah @katiehockman @heschik

[heschi]

If you visit the URL directly you'll see that it says "invalid version". That could be improved, but I don't know how much we should invest in it until #30748 is fixed; odds are nobody will see it anyway.

[heschi]

@bcmills: https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.info is (for better or worse) a pointer to v0.16.1. Do you know why the go command is still reading v0.16.2-0.20190418055600-c6010b917a55.mod? This seems like a variation of the wrong .zip bug that was fixed in 1.13?

[bcmills]

@heschik, the validation rule in 1.13beta1 is that the base tag for the pseudo version must be on the commit itself or one of its ancestors, so v0.16.2-0.20190418055600-c6010b917a55.info is an allowed name for that commit.

I suppose we could tighten up that validation so that the base tag must be on some actual ancestor rather than the commit itself. I don't have a solid intuition one way or the other on that (cc @jayconrod).

[bcmills]

Hrm, that's odd. Using go1.13beta1, I get a different result from what @it-philpennock observed:

$ GO111MODULE=on go1.13beta1 get -d github.com/tomoyamachi/[email protected]
go: finding github.com/tomoyamachi/reg v0.16.2-0.20190418055600-c6010b917a55
go: finding github.com v0.16.2-0.20190418055600-c6010b917a55
go: finding github.com/tomoyamachi v0.16.2-0.20190418055600-c6010b917a55
go get github.com/tomoyamachi/[email protected]: github.com/tomoyamachi/[email protected]: proxy returned info for version v0.16.1 instead of requested version

[ghost]

nb: I was installing github.com/future-architect/vuls and tomoyamachi/reg is a dependency of that.

[bcmills]

Ah, here we go. I guess we try to download the .mod without bothering with the .info if it's already in the go.mod and go.sum files.

For completeness:

$ GO111MODULE=on go1.13beta1 get -d github.com/future-architect/vuls
go: finding github.com/future-architect/vuls v0.8.2
verifying github.com/genuinetools/[email protected]/go.mod: github.com/genuinetools/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/genuinetools/[email protected]: 410 Gone

$ git clone https://github.com/future-architect/vuls
Cloning into 'vuls'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 5310 (delta 10), reused 16 (delta 8), pack-reused 5286
Receiving objects: 100% (5310/5310), 5.33 MiB | 27.58 MiB/s, done.
Resolving deltas: 100% (3679/3679), done.

$ cd vuls

vuls$ go1.13beta1 list all
go: github.com/tomoyamachi/[email protected]: reading https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.mod: 400 Bad Request

[bcmills]

FWIW, there are other bogus versions mentioned in that same go.mod file:

vuls$ GOPROXY=direct go1.13beta1 list all
go: github.com/kotakanbe/[email protected]: invalid pseudo-version: does not match version-control timestamp (2019-06-13T06:46:14Z)

[gopherbot]

Change https://golang.org/cl/184720 mentions this issue: cmd/go: tighten the check for pseudo-version base tags

[bcmills]

CL 184720 will tighten up the validation so that this pseudo-version is treated as actually-invalid.

However, there remains a bug in the proxy for valid pseudo-versions naming commits that have subsequently had canonical tags applied:

example.com$ go mod init example.com
go: creating new go.mod: module example.com

example.com$ go version
go version devel +fbde753a58 Tue Jul 2 13:52:21 2019 +0000 linux/amd64

example.com$ GOPROXY='' go get -d golang.org/x/[email protected]
go: finding golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org/x v0.1.1-0.20171213102548-c4d099d611ac
go get golang.org/x/[email protected]: golang.org/x/[email protected]: proxy returned info for version v0.2.0 instead of requested version

example.com$ GOPROXY=direct go get -d golang.org/x/[email protected]
go: finding golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
verifying golang.org/x/[email protected]/go.mod: golang.org/x/[email protected]/go.mod: reading https://sum.golang.org/lookup/golang.org/x/[email protected]: 410 Gone

example.com$ GOPROXY=direct GOSUMDB=off go get -d golang.org/x/[email protected]
1ac
go: downloading golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
go: extracting golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac

[ghost]

Another instance of this error message spotted in the wild, fingers crossed on my part that you've already fixed it with the above?

In github.com/minio/mc a go build hits:

go: github.com/minio/[email protected] requires
	github.com/minio/[email protected] requires
	github.com/minio/[email protected] requires
	github.com/minio/[email protected]: reading https://proxy.golang.org/github.com/minio/lsync/@v/v0.0.0-20190207022115-a4e43e3d0887.mod: 400 Bad Request

[heschi]

This is the case Bryan mentioned in #32879 (comment). We don't know what we're going to do about it yet, but it's on our radar.

[bcmills]

@it-philpennock, I just checked go get -d github.com/minio/[email protected] and go build ./... within a clone of the same, and they both seem to be working now. (There have been some changes in the proxy configuration in the time since your last comment.)

Are you still observing this issue? If not, I'll close it out.

[ghost]

Works for me now. Thanks.

[andybons]

Closing per last two comments.