cmd/go: git remote (Gerrit) did not send all necessary objects

[ukai]

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/ukai/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/ukai/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build741696436=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ls /tmp/go
ls: cannot access '/tmp/go': No such file or directory
$ mkdir /tmp/go
$ GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy

What did you expect to see?

go modules downloaded and binary is built.

What did you see instead?

...
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: google.golang.org/[email protected]: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
        error: remote did not send all necessary objects
..
go: error loading module requirements

I could fix it by

$ cd /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0; git fetch --unshallow

then go install works as expected.

https://bugs.chromium.org/p/chromium/issues/detail?id=951546

[agnivade]

@bcmills @jayconrod

[KillerX]

I can confirm that this is happening to me as well. It is not 100% constant. My docker builds experience it about 98% of the time.

go: google.golang.org/[email protected]: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
	fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
	error: https://code.googlesource.com/google-api-go-client did not send all necessary objects

A git fetch seems to fix it.

Same issues with go 1.11.5, 1.11.8, 1.12.3 in Docker (golang:1.12.3), and installed locally on Ubuntu.

Local git:

➜   git --version 
git version 2.19.1

I also less frequently get this issue:

go: cloud.google.com/[email protected]: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 1:
        fatal: bad object 3fdddcac5af93820a98679aa0008db7413a13265
        error: https://code.googlesource.com/gocloud did not send all necessary objects

EDIT:

It appears that this patch resolves the issue for my builds:

--- a/packages/go/packagename/go.mod
+++ b/packages/go/packagename/go.mod
@@ -7,7 +7,7 @@ require (
        redacted/logging v0.0.0
        golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53 // indirect
        golang.org/x/oauth2 v0.0.0-20190319182350-c85d3e98c914 // indirect
-       google.golang.org/api v0.3.1
+       google.golang.org/api  v0.0.0-20180603000442-8e296ef26005
        google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
        google.golang.org/grpc v1.19.1
 )

[jayconrod]

Reproduced locally with the same error.

Also seen with the same instructions as above:

go: golang.org/x/[email protected]: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/tmp.yf9bsDvK/pkg/mod/cache/vcs/abc0aaaae9fbb1a906994bfb84d444789fc395ac48e2b740ecbbfeebd3047a32: exit status 128:
	fatal: remote error: authentication error

Perhaps Gerrit is acting up again?

cc @heschik @andybons @hakim @dmitshur

[jayconrod]

I've traced this to a sequence of Git commands that reproduces this issue outside of Go. The Gerrit team has been notified and is working to resolve the issue.

[mr-tron]

Is it some workaround for the problem?

[heschi]

The latest we have from the Gerrit team is that this might be a bug specific to the content of the google.golang.org/api repository. As suggested above, a git fetch should work around it. I think fetching a different version might help too.

[mr-tron]

Yes. Second running also helps, but it very ugly hack. I added vendor dir to my projects. It looks like better workaround.

[draftcode]

Hello, this is the team that manages googlesource.com. *.googlesource.com is hosted by using JGit, and we believe that there's a bug in upstream JGit that cannot handle unshallow git-fetch. We recognize that this is reproducible easily, but we cannot spot the buggy code in the JGit. If there's an option to do a full clone always instead of partial clone, that can be a workaround.

[draftcode]

We identified the root cause and the fix is uploaded at https://git.eclipse.org/r/c/140463/. Our in-house JGit comitters are already gone home, so this will be reviewed tomorrow at best. If everything goes well and our SREs are happy about a Friday release, we might be able to cherry-pick this fix tomorrow. The worst case ETA would be next week.

[jrn]

https://git.eclipse.org/r/c/140463/

That did the trick. If it happens again, we want to know.

[chmking]

@jrn @draftcode

Everything was working this morning but I've seen a regression in the past 10 mins:

12-Apr-2019 13:21:54 | go: google.golang.org/[email protected]: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
12-Apr-2019 13:21:54 | fatal: bad object 19e022d8cf43ce81f046bae8cc18c5397cc7732f
12-Apr-2019 13:21:54 | error: https://code.googlesource.com/google-api-go-client did not send all necessary objects

12-Apr-2019 13:27:12 | verifying google.golang.org/[email protected]/go.mod: checksum mismatch
12-Apr-2019 13:27:12 | downloaded: h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
12-Apr-2019 13:27:12 | go.sum:     h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=

[mr-tron]

Still doesnt work for me.

[jayconrod]

I've tried to reproduce this again, and it's now working for me.

@chmking @mr-tron Are you able to reproduce this with a clean module cache using the original instructions at the top of the issue? Could you post the output of go install -x go.chromium.org/goma/server/cmd/remoteexec_proxy with the failure? The -x flag should show the git commands run in each vcs cache, so that will help us isolate the issue.

[chmking]

@jayconrod This is happening on our CI machines running the build in a docker without a go mod cache. It worked this morning without fail many times until it started failing at the time of my post. Now the issues is intermittent.

I can try to replicate it for you later.

[draveness]

I met the same problem when using go get kubernetes/client-go#590

$ go get -u -v k8s.io/client-go
Fetching https://cloud.google.com/go?go-get=1
Parsing meta tags from https://cloud.google.com/go?go-get=1 (status code 200)
get "cloud.google.com/go": found meta tag get.metaImport{Prefix:"cloud.google.com/go", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://cloud.google.com/go?go-get=1
go: finding cloud.google.com/go v0.0.0-20160913182117-3b1ae45394a2
go: cloud.google.com/[email protected]: git -c protocol.version=0 fetch --unshallow -f https://code.googlesource.com/gocloud refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/draveness/go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 128:
	fatal: bad object a5913b3f7deecba45e98ff33cefbac4fd204ddd7
	error: remote did not send all necessary objects
go: error loading module requirements

$ git --version
git version 2.21.0

[hazcod]

Is there a workaround for go get? I am unable to develop.
Edit: I skipped over the workaround.

[agnivade]

Please see the first comment for the workaround.

[draftcode]

Reproduced with JGit:

git init . --bare
git remote add origin git://localhost:9418/google-api-go-client.git
git fetch -f --depth=1 origin refs/tags/v0.3.2:refs/tags/v0.3.2
git fetch -f --depth=1 origin refs/tags/v0.1.0:refs/tags/v0.1.0
git fetch -f --depth=1 origin refs/tags/v0.3.1:refs/tags/v0.3.1
git fetch -f --unshallow origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'

[draftcode]

I found another bug in JGit code that made it exclude dcbf3e3c23816703a8f5ebfe6583cc3fc2793526. But after I fix that, I found another bug that doesn't send a parent of v0.1.0. I'm debugging JGit more.

[draftcode]

https://git.eclipse.org/r/c/140900/ and https://git.eclipse.org/r/c/140901/ will fix the the three tag unshallow case. Tried all three previously reproduced cases.

[iamtakingiteasy]

A possible temporary workaround for docker builds might be

RUN ( \
      export GO111MODULE=on && cd / && go get google.golang.org/api && \
      cd "$(grep -l https://code.googlesource.com/google-api-go-client $GOPATH/pkg/mod/cache/vcs/*.info | sed 's/\.info$//')" && \
      git fetch -f https://code.googlesource.com/google-api-go-client 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*' \
    )

[jayconrod]

Still seems to be happening. Pinged internal bug.

[draftcode]

#31399 (comment) is not rolled out yet.

[jayconrod]

Oops, I thought those CLs had been deployed already.

@draftcode Do you expect this will fix the GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy command from the original example?

[anasanzari]

@iamtakingiteasy not working anymore

[iamtakingiteasy]

@anasanzari true, but it seems original issue of fetching https://code.googlesource.com/google-api-go-client from go mod tools was somehow fixed despite commits not being merged (or at least I couldn't find a way to reproduce), this workaround is not needed anymore.

You can remove it from your pipeline and build as normal if that was your use-case.

Also it is indeed odd such fetch would not work. Might be somewhat related issue (which those commits might fix).

[jayconrod]

All fixes are rolled out now, and this seems to be working. Thanks @draftcode for working on this!

I'll close this but everyone please comment if you're still seeing this, and we'll investigate.

[anasanzari]

@jayconrod issue is still there. I've added my case in https://github.com/anasanzari/golang_build_check.
docker build . fails with the same error.

[bradfitz]

@anasanzari, thanks. I can reproduce the failure.

Gerrit/jgit is still broken:

go: google.golang.org/[email protected]: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: missing tree object 'dcbf3e3c23816703a8f5ebfe6583cc3fc2793526'
        error: remote did not send all necessary objects
go: error loading module requirements

Reopening.

[bradfitz]

@draftcode, please investigate?

[draftcode]

I can reproduce this, but I cannot reproduce this with JGit.

[anasanzari]

can someone give a quick fix for this?

[tparker]

The reproduction case above only failed in production-like environments, so it took longer to track down. The next fix forgoes potential data transfer optimizations in the name of correctness. Unshallow operations will be full clones. We will work on reintroducing the optimizations but that is a multi-week project that won't begin immediately.

The patch is in our release process, building and verifying. It should start rolling it out in a few hours.

[tparker]

Sorry to say we hit a snag in the release process. We are working through it and will have a release ready later this afternoon. But releasing on a Friday afternoon is...dicey. We will wait until Monday to update the servers.

Good news is that the fix bypasses all the finicky calculations that have caused this problem to recur, and we have verified it in a production-like environment.

[tparker]

This was fully deployed by yesterday afternoon, PDT.

[jayconrod]

Thanks @tparker and @draftcode for working on this. I'll close this issue now.

If anyone is still seeing this, please let us know.

[anjmao]

@jayconrod Hi, I still see this issue:

go get -y github.com/jinzhu/gorm

I get error:

go: golang.org/x/[email protected]: git fetch -f https://go.googlesource.com/net refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/anjmao/go/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 1:
	error: Could not read 65e2d4e15006aab9813ff8769e768bbf4bb667a0
	fatal: bad object 65e2d4e15006aab9813ff8769e768bbf4bb667a0
	error: https://go.googlesource.com/net did not send all necessary objects

[tparker]

Does anyone have a full reproduction recipe? I'm unable to reproduce it.

[jayconrod]

@anjmao I couldn't reproduce this. Are you seeing this consistently? Does this happen with an empty module cache (i.e., export GOPATH=$(mktemp -d))? Anything unusual about your environment (git config, git version, proxy settings)?

If you can give us a sequence of git commands that reproduces this (i.e., from go get -x), that would be very helpful.