Closed
Description
The "net" subrepo has stopped mirroring to github.
Note: https://github.com/golang/net/commits/master (stuck at Jul 7, 2016, f841c3)
Versus: https://go.googlesource.com/net (3 new commits since then, with e90d6d0 currently at top)
The watcher says:
http://farmer.golang.org/debug/watcher
! [remote rejected] ---40hexomitted--- -> ---40hexomitted--- (pre-receive hook declined)
error: failed to push some refs to 'https://gopherbot:---40hexomitted---@github.com/golang/net'
remote: error: GH002: Sorry, branch or tag names consisting of 40 hex characters are not allowed.
remote: error: Invalid branch or tag name "---40hexomitted---"
So Github doesn't accept some ref that Gerrit has?
/cc @spearce @adg @broady @quentinmit
Related: #11811
Metadata
Metadata
Assignees
Type
Projects
Relationships
Development
No branches or pull requests
Participants
Activity
bradfitz commentedon Jul 15, 2016
In Gerrit for the "net" repo, I see only:
... and no tags.
bradfitz commentedon Jul 15, 2016
But on the watcher's git cache, I see:
I don't know what created that head.
It seems to be the only screwed up one:
Did somebody accidentally create that recently?
quentinmit commentedon Jul 15, 2016
@aclements, @cherrymui and I have been looking at it for the past hour. We removed that branch about an hour ago; no idea when it was created but it pointed at a commit from you in February.
bradfitz commentedon Jul 15, 2016
I've nuked the watcher's "net" git cache and restarted the watcher, which re-cloned it, and github & the build dashboard are happy again.
quentinmit commentedon Jul 15, 2016
The build dashboard certainly doesn't look happy to me; the latest commit on master is still from July 8th.
The git push of the go repo is taking for-freaking-ever, but I hope when it finally finishes that the watcher will notice the commits since July 8th.
bradfitz commentedon Jul 15, 2016
Going forward, we need monitoring on the watcher (that is #15760)
This bug can be about making sure the watcher's git cache stays in sync better, deleting refs when they're deleted from upstream gerrit.
bradfitz commentedon Jul 15, 2016
It's still building, but it noticed new stuff. The red "fail" went away and was replaced by blank spots it knows it needs to build.
quentinmit commentedon Jul 15, 2016
That's true for the subrepos, but the go repo at the top does not show any blank spots.
bradfitz commentedon Jul 15, 2016
Ah, I was only debugging the "net" repo. I hadn't noticed the main repo.
aclements commentedon Jul 15, 2016
I think I may know what's going on here. If you
git clone --mirror https://go.googlesource.com/go(which I believe is what the watcher does), you'll get 40,370 refs, including every version of every CL on Gerrit. I suspect thesend-packrun bypush --mirroris taking forever because it's processing all of these refs.aclements commentedon Jul 15, 2016
Also, it appears these have all been pushed up to GitHub. They don't show up in the UI (presumably because they're not in refs/heads), but if you
git clone --mirror https://github.com/golang/go, you'll see all 40k refs, as well.bradfitz commentedon Jul 15, 2016
@aclements that's what I suspected at first too, but even though we do run git clone --mirror, I only see 330 refs on the git cache:
Ooooh! That's where they're hiding. In packed-refs:
Okay, mystery solved.
We should probably pusher smarter, and prioritize non-change refs, only pushing code review refs when we're otherwise idle.
7 remaining items
bradfitz commentedon Jul 20, 2016
Ping @quentinmit. This appears to be slow again today and I need to decide how to fix it. You say modern git is still slow with many refs, but my tests seemed fine. What did you run?
In any case, I'll start with adding more visibility into what it's doing.
quentinmit commentedon Jul 20, 2016
@bradfitz Here is my repro with git 2.8.0.rc3.226.g39d4020:
(Obviously I killed it after 1.5 minutes; I don't know how long it would have taken.)
Note that I did not clone with --mirror, so the repo does have slightly different refs. Perhaps that's the difference.
bradfitz commentedon Jul 20, 2016
I assume there was a "cd go" in between those lines?
In any case, such a git push is 2 seconds for me, with git 1.9.1, when my current directory is a bare git checkout (from git clone --mirror).
quentinmit commentedon Jul 20, 2016
Yes, there was, I forgot to copy+paste it.
In fact, from a clone --mirror it takes almost no time, so that explains the difference. I think our time would be best spent upgrading git in go-watcher-world.
bradfitz commentedon Jul 20, 2016
Good to hear.
I'll update go-watcher-world after I deploy a new version with more status page visibility into what's happening. I'd like to see the problem more clearly before fixing it.
bradfitz commentedon Jul 20, 2016
I updated go-watcher-world to sid, which has git 2.8.0, and I still see it sucking with the new status pages:
But running my own watcher-world container on the coordinator with the /var/cache/watcher-git bind-mounted, I timed various operations:
There's no reason it should be slow.
I'm inclined to do the syncing myself as in https://gist.github.com/bradfitz/b64ca2917fbc7a447141c580ec820da5
Which also has the advantage of prioritization and the ability to get more visibility into its state.
[-]build: mirroring of net repo is broken[/-][+]x/build/cmd/coordinator: watcher's git mirroring is slow/flaky[/+]bradfitz commentedon Jul 21, 2016
After debugging with @aclements, I switched to using the ssh transport, and things now look good and very fast:
It seems the
git-transport-httpsis just buggy or inefficient with 40k+ refs.gopherbot commentedon Jul 21, 2016
CL https://golang.org/cl/25110 mentions this issue.