x/build: s390x builder not receiving/detecting new builds

[billotosyr]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

master tip (1.12)

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="s390x"
GOBIN=""
GOEXE=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/data/golang/go"
GOTOOLDIR="/data/golang/go/pkg/tool/linux_s390x"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build637051903=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

loaded the build dashboard

What did you expect to see?

s390x building with all the new CL's

What did you see instead?

Since yesterday (Aug. 22) it's been silent. The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?

[billotosyr]

@mundaym

[dmitshur]

I see 23 entries of linux-s390x-ibm under active builds. They're all started anywhere between 51 and 46 hours ago, and stuck on this step:

waiting_machine_in_use 

That might (or might not) be related.

[dmitshur]

The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?

@billotosyr @mundaym Can you try restarting it (the buildlet process)? Perhaps it's running but disconnected, and hasn't detected that.

You can also try updating to the latest version (i.e., go get -u golang.org/x/build/cmd/buildlet).

[billotosyr]

restarted buildlet process - doesn't seem to have had any effect.

[billotosyr]

I'm pretty sure its not a version thing -- as the buildlet was working up untl a benign unrelated CL yesterday ("skip TestGcSys on Windows").
Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

[dmitshur]

Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

Great to hear that it's running again!

Do you know if the machine ran out of space due to the Go builder work, or was it for unrelated reasons?

About the queue, after talking to @bradfitz about it, I don't think there's an existing way to clear it out. Sorry about that. We have an open bug for a scheduler (#19178) and that would be helpful here (e.g., prioritize more recent commits). Let's keep an eye on how the builder runs, and I'll investigate what else can be done to help.

[billotosyr]

Specifically, it's build cache that is filling up the drive. My current workaround is to use
'go clean -cache '
as root, frequently. Perhaps I will have to turn off the build cache.

[bradfitz]

I thought the buildlet wiped its work directory environment per run?

[billotosyr]

Our buildlet doesn't do that. Maybe because it's an older version.

[bradfitz]

No, I was just misremembering. But if that helps your environment & wrapper process that's managing keeping it alive, we could add it to the buildlet.