x/build: linux-arm builder can't finish an all.bash run when test sharding isn't used

[randall77]

When I run a trybot on linux/arm, I get "out of memory" or "no space left on device" errors.

ok  	reflect	3.259s
ok  	regexp	0.949s
ok  	regexp/syntax	4.910s
# cmd/compile/internal/ssa
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x29ca71, 0x16)
	/workdir/go/src/runtime/panic.go:1116 +0x5c
runtime.sysMap(0x11400000, 0x3000000, 0x42bc70)
	/workdir/go/src/runtime/mem_linux.go:169 +0xa8
runtime.(*linearAlloc).alloc(0x41d280, 0x3000000, 0x400000, 0x42bc70, 0x0)
	/workdir/go/src/runtime/malloc.go:1447 +0x94

ok  	cmd/addr2line	28.924s
ok  	cmd/api	113.949s
ok  	cmd/asm/internal/asm	33.778s
ok  	cmd/asm/internal/lex	0.233s
# cmd/fix.test
panic: no space left on device

goroutine 1 [running]:
cmd/link/internal/ld.Main(0x3f33c0, 0x4, 0x8, 0x1, 0xd, 0xe, 0x0, 0x0, 0x285316, 0x12, ...)
	/workdir/go/src/cmd/link/internal/ld/main.go:319 +0x1c08
main.main()
	/workdir/go/src/cmd/link/main.go:68 +0x12c
/workdir/go/pkg/tool/linux_arm/vet: fork/exec /workdir/go/pkg/tool/linux_arm/vet: cannot allocate memory
/workdir/go/pkg/tool/linux_arm/vet: fork/exec /workdir/go/pkg/tool/linux_arm/vet: cannot allocate memory
/workdir/go/pkg/tool/linux_arm/vet: fork/exec /workdir/go/pkg/tool/linux_arm/vet: cannot allocate memory

Is there anything we can do to fix this? Can we get more memory and/or disk space on these builders?

I could work on making cmd/compile/internal/ssa tests take less memory, perhaps. Not sure how much we could save.

[randall77]

How do the trybots succeed? Is it because they shard out the tests to multiple machines?

[dmitshur]

Can you please include links to CLs where you've seen this? That way we'll have more information (e.g., which commit was being tested exactly, etc.). How often is this happening? Did it start recently?

The linux-arm builder is defined https://github.com/golang/build/blob/148ff27ab5b70970002d390c9e1da4b861f6da9f/dashboard/builders.go#L1736-L1756. They run on Scaleway (also see here), so adjusting resources will be limited to what's available there (we might already be maxed out; but need to look again to be more confident).

I see that linux-arm trybots are currently disabled because of other issues:

tryBot:            nil, // Issue 22748, Issue 22749

Is this issue about that builder when requested via SlowBots or something else?

/cc @cagedmantis @toothrot @andybons per builder owners.

[randall77]

This happens when using gomote to run all.bash manually:

gomote create linux-arm
gomote push user-khr-linux-arm-0
gomote run go/src/all.bash

Sorry, I guess I'm using the term "trybot" to mean both the thing that tests CLs as well as manual gomotes. I mean the latter (except in the context of my second comment).

[dmitshur]

@cagedmantis Do you expect #36841 will be able to help with this (by enabling a linux-arm builder with bigger limits)?

[cagedmantis]

@dmitshur Yes, I'm actively working on the linux-arm-aws builder with more resources. I will assign myself to this issue.

[dmitshur]

Oh, I believe this is the same issue as #35628. /cc @cherrymui I'll close it in favor of that one, and move your assignment @cagedmantis if you don't mind.