runtime: memory corruption crashes since Go 1.9

[tedli]

Go version

go version go1.21.11 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.11'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build747085668=/tmp/go-build -gno-record-gcc-switches'

What did you do?

A bare metal kubernetes cluster, runs for over half a year. Joined some new nodes into this cluster.
Found docker daemon on one new joined node hangs.
By restarting the docker daemon, it runs again, but after hours, it will hang again.
Built the same version docker daemon with the same golang toolchain as the official binary release (golang 1.21) with debug turing on, replaced the dockerd with the debug version.
Repeat restarting docker, hangs, restart, many times. Dumping the calling stack when hangs, it varies, and occasionally it panics (runtime, not app).
Trying to use the latest golang release (1.23.1), got no luck, nor did the latest docker release.
With godebug gctrace 1, when it hangs, the gc also stopped logging.

By searching the panic messages and the calling stack, found the some issues, but almost all of those are closed due to age.

#15658
this issue gave a reproduce code, and the code can stably reproduce on this node (golang 1.21, cgo off), and by setting GOMAXPROCS=1, the reproduce code no to crash any more.
Turn to use golang 1.9.3, which commented in this issue that includes a fix, to build the reproduce code, and with 1.9.3 built binary, it no to crash any more (without GOMAXPROCS=1).

#20427

go/src/runtime/sys_linux_amd64.s

Lines 222 to 229 in b521ebb

	// func nanotime1() int64
	TEXT runtime·nanotime1(SB),NOSPLIT,$16-8
	// We don't know how much stack space the VDSO code will need,
	// so switch to g0.
	// In particular, a kernel configured with CONFIG_OPTIMIZE_INLINING=n
	// and hardening can use a full page of stack space in gettime_sym
	// due to stack probes inserted to avoid stack/heap collisions.
	// See issue #20427.

torvalds/linux@889b3c1#diff-c1a25be6ec9efccf08bb1dd54dd545b0ce4a12f6fc1aba602a78bff5a016a8a4L141

linux removed the CONFIG_OPTIMIZE_INLINING option since 5.4. I tried to follow this manual to rebuild the kernel by hardcoding the inline marco to always_inline (CONFIG_OPTIMIZE_INLINING=no), replace the always_inline kernel, go no luck. But reproduce code seemed live longer, without always inline, reproduce code crash within 10 seconds, it can live up to one minute with always inline. still the 1.9.3 built one never crash.

The poor wretch node is in same specs with others, and was setup using the same ansible script. A full memtest86+ is done shown all passed.
Other nodes works as expect, without any touch on any binary.

One thing that, these nodes is in an awful data center and lack of maintenance, thermal issue, dusts made troubles before on other nodes. But it's not seemed like a hardware issue, since it's only breaks golang programs, I can still ssh to login to do operations, the rest of all system components also works as expect.

What did you see happen?

Reproduce code in #15658, can stably reproduce on my machine.

What did you expect to see?

The reproduce code should not reproduce any more, as it fixed since 1.9.3.

[gabyhelp]

Related Issues and Documentation

• runtime: memory corruption crashes with os/exec on Linux kernel 4.4 #20427 (closed)
• runtime: sweep increased allocation count crash on arm64 #36101 (closed)
• runtime: r2 corrupted by mid-stack inlining on ppc64le #30283 (closed)
• runtime: bad p & random crashes since 1.14 on linux 3.18.11 #40722
• runtime: sporadic crashes with CONFIG_NO_HZ_FULL=y enabled (usually during cgo compilation) #18067 (closed)
• runtime: runtime crashes, reproducer same as #20427 #36371 (closed)
• affected/package: #58593 (closed)
• runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 #69255
• runtime: panic: non-empty mark queue after concurrent mark (Go1.14, Go1.15) #41303 (closed)
• runtime: crashes with "runtime: traceback stuck" #62086

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

CC @golang/runtime

I'm sorry, I'm having trouble understanding exactly what you are describing.

Is there a way that we can reproduce the bug ourselves? Please give exact instructions. For example, exactly which code in #15658 are you using?

Note that Go 1.21 is no longer supported. You said you had "no luck" with Go 1.23; what exactly does that mean?

Thanks.

[tedli]

Hi @ianlancetaylor ,
Thanks for reply. I didn't express well.

Is there a way that we can reproduce the bug ourselves?

No, this only reproduce on the specific node.

exactly which code in #15658 are you using?

package main

/* stdlib includes */
import (
        "fmt"
        "os/exec"
)

func run(done chan struct{}) {
        cmd := exec.Command("true")
        if err := cmd.Start(); err != nil {
                goto finished
        }

        cmd.Wait()

finished:
        done <- struct{}{}
        return
}

func main() {
        fmt.Println("Starting a bunch of goroutines...")

        // 8 & 16 are arbitrary
        done := make(chan struct{}, 16)

        for i := 0; i < 8; i++ {
                go run(done)
        }

        for {
                select {
                case <-done:
                        go run(done)
                }
        }
}

had "no luck" with Go 1.23; what exactly does that mean?

I built docker using golang 1.23 myself, the official docker binary release (26.1.4) was built by golang 1.21. By building with 1.23, dockerd still hangs.

The above pasted reproduce code, I tried build with golang 1.9.3, 1.21.11, 1.23.1, 1.23.2, all CGO_ENABLED=0.

	1.9.3	1.21.11	1.23.1	1.23.2	1.10.8	1.11.0	1.11.12	1.12.17
GOMAXPROCS=1	OK	OK	OK	OK	OK	OK	OK	OK
unset GOMAXPROCS	OK	crash	crash	crash	OK	crash	crash	crash
on other machine	OK	OK	OK	OK	OK	OK	OK	OK

Here is one crash

root@node-81:/tmp# ./reproduce.1.21.11.elf 
Starting a bunch of goroutines...
runtime: want=4 got=0
fatal error: limiterEvent.stop: found wrong event in p's limiter event slot

runtime stack:
runtime.throw({0x4cbd49?, 0xc000237f28?})
	/usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc000237ee8 sp=0xc000237eb8 pc=0x433a3c
runtime.(*limiterEvent).stop(0x1325c?, 0x4, 0x9?)
	/usr/local/go/src/runtime/mgclimit.go:454 +0x14a fp=0xc000237f10 sp=0xc000237ee8 pc=0x41b4ca
runtime.pidleget(0x47547daa9856?)
	/usr/local/go/src/runtime/proc.go:6131 +0x12f fp=0xc000237f48 sp=0xc000237f10 pc=0x442aaf
runtime.stopTheWorldWithSema(0x5c?)
	/usr/local/go/src/runtime/proc.go:1387 +0x16b fp=0xc000237fb0 sp=0xc000237f48 pc=0x438bab
runtime.gcStart.func1()
	/usr/local/go/src/runtime/mgc.go:684 +0x18 fp=0xc000237fc8 sp=0xc000237fb0 pc=0x45d138
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0xc000237fd8 sp=0xc000237fc8 pc=0x4611ea

goroutine 42171 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000918a18 sp=0xc000918a08 pc=0x461188
runtime.gcStart({0x7f1d2e9b0878?, 0x4f?, 0xd8?})
	/usr/local/go/src/runtime/mgc.go:684 +0x2c5 fp=0xc000918ab0 sp=0xc000918a18 pc=0x418805
runtime.mallocgc(0x886, 0x4aa4c0, 0x1)
	/usr/local/go/src/runtime/malloc.go:1246 +0x76f fp=0xc000918b18 sp=0xc000918ab0 pc=0x40ca8f
runtime.makeslice(0xc0000120a0?, 0xc?, 0x0?)
	/usr/local/go/src/runtime/slice.go:103 +0x49 fp=0xc000918b40 sp=0xc000918b18 pc=0x44aae9
syscall.SlicePtrFromStrings({0xc0006116c0, 0x1a, 0xc000918bf0?})
	/usr/local/go/src/syscall/exec_unix.go:93 +0xef fp=0xc000918bb8 sp=0xc000918b40 pc=0x47a26f
syscall.forkExec({0xc000650b80?, 0x44b0ef?}, {0xc0004a6530, 0x1, 0x1}, 0x0?)
	/usr/local/go/src/syscall/exec_unix.go:163 +0x11a fp=0xc000918cd8 sp=0xc000918bb8 pc=0x47a59a
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:251
os.startProcess({0xc000650b80, 0xd}, {0xc0004a6530, 0x1, 0x1}, 0xc000918f18)
	/usr/local/go/src/os/exec_posix.go:54 +0x312 fp=0xc000918db8 sp=0xc000918cd8 pc=0x48b852
os.StartProcess({0xc000650b80, 0xd}, {0xc0004a6530, 0x1, 0x1}, 0x1ed00000002?)
	/usr/local/go/src/os/exec.go:111 +0x54 fp=0xc000918e00 sp=0xc000918db8 pc=0x48b474
os/exec.(*Cmd).Start(0xc00036be40)
	/usr/local/go/src/os/exec/exec.go:693 +0x5e5 fp=0xc000918f78 sp=0xc000918e00 pc=0x49dc85
main.run(0x0?)
	/host/tmp/main.go:11 +0x35 fp=0xc000918fc8 sp=0xc000918f78 pc=0x4a0255
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc000918fe0 sp=0xc000918fc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000918fe8 sp=0xc000918fe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 1 [chan receive]:
runtime.gopark(0xc000196eb0?, 0x40c925?, 0xc0?, 0x45?, 0x10?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000196e48 sp=0xc000196e28 pc=0x43686e
runtime.chanrecv(0xc00018c0c0, 0x0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000196ec0 sp=0xc000196e48 pc=0x40650d
runtime.chanrecv1(0xc0007983d0?, 0x10?)
	/usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000196ee8 sp=0xc000196ec0 pc=0x406132
main.main()
	/host/tmp/main.go:34 +0xd9 fp=0xc000196f40 sp=0xc000196ee8 pc=0x4a0379
runtime.main()
	/usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc000196fe0 sp=0xc000196f40 pc=0x43641b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000196fe8 sp=0xc000196fe0 pc=0x462fe1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000106fa8 sp=0xc000106f88 pc=0x43686e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000106fe0 sp=0xc000106fa8 pc=0x4366f3
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000106fe8 sp=0xc000106fe0 pc=0x462fe1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000107778 sp=0xc000107758 pc=0x43686e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0001077c8 sp=0xc000107778 pc=0x422fff
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0001077e0 sp=0xc0001077c8 pc=0x418365
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001077e8 sp=0xc0001077e0 pc=0x462fe1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x18cac1?, 0x17d472?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000107f70 sp=0xc000107f50 pc=0x43686e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x56c380)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000107fa0 sp=0xc000107f70 pc=0x420889
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000107fc8 sp=0xc000107fa0 pc=0x420e39
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000107fe0 sp=0xc000107fc8 pc=0x418305
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0x462fe1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 17 [finalizer wait]:
runtime.gopark(0x0?, 0x4cc500?, 0x0?, 0xa0?, 0x2000000020?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000106620 sp=0xc000106600 pc=0x43686e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0001067e0 sp=0xc000106620 pc=0x417387
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001067e8 sp=0xc0001067e0 pc=0x462fe1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 255 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0xc9?, 0x2f?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d8750 sp=0xc0001d8730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d87e0 sp=0xc0001d8750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d87e8 sp=0xc0001d87e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 42185 [syscall]:
syscall.Syscall6(0xc000707a20?, 0xc0006ae4e0?, 0xc000b05e46?, 0xc000b05f68?, 0x49e084?, 0xc0006d8a50?, 0xd?)
	/usr/local/go/src/syscall/syscall_linux.go:91 +0x30 fp=0xc000b05dd8 sp=0xc000b05d50 pc=0x47cf90
os.(*Process).blockUntilWaitable(0xc00056e4e0)
	/usr/local/go/src/os/wait_waitid.go:32 +0x76 fp=0xc000b05eb0 sp=0xc000b05dd8 pc=0x48dc76
os.(*Process).wait(0xc00056e4e0)
	/usr/local/go/src/os/exec_unix.go:22 +0x25 fp=0xc000b05f10 sp=0xc000b05eb0 pc=0x48bbe5
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000707a20)
	/usr/local/go/src/os/exec/exec.go:890 +0x45 fp=0xc000b05f78 sp=0xc000b05f10 pc=0x49ea85
main.run(0x0?)
	/host/tmp/main.go:15 +0x45 fp=0xc000b05fc8 sp=0xc000b05f78 pc=0x4a0265
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc000b05fe0 sp=0xc000b05fc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b05fe8 sp=0xc000b05fe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 292 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc000205ce0?, 0xe8?, 0x3e?, 0xc000103fb8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000103f50 sp=0xc000103f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000103fe0 sp=0xc000103f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000103fe8 sp=0xc000103fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 403 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0x93?, 0xb3?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000436750 sp=0xc000436730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004367e0 sp=0xc000436750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004367e8 sp=0xc0004367e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 402 [GC worker (idle)]:
runtime.gopark(0x47547cddb2cd?, 0x1?, 0x8f?, 0xf9?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000498f50 sp=0xc000498f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000498fe0 sp=0xc000498f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000498fe8 sp=0xc000498fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 286 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc00040f080?, 0xe8?, 0x96?, 0xc0004997b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000499750 sp=0xc000499730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004997e0 sp=0xc000499750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004997e8 sp=0xc0004997e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 254 [GC worker (idle)]:
runtime.gopark(0x47547cdddf75?, 0x1?, 0xcc?, 0x3c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000105750 sp=0xc000105730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001057e0 sp=0xc000105750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001057e8 sp=0xc0001057e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 285 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc0004546e0?, 0xe8?, 0x86?, 0xc0004987b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000498750 sp=0xc000498730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004987e0 sp=0xc000498750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004987e8 sp=0xc0004987e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 173 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc0004d62c0?, 0xe8?, 0x26?, 0xc0004327b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000432750 sp=0xc000432730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004327e0 sp=0xc000432750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004327e8 sp=0xc0004327e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 174 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc00049d600?, 0xe8?, 0x9e?, 0xc000499fb8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000499f50 sp=0xc000499f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000499fe0 sp=0xc000499f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 309 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc000174c60?, 0xe8?, 0x26?, 0xc0001027b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000102750 sp=0xc000102730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001027e0 sp=0xc000102750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001027e8 sp=0xc0001027e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 321 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d4750 sp=0xc0001d4730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d47e0 sp=0xc0001d4750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d47e8 sp=0xc0001d47e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 238 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc0002dc840?, 0xe8?, 0x46?, 0xc0001047b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000104750 sp=0xc000104730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001047e0 sp=0xc000104750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001047e8 sp=0xc0001047e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 310 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000108750 sp=0xc000108730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001087e0 sp=0xc000108750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001087e8 sp=0xc0001087e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 322 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d4f50 sp=0xc0001d4f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d4fe0 sp=0xc0001d4f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d4fe8 sp=0xc0001d4fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 337 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000686750 sp=0xc000686730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006867e0 sp=0xc000686750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006867e8 sp=0xc0006867e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 239 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000682750 sp=0xc000682730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006827e0 sp=0xc000682750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006827e8 sp=0xc0006827e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 323 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d5750 sp=0xc0001d5730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d57e0 sp=0xc0001d5750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d57e8 sp=0xc0001d57e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 338 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000686f50 sp=0xc000686f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000686fe0 sp=0xc000686f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000686fe8 sp=0xc000686fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 324 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d5f50 sp=0xc0001d5f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d5fe0 sp=0xc0001d5f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d5fe8 sp=0xc0001d5fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 175 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc00049d1e0?, 0xe8?, 0xa6?, 0xc00049a7b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049a750 sp=0xc00049a730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049a7e0 sp=0xc00049a750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049a7e8 sp=0xc00049a7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 176 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049af50 sp=0xc00049af30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049afe0 sp=0xc00049af50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049afe8 sp=0xc00049afe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 354 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc0003dc9a0?, 0xe8?, 0x36?, 0xc0004337b8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000433750 sp=0xc000433730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004337e0 sp=0xc000433750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004337e8 sp=0xc0004337e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 325 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d6750 sp=0xc0001d6730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d67e0 sp=0xc0001d6750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d67e8 sp=0xc0001d67e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 369 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049b750 sp=0xc00049b730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049b7e0 sp=0xc00049b750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049b7e8 sp=0xc00049b7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 370 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00049bf50 sp=0xc00049bf30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00049bfe0 sp=0xc00049bf50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 371 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e2750 sp=0xc0004e2730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e27e0 sp=0xc0004e2750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e27e8 sp=0xc0004e27e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 339 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000687750 sp=0xc000687730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006877e0 sp=0xc000687750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006877e8 sp=0xc0006877e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 372 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e2f50 sp=0xc0004e2f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e2fe0 sp=0xc0004e2f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e2fe8 sp=0xc0004e2fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 355 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000433f50 sp=0xc000433f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000433fe0 sp=0xc000433f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000433fe8 sp=0xc000433fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 373 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e3750 sp=0xc0004e3730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e37e0 sp=0xc0004e3750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e37e8 sp=0xc0004e37e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 340 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000687f50 sp=0xc000687f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000687fe0 sp=0xc000687f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000687fe8 sp=0xc000687fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 356 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000434750 sp=0xc000434730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004347e0 sp=0xc000434750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004347e8 sp=0xc0004347e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 374 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e3f50 sp=0xc0004e3f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e3fe0 sp=0xc0004e3f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e3fe8 sp=0xc0004e3fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 341 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000688750 sp=0xc000688730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006887e0 sp=0xc000688750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006887e8 sp=0xc0006887e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 326 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d6f50 sp=0xc0001d6f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d6fe0 sp=0xc0001d6f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d6fe8 sp=0xc0001d6fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 357 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000434f50 sp=0xc000434f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000434fe0 sp=0xc000434f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000434fe8 sp=0xc000434fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 342 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000688f50 sp=0xc000688f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000688fe0 sp=0xc000688f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000688fe8 sp=0xc000688fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 327 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d7750 sp=0xc0001d7730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d77e0 sp=0xc0001d7750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d77e8 sp=0xc0001d77e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 358 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000435750 sp=0xc000435730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004357e0 sp=0xc000435750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004357e8 sp=0xc0004357e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 343 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000689750 sp=0xc000689730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006897e0 sp=0xc000689750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006897e8 sp=0xc0006897e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 344 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000689f50 sp=0xc000689f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000689fe0 sp=0xc000689f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000689fe8 sp=0xc000689fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 359 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000435f50 sp=0xc000435f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000435fe0 sp=0xc000435f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000435fe8 sp=0xc000435fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 360 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004de750 sp=0xc0004de730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004de7e0 sp=0xc0004de750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004de7e8 sp=0xc0004de7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 375 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e4750 sp=0xc0004e4730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e47e0 sp=0xc0004e4750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e47e8 sp=0xc0004e47e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 328 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0001d7f50 sp=0xc0001d7f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0001d7fe0 sp=0xc0001d7f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001d7fe8 sp=0xc0001d7fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 361 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004def50 sp=0xc0004def30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004defe0 sp=0xc0004def50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004defe8 sp=0xc0004defe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 362 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004df750 sp=0xc0004df730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004df7e0 sp=0xc0004df750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004df7e8 sp=0xc0004df7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 363 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004dff50 sp=0xc0004dff30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004dffe0 sp=0xc0004dff50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004dffe8 sp=0xc0004dffe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 364 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e0750 sp=0xc0004e0730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e07e0 sp=0xc0004e0750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e07e8 sp=0xc0004e07e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 365 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e0f50 sp=0xc0004e0f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e0fe0 sp=0xc0004e0f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e0fe8 sp=0xc0004e0fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 287 [GC worker (idle)]:
runtime.gopark(0x0?, 0xc00040ef20?, 0xe8?, 0x6e?, 0xc000436fb8?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000436f50 sp=0xc000436f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000436fe0 sp=0xc000436f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000436fe8 sp=0xc000436fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 329 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000608750 sp=0xc000608730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006087e0 sp=0xc000608750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006087e8 sp=0xc0006087e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 330 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000608f50 sp=0xc000608f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000608fe0 sp=0xc000608f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000608fe8 sp=0xc000608fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 288 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000437750 sp=0xc000437730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004377e0 sp=0xc000437750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004377e8 sp=0xc0004377e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 385 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000437f50 sp=0xc000437f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000437fe0 sp=0xc000437f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000437fe8 sp=0xc000437fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 386 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438750 sp=0xc000438730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004387e0 sp=0xc000438750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004387e8 sp=0xc0004387e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 387 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000438f50 sp=0xc000438f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000438fe0 sp=0xc000438f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000438fe8 sp=0xc000438fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 388 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000439750 sp=0xc000439730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004397e0 sp=0xc000439750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004397e8 sp=0xc0004397e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 389 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000439f50 sp=0xc000439f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000439fe0 sp=0xc000439f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000439fe8 sp=0xc000439fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 390 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000604750 sp=0xc000604730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006047e0 sp=0xc000604750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006047e8 sp=0xc0006047e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 376 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e4f50 sp=0xc0004e4f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e4fe0 sp=0xc0004e4f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e4fe8 sp=0xc0004e4fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 331 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000609750 sp=0xc000609730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006097e0 sp=0xc000609750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006097e8 sp=0xc0006097e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 391 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000604f50 sp=0xc000604f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000604fe0 sp=0xc000604f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000604fe8 sp=0xc000604fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 332 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000609f50 sp=0xc000609f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000609fe0 sp=0xc000609f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000609fe8 sp=0xc000609fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 345 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000690750 sp=0xc000690730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006907e0 sp=0xc000690750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006907e8 sp=0xc0006907e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 346 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000690f50 sp=0xc000690f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000690fe0 sp=0xc000690f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000690fe8 sp=0xc000690fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 366 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e1750 sp=0xc0004e1730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e17e0 sp=0xc0004e1750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e17e8 sp=0xc0004e17e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 347 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000691750 sp=0xc000691730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006917e0 sp=0xc000691750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006917e8 sp=0xc0006917e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 348 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000691f50 sp=0xc000691f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000691fe0 sp=0xc000691f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000691fe8 sp=0xc000691fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 377 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e5750 sp=0xc0004e5730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e57e0 sp=0xc0004e5750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e57e8 sp=0xc0004e57e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 349 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000692750 sp=0xc000692730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006927e0 sp=0xc000692750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006927e8 sp=0xc0006927e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 333 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00060a750 sp=0xc00060a730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00060a7e0 sp=0xc00060a750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00060a7e8 sp=0xc00060a7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 378 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e5f50 sp=0xc0004e5f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e5fe0 sp=0xc0004e5f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e5fe8 sp=0xc0004e5fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 350 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000692f50 sp=0xc000692f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000692fe0 sp=0xc000692f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000692fe8 sp=0xc000692fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 334 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00060af50 sp=0xc00060af30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00060afe0 sp=0xc00060af50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00060afe8 sp=0xc00060afe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 367 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0004e1f50 sp=0xc0004e1f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0004e1fe0 sp=0xc0004e1f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e1fe8 sp=0xc0004e1fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 392 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000605750 sp=0xc000605730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006057e0 sp=0xc000605750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006057e8 sp=0xc0006057e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 379 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00068c750 sp=0xc00068c730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00068c7e0 sp=0xc00068c750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00068c7e8 sp=0xc00068c7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 368 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0xe9?, 0x48?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f2750 sp=0xc0003f2730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f27e0 sp=0xc0003f2750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f27e8 sp=0xc0003f27e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 380 [GC worker (idle)]:
runtime.gopark(0x47547cdd7869?, 0x1?, 0xba?, 0xe7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00068cf50 sp=0xc00068cf30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00068cfe0 sp=0xc00068cf50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00068cfe8 sp=0xc00068cfe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 393 [GC worker (idle)]:
runtime.gopark(0x475468105040?, 0x1?, 0x52?, 0x8d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000605f50 sp=0xc000605f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000605fe0 sp=0xc000605f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000605fe8 sp=0xc000605fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 417 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0xf5?, 0xd6?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f2f50 sp=0xc0003f2f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f2fe0 sp=0xc0003f2f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f2fe8 sp=0xc0003f2fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 394 [GC worker (idle)]:
runtime.gopark(0x47546af39f13?, 0x1?, 0x77?, 0x67?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000606750 sp=0xc000606730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006067e0 sp=0xc000606750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006067e8 sp=0xc0006067e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 418 [GC worker (idle)]:
runtime.gopark(0x47547cdda850?, 0x1?, 0x16?, 0x35?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f3750 sp=0xc0003f3730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f37e0 sp=0xc0003f3750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f37e8 sp=0xc0003f37e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 395 [GC worker (idle)]:
runtime.gopark(0x475468103097?, 0x1?, 0x76?, 0xe7?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000606f50 sp=0xc000606f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000606fe0 sp=0xc000606f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000606fe8 sp=0xc000606fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 351 [GC worker (idle)]:
runtime.gopark(0x475406203051?, 0x1?, 0x9b?, 0x8e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000693750 sp=0xc000693730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0006937e0 sp=0xc000693750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0006937e8 sp=0xc0006937e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 335 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0xc1?, 0x26?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00060b750 sp=0xc00060b730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00060b7e0 sp=0xc00060b750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00060b7e8 sp=0xc00060b7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 419 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0x76?, 0x1d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f3f50 sp=0xc0003f3f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f3fe0 sp=0xc0003f3f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f3fe8 sp=0xc0003f3fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 420 [GC worker (idle)]:
runtime.gopark(0x47546810364e?, 0x1?, 0x1e?, 0xb2?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f4750 sp=0xc0003f4730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f47e0 sp=0xc0003f4750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f47e8 sp=0xc0003f47e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 336 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0xb3?, 0x3?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00060bf50 sp=0xc00060bf30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00060bfe0 sp=0xc00060bf50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00060bfe8 sp=0xc00060bfe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 381 [GC worker (idle)]:
runtime.gopark(0x47547cddd84b?, 0x1?, 0x52?, 0x5d?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00068d750 sp=0xc00068d730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00068d7e0 sp=0xc00068d750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00068d7e8 sp=0xc00068d7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 433 [GC worker (idle)]:
runtime.gopark(0x59a860?, 0x1?, 0x48?, 0x3a?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003ee750 sp=0xc0003ee730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003ee7e0 sp=0xc0003ee750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003ee7e8 sp=0xc0003ee7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 382 [GC worker (idle)]:
runtime.gopark(0x47547cdde8ed?, 0x1?, 0xbf?, 0x99?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00068df50 sp=0xc00068df30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00068dfe0 sp=0xc00068df50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00068dfe8 sp=0xc00068dfe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 421 [GC worker (idle)]:
runtime.gopark(0x475468112a0b?, 0x1?, 0x38?, 0x50?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003f4f50 sp=0xc0003f4f30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003f4fe0 sp=0xc0003f4f50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003f4fe8 sp=0xc0003f4fe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 434 [GC worker (idle)]:
runtime.gopark(0x475468103694?, 0x1?, 0xc5?, 0xe5?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003eef50 sp=0xc0003eef30 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003eefe0 sp=0xc0003eef50 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003eefe8 sp=0xc0003eefe0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 435 [GC worker (idle)]:
runtime.gopark(0x47546f487cac?, 0x1?, 0xaf?, 0x26?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003ef750 sp=0xc0003ef730 pc=0x43686e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003ef7e0 sp=0xc0003ef750 pc=0x419ce5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003ef7e8 sp=0xc0003ef7e0 pc=0x462fe1
created by runtime.gcBgMarkStartWorkers in goroutine 308
	/usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 42170 [syscall]:
syscall.Syscall(0x0?, 0x47adc0?, 0x599ef0?, 0xc00072fb90?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25 fp=0xc00072fb88 sp=0xc00072fb18 pc=0x47cf05
syscall.Close(0xc00072fd60?)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:320 +0x25 fp=0xc00072fbb8 sp=0xc00072fb88 pc=0x47c0c5
syscall.forkExec({0xc00035a920?, 0x44b0ef?}, {0xc000720430, 0x1, 0x1}, 0x0?)
	/usr/local/go/src/syscall/exec_unix.go:215 +0x376 fp=0xc00072fcd8 sp=0xc00072fbb8 pc=0x47a7f6
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:251
os.startProcess({0xc00035a920, 0xd}, {0xc000720430, 0x1, 0x1}, 0xc00072ff18)
	/usr/local/go/src/os/exec_posix.go:54 +0x312 fp=0xc00072fdb8 sp=0xc00072fcd8 pc=0x48b852
os.StartProcess({0xc00035a920, 0xd}, {0xc000720430, 0x1, 0x1}, 0x1ed00000002?)
	/usr/local/go/src/os/exec.go:111 +0x54 fp=0xc00072fe00 sp=0xc00072fdb8 pc=0x48b474
os/exec.(*Cmd).Start(0xc000347600)
	/usr/local/go/src/os/exec/exec.go:693 +0x5e5 fp=0xc00072ff78 sp=0xc00072fe00 pc=0x49dc85
main.run(0x0?)
	/host/tmp/main.go:11 +0x35 fp=0xc00072ffc8 sp=0xc00072ff78 pc=0x4a0255
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc00072ffe0 sp=0xc00072ffc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00072ffe8 sp=0xc00072ffe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 42211 [syscall]:
syscall.Syscall6(0xc0002fbce0?, 0xc0001445a0?, 0xc000b09e46?, 0xc000b09f68?, 0x49e084?, 0xc000638c90?, 0xd?)
	/usr/local/go/src/syscall/syscall_linux.go:91 +0x30 fp=0xc000b09dd8 sp=0xc000b09d50 pc=0x47cf90
os.(*Process).blockUntilWaitable(0xc0008386c0)
	/usr/local/go/src/os/wait_waitid.go:32 +0x76 fp=0xc000b09eb0 sp=0xc000b09dd8 pc=0x48dc76
os.(*Process).wait(0xc0008386c0)
	/usr/local/go/src/os/exec_unix.go:22 +0x25 fp=0xc000b09f10 sp=0xc000b09eb0 pc=0x48bbe5
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0002fbce0)
	/usr/local/go/src/os/exec/exec.go:890 +0x45 fp=0xc000b09f78 sp=0xc000b09f10 pc=0x49ea85
main.run(0x0?)
	/host/tmp/main.go:15 +0x45 fp=0xc000b09fc8 sp=0xc000b09f78 pc=0x4a0265
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc000b09fe0 sp=0xc000b09fc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000b09fe8 sp=0xc000b09fe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 41836 [syscall]:
syscall.Syscall(0x0?, 0x47adc0?, 0x599ef0?, 0xc00091fb90?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25 fp=0xc00091fb88 sp=0xc00091fb18 pc=0x47cf05
syscall.Close(0xc00091fd60?)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:320 +0x25 fp=0xc00091fbb8 sp=0xc00091fb88 pc=0x47c0c5
syscall.forkExec({0xc00063c380?, 0x44b0ef?}, {0xc00060e190, 0x1, 0x1}, 0x0?)
	/usr/local/go/src/syscall/exec_unix.go:215 +0x376 fp=0xc00091fcd8 sp=0xc00091fbb8 pc=0x47a7f6
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:251
os.startProcess({0xc00063c380, 0xd}, {0xc00060e190, 0x1, 0x1}, 0xc00091ff18)
	/usr/local/go/src/os/exec_posix.go:54 +0x312 fp=0xc00091fdb8 sp=0xc00091fcd8 pc=0x48b852
os.StartProcess({0xc00063c380, 0xd}, {0xc00060e190, 0x1, 0x1}, 0x1ed00000002?)
	/usr/local/go/src/os/exec.go:111 +0x54 fp=0xc00091fe00 sp=0xc00091fdb8 pc=0x48b474
os/exec.(*Cmd).Start(0xc000596840)
	/usr/local/go/src/os/exec/exec.go:693 +0x5e5 fp=0xc00091ff78 sp=0xc00091fe00 pc=0x49dc85
main.run(0x0?)
	/host/tmp/main.go:11 +0x35 fp=0xc00091ffc8 sp=0xc00091ff78 pc=0x4a0255
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc00091ffe0 sp=0xc00091ffc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00091ffe8 sp=0xc00091ffe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 42158 [syscall]:
syscall.Syscall(0x0?, 0x47adc0?, 0x599ef0?, 0xc000733b90?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25 fp=0xc000733b88 sp=0xc000733b18 pc=0x47cf05
syscall.Close(0xc000733d60?)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:320 +0x25 fp=0xc000733bb8 sp=0xc000733b88 pc=0x47c0c5
syscall.forkExec({0xc0005a4d20?, 0x44b0ef?}, {0xc0006aa5c0, 0x1, 0x1}, 0x0?)
	/usr/local/go/src/syscall/exec_unix.go:215 +0x376 fp=0xc000733cd8 sp=0xc000733bb8 pc=0x47a7f6
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:251
os.startProcess({0xc0005a4d20, 0xd}, {0xc0006aa5c0, 0x1, 0x1}, 0xc000733f18)
	/usr/local/go/src/os/exec_posix.go:54 +0x312 fp=0xc000733db8 sp=0xc000733cd8 pc=0x48b852
os.StartProcess({0xc0005a4d20, 0xd}, {0xc0006aa5c0, 0x1, 0x1}, 0x1ed00000002?)
	/usr/local/go/src/os/exec.go:111 +0x54 fp=0xc000733e00 sp=0xc000733db8 pc=0x48b474
os/exec.(*Cmd).Start(0xc00042e000)
	/usr/local/go/src/os/exec/exec.go:693 +0x5e5 fp=0xc000733f78 sp=0xc000733e00 pc=0x49dc85
main.run(0x0?)
	/host/tmp/main.go:11 +0x35 fp=0xc000733fc8 sp=0xc000733f78 pc=0x4a0255
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc000733fe0 sp=0xc000733fc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000733fe8 sp=0xc000733fe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 42196 [runnable]:
syscall.Syscall6(0xc0003b8000?, 0xc0004ae5a0?, 0xc000197e46?, 0xc000197f68?, 0x49e084?, 0xc0005a8ca0?, 0xd?)
	/usr/local/go/src/syscall/syscall_linux.go:91 +0x30 fp=0xc000197dd8 sp=0xc000197d50 pc=0x47cf90
os.(*Process).blockUntilWaitable(0xc000626840)
	/usr/local/go/src/os/wait_waitid.go:32 +0x76 fp=0xc000197eb0 sp=0xc000197dd8 pc=0x48dc76
os.(*Process).wait(0xc000626840)
	/usr/local/go/src/os/exec_unix.go:22 +0x25 fp=0xc000197f10 sp=0xc000197eb0 pc=0x48bbe5
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0003b8000)
	/usr/local/go/src/os/exec/exec.go:890 +0x45 fp=0xc000197f78 sp=0xc000197f10 pc=0x49ea85
main.run(0x0?)
	/host/tmp/main.go:15 +0x45 fp=0xc000197fc8 sp=0xc000197f78 pc=0x4a0265
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc000197fe0 sp=0xc000197fc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

goroutine 42139 [semacquire]:
runtime.gopark(0xc00091da28?, 0x41fdbc?, 0xc0?, 0x12?, 0x414913?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00091d9e8 sp=0xc00091d9c8 pc=0x43686e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0x56c7b8, 0xeb?, 0x0, 0x0, 0x2?)
	/usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc00091da50 sp=0xc00091d9e8 pc=0x4471f8
runtime.semacquire(...)
	/usr/local/go/src/runtime/sema.go:111
runtime.gcStart({0xc000220000?, 0x500?, 0x4c0?})
	/usr/local/go/src/runtime/mgc.go:629 +0xfe fp=0xc00091dae8 sp=0xc00091da50 pc=0x41863e
runtime.mallocgc(0x4c0, 0x4b7860, 0x1)
	/usr/local/go/src/runtime/malloc.go:1246 +0x76f fp=0xc00091db50 sp=0xc00091dae8 pc=0x40ca8f
runtime.newarray(0x1a0?, 0x1a0?)
	/usr/local/go/src/runtime/malloc.go:1350 +0x45 fp=0xc00091db78 sp=0xc00091db50 pc=0x40cd05
runtime.makeBucketArray(0xc0006d0b60?, 0x0?, 0xc00091dbf0?)
	/usr/local/go/src/runtime/map.go:364 +0x18d fp=0xc00091dbb8 sp=0xc00091db78 pc=0x40daad
runtime.makemap(0x8?, 0x18?, 0x8001?)
	/usr/local/go/src/runtime/map.go:330 +0xe8 fp=0xc00091dc00 sp=0xc00091dbb8 pc=0x40d848
os/exec.dedupEnvCase(0x0, 0x0, {0xc0006d09c0, 0x1a, 0xc0000145e0?})
	/usr/local/go/src/os/exec/exec.go:1228 +0xd7 fp=0xc00091dd78 sp=0xc00091dc00 pc=0x49f317
os/exec.dedupEnv(...)
	/usr/local/go/src/os/exec/exec.go:1217
os/exec.(*Cmd).environ(0xc0002b6160?)
	/usr/local/go/src/os/exec/exec.go:1196 +0x157 fp=0xc00091de00 sp=0xc00091dd78 pc=0x49f1f7
os/exec.(*Cmd).Start(0xc0002b6160)
	/usr/local/go/src/os/exec/exec.go:688 +0x4d6 fp=0xc00091df78 sp=0xc00091de00 pc=0x49db76
main.run(0x0?)
	/host/tmp/main.go:11 +0x35 fp=0xc00091dfc8 sp=0xc00091df78 pc=0x4a0255
main.main.func2()
	/host/tmp/main.go:35 +0x25 fp=0xc00091dfe0 sp=0xc00091dfc8 pc=0x4a03e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00091dfe8 sp=0xc00091dfe0 pc=0x462fe1
created by main.main in goroutine 1
	/host/tmp/main.go:35 +0xcd

[tedli]

I found that the output of lscpu is different from other nodes.

[mknyszek]

What version of Linux are you using?

Also upgrading from Go 1.9.3 to Go 1.21 or Go 1.23 is quite the leap. It should work out of the box, but in between there have been all sorts of problems with certain versions of the Linux kernel.

In Go 1.14, which added asynchronous preemption, we discovered a bug with Linux 5.2 (?) resulting in memory corruption. It was updated in a Linux minor release and we briefly shipped a mitigation that I think we removed later. As of Go 1.14, the runtime sends itself OS signals at a much higher rate than before. Maybe try setting GODEBUG=asyncpreemptoff=1? FWIW, we have not seen any similar reports of crashes like those you have seen on Linux.

[tedli]

Hi @mknyszek ,
Thanks for reply.

Also upgrading from Go 1.9.3 to Go 1.21 or Go 1.23 is quite the leap.

I didn't upgrade go toolchain. Since I didn't have related skills to dive into kernel or go runtime, so I have no choice but searching for crash stack messages. Then I found #20427 , #15658 .
I tried the reproduce code, if it's built with golang 1.9.3, it didn't crash. and after tried with other version of go, I found up to 1.10.8, it won't crash, start with 1.11.0, it crashes.

certain versions of the Linux kernel.

I've tried Linux 6.8.0-45-generic, Linux 5.15.0-122-generic, Linux 5.15.0-25-generic. all be behaves the same.

try setting GODEBUG=asyncpreemptoff=1

Both with and without crashed.

The crash detail

root@node-81:~/reproduce# GODEBUG=asyncpreemptoff=1 ./reproduce.1.14.15.elf 
Starting a bunch of goroutines...
fatal error: slice bounds out of range

runtime stack:
runtime.throw(0x5109d5, 0x19)
	/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.panicCheck1(0x430416, 0x5109d5, 0x19)
	/usr/local/go/src/runtime/panic.go:34 +0xd5
runtime.goPanicSliceAlenU(0x1fffffe7fffe0615, 0x200)
	/usr/local/go/src/runtime/panic.go:101 +0x44
runtime.wbBufFlush1(0xc0000fb800)
	/usr/local/go/src/runtime/mwbbuf.go:247 +0x2a6
runtime.gcMarkDone.func1.1(0xc0000fb800)
	/usr/local/go/src/runtime/mgc.go:1459 +0x2b
runtime.forEachP(0x5151e8)
	/usr/local/go/src/runtime/proc.go:1283 +0x134
runtime.gcMarkDone.func1()
	/usr/local/go/src/runtime/mgc.go:1456 +0x5e
runtime.systemstack(0x2)
	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1056

goroutine 303 [GC worker (idle)]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc000395f20 sp=0xc000395f18 pc=0x464f80
runtime.gcMarkDone()
	/usr/local/go/src/runtime/mgc.go:1449 +0x86 fp=0xc000395f58 sp=0xc000395f20 pc=0x41c666
runtime.gcBgMarkWorker(0xc00003d800)
	/usr/local/go/src/runtime/mgc.go:2000 +0x296 fp=0xc000395fd8 sp=0xc000395f58 pc=0x41d646
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000395fe0 sp=0xc000395fd8 pc=0x466f11
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1821 +0x77

goroutine 1 [chan receive]:
main.main()
	/host/tmp/main.go:34 +0x113

goroutine 132299 [runnable]:
runtime.SetFinalizer(0x4ef3a0, 0xc0005c60c0, 0x4ed820, 0x514fe0)
	/usr/local/go/src/runtime/mfinal.go:384 +0x23c
os.newFile(0x11, 0x50d6dc, 0x9, 0x1, 0x0)
	/usr/local/go/src/os/file_unix.go:170 +0x273
os.openFileNolog(0x50d6dc, 0x9, 0x1, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:226 +0x262
os.OpenFile(0x50d6dc, 0x9, 0x1, 0xc000000000, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:307 +0x97
os/exec.(*Cmd).writerDescriptor(0xc0003ea000, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:291 +0x58b
os/exec.(*Cmd).stdout(0xc0003ea000, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:279 +0x80
os/exec.(*Cmd).Start(0xc0003ea000, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:407 +0x4b8
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132914 [semacquire]:
sync.runtime_SemacquireMutex(0x5edf54, 0x0, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0x5edf50)
	/usr/local/go/src/sync/mutex.go:138 +0x295
sync.(*Mutex).Lock(0x5edf50)
	/usr/local/go/src/sync/mutex.go:81 +0x58
sync.(*RWMutex).Lock(0x5edf50)
	/usr/local/go/src/sync/rwmutex.go:98 +0x38
syscall.forkExec(0xc0007945d0, 0xd, 0xc0004feae0, 0x1, 0x1, 0xc0004afb28, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:193 +0x57c
syscall.StartProcess(0xc0007945d0, 0xd, 0xc0004feae0, 0x1, 0x1, 0xc0004afb28, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:248 +0x98
os.startProcess(0xc0007945d0, 0xd, 0xc0004feae0, 0x1, 0x1, 0xc0004aff20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_posix.go:53 +0x555
os.StartProcess(0xc0007945d0, 0xd, 0xc0004feae0, 0x1, 0x1, 0xc0004aff20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:102 +0xb9
os/exec.(*Cmd).Start(0xc0005f7080, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:422 +0xc5e
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132847 [runnable]:
syscall.Syscall6(0xf7, 0x1, 0x93cc, 0xc000313cb8, 0x1000004, 0x0, 0x0, 0x0, 0xc000313cb8, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc0004d4d80, 0xc00053af00, 0x0, 0x0)
	/usr/local/go/src/os/wait_waitid.go:31 +0xc5
os.(*Process).wait(0xc0004d4d80, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_unix.go:22 +0xce
os.(*Process).Wait(0xc0004d4d80, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:125 +0x51
os/exec.(*Cmd).Wait(0xc0001cf8c0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:507 +0xff
main.run(0xc000144060)
	/host/tmp/main.go:15 +0xb0
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132879 [semacquire]:
sync.runtime_SemacquireMutex(0x5edf54, 0x1, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0x5edf50)
	/usr/local/go/src/sync/mutex.go:138 +0x295
sync.(*Mutex).Lock(0x5edf50)
	/usr/local/go/src/sync/mutex.go:81 +0x58
sync.(*RWMutex).Lock(0x5edf50)
	/usr/local/go/src/sync/rwmutex.go:98 +0x38
syscall.forkExec(0xc000215540, 0xd, 0xc00051e650, 0x1, 0x1, 0xc000159b28, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:193 +0x57c
syscall.StartProcess(0xc000215540, 0xd, 0xc00051e650, 0x1, 0x1, 0xc000159b28, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:248 +0x98
os.startProcess(0xc000215540, 0xd, 0xc00051e650, 0x1, 0x1, 0xc000159f20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_posix.go:53 +0x555
os.StartProcess(0xc000215540, 0xd, 0xc00051e650, 0x1, 0x1, 0xc000159f20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:102 +0xb9
os/exec.(*Cmd).Start(0xc00026d1e0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:422 +0xc5e
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132572 [runnable]:
sync.runtime_Semrelease(0x5edf54, 0x1, 0x1)
	/usr/local/go/src/runtime/sema.go:66 +0x3e
sync.(*Mutex).unlockSlow(0x5edf50, 0x24)
	/usr/local/go/src/sync/mutex.go:224 +0x142
sync.(*Mutex).Unlock(0x5edf50)
	/usr/local/go/src/sync/mutex.go:190 +0x52
sync.(*RWMutex).Unlock(0x5edf50)
	/usr/local/go/src/sync/rwmutex.go:136 +0xd0
syscall.forkExec(0xc00080a040, 0xd, 0xc00077e000, 0x1, 0x1, 0xc000690b28, 0x93cd, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:206 +0x79c
syscall.StartProcess(0xc00080a040, 0xd, 0xc00077e000, 0x1, 0x1, 0xc000690b28, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:248 +0x98
os.startProcess(0xc00080a040, 0xd, 0xc00077e000, 0x1, 0x1, 0xc000387f20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_posix.go:53 +0x555
os.StartProcess(0xc00080a040, 0xd, 0xc00077e000, 0x1, 0x1, 0xc000690f20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:102 +0xb9
os/exec.(*Cmd).Start(0xc000720000, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:422 +0xc5e
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132880 [semacquire]:
sync.runtime_SemacquireMutex(0x5edf54, 0x0, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0x5edf50)
	/usr/local/go/src/sync/mutex.go:138 +0x295
sync.(*Mutex).Lock(0x5edf50)
	/usr/local/go/src/sync/mutex.go:81 +0x58
sync.(*RWMutex).Lock(0x5edf50)
	/usr/local/go/src/sync/rwmutex.go:98 +0x38
syscall.forkExec(0xc0002155e0, 0xd, 0xc00051e680, 0x1, 0x1, 0xc0004aab28, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:193 +0x57c
syscall.StartProcess(0xc0002155e0, 0xd, 0xc00051e680, 0x1, 0x1, 0xc0004aab28, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:248 +0x98
os.startProcess(0xc0002155e0, 0xd, 0xc00051e680, 0x1, 0x1, 0xc0004aaf20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_posix.go:53 +0x555
os.StartProcess(0xc0002155e0, 0xd, 0xc00051e680, 0x1, 0x1, 0xc0004aaf20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:102 +0xb9
os/exec.(*Cmd).Start(0xc00026d340, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:422 +0xc5e
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132799 [syscall]:
syscall.Syscall6(0x101, 0xffffffffffffff9c, 0xc000522040, 0x80000, 0x0, 0x0, 0x0, 0x2, 0x7f999c224df0, 0x10000c000522004)
	/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
syscall.openat(0xffffffffffffff9c, 0x50d6dc, 0x9, 0x80000, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:68 +0x141
syscall.Open(0x50d6dc, 0x9, 0x80000, 0xc000000000, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/syscall_linux.go:138 +0x7f
os.openFileNolog(0x50d6dc, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:200 +0xb9
os.OpenFile(0x50d6dc, 0x9, 0x0, 0x7f9900000000, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:307 +0x97
os.Open(0x50d6dc, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:287 +0x75
os/exec.(*Cmd).stdin(0xc000653e40, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:246 +0x576
os/exec.(*Cmd).Start(0xc000653e40, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:407 +0x4b8
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

goroutine 132848 [semacquire]:
sync.runtime_SemacquireMutex(0x5edf54, 0x0, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0x5edf50)
	/usr/local/go/src/sync/mutex.go:138 +0x295
sync.(*Mutex).Lock(0x5edf50)
	/usr/local/go/src/sync/mutex.go:81 +0x58
sync.(*RWMutex).Lock(0x5edf50)
	/usr/local/go/src/sync/rwmutex.go:98 +0x38
syscall.forkExec(0xc0005e1a60, 0xd, 0xc000012870, 0x1, 0x1, 0xc0004adb28, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:193 +0x57c
syscall.StartProcess(0xc0005e1a60, 0xd, 0xc000012870, 0x1, 0x1, 0xc0004adb28, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/exec_unix.go:248 +0x98
os.startProcess(0xc0005e1a60, 0xd, 0xc000012870, 0x1, 0x1, 0xc0004adf20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec_posix.go:53 +0x555
os.StartProcess(0xc0005e1a60, 0xd, 0xc000012870, 0x1, 0x1, 0xc0004adf20, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/exec.go:102 +0xb9
os/exec.(*Cmd).Start(0xc0001cfa20, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:422 +0xc5e
main.run(0xc000144060)
	/host/tmp/main.go:11 +0x63
created by main.main
	/host/tmp/main.go:35 +0x135

[mknyszek]

start with 1.11.0, it crashes.

That's very interesting. Thanks for trying many configurations!

So it must be some change that went into Go 1.11.0 (which also explains why asyncpreemptoff doesn't change anything; that was introduced in Go 1.14). I don't know how easy it is for you to try more things, but if you can, it should be possible to bisect commits between Go 1.10.0 and Go 1.11.0 to pinpoint exactly which change caused the problem.

You can check out Go by cloning git clone https://go.googlesource.com/go and you can build it by doing cd $REPO/src && ./make.bash. Each toolchain build takes only a couple minutes at most on Linux. The go command you need will be in $REPO/bin/go so you can invoke it directly to build or just add it to PATH.

I think the commit range you want to bisect is b2d3d6e676450cc1a5d5a611d3711dce2800bc0d..41e62b8c49d21659b48a95216e3062032285250f.

(If you haven't done bisection with git before, see https://git-scm.com/docs/git-bisect.)

[tedli]

Hi @mknyszek ,
Thanks for reply.

I'm sorry, I was confused. 1.9.3, 1.10.8 also crashed.
I tried to bisect to find the commit caused the crash.
I ran a reproduce code built by a bisect commit, it crashed just before i'm going to ctrl-z it,
because it ran for about a hour, and I considered the commit is good, so i'm want to ctrl-z, to test next commit,
but it crashed before I typed ctrl-z.
Then I re-tried 1.9.3 and 1.10.8 again, which thought was good previously, leave it ran for a long enough duration (about 2 hours),
then crashed.
So I gave up bisect, since I can't wait forever to make sure a commit won't crash.

GOMAXPROCS=1, strace -f seemed preventable the crash. I will try with GOMAXPROCS=1 or strace with a longer time.
Non-go programs works as expect on this machine.

[tedli]

up to now, longest wait duration, that not crash.

root@node-81:~/reproduce# time ./reproduce.33ce1682c7.elf 
Starting a bunch of goroutines...
^C
real	206m27.422s
user	471m46.162s
sys	131m48.507s

[mknyszek]

I'm sorry, I was confused. 1.9.3, 1.10.8 also crashed.

Hm, given that all the versions you've tested crashed, I'm less inclined to think this is a Go-specific bug. Or at least, I'm thinking it could be a bug in some part of the OS (or system library, or something) that Go is using, but other languages aren't (or are using less often). Unfortunately such bugs are incredibly difficult to track down, and we haven't seen any such other reports elsewhere, so I'm not really sure how to proceed.