runtime: unexpected return pc for golang.gopark

[imcom]

What version of Go are you using (go version)?

$ go version
1.16

Does this issue reproduce with the latest release?

Did not try

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build814691616=/tmp/go-build -gno-record-gcc-switches"
uname -a
Linux USNATGW01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
lsb_release -a
No LSB modules are available.

Distributor ID:	Ubuntu

Description:	Ubuntu 18.04.4 LTS

Release:	18.04

Codename:	bionic

What did you do?

I was running gobgpd:v2.23 in production environment and I have no clue how to troubleshoot or reproduce this issue.
This only happened once in my production environment, no applicable logs when the crash happened

What did you expect to see?

I do not see why there is this unexpected return pc fatal error. My main concern is that since I do not know why it happened, I cannot tell when it will happen again if ever. Also I need help on interpreting the error dump and how to troubleshoot a runtime error like this. Thanks in advance.

What did you see instead?

runtime: unexpected return pc for runtime.gopark called from 0xc00a1946f0
stack: frame={sp:0xc000e694c8, fp:0xc000e694e8} stack=[0xc000e68000,0xc000e6a000)
000000c000e693c8:  0000000000000000  0000000000000000 
000000c000e693d8:  000000c000d3f400  000000000048b4dc  
000000c000e693e8:  000000c0000428d8  000000c000190800 
000000c000e693f8:  0000000000000004  000000c000d3f428 
000000c000e69408:  000000000075e9a7   000000c0000428d8 
000000c000e69418:  0000000000d02820  000000c0042ae000 
000000c000e69428:  000000c000d3f470  000000000075ec4c  
000000c000e69438:  000000c0000428a0  000000c0042ae000 
000000c000e69448:  000000c001497ae0  0000000000000010 
000000c000e69458:  000000c0042ae000  000000c0000428a0 
000000c000e69468:  0000000000d4e9e8  000000c000d3fe58 
000000c000e69478:  0000000000b21f45   000000c001497ae0 
000000c000e69488:  000000c000d3f588  0000000000000001 
000000c000e69498:  0000000000000001  000000c000d3f9c0 
000000c000e694a8:  0000000000000000  0000000000000000 
000000c000e694b8:  0000000000000000  0000000000000000 
000000c000e694c8: <000000c009302408  000000c00483f230 
000000c000e694d8:  000000c007c172d8 !000000c00a1946f0 
000000c000e694e8: >0000000000e238c8  000000c00d9b66d8 
000000c000e694f8:  000000c00a8af8c0  0000000000000013 
000000c000e69508:  0000000000000013  000000c004e5fce0 
000000c000e69518:  000000c0073908b8  000000c00e24e468 
000000c000e69528:  000000c000d3ff10  000000c00045b2c0 
000000c000e69538:  000000c00a079008  000000c000464c40 
000000c000e69548:  000000c00045b2b0  000000c000f7e420 
000000c000e69558:  000000c0074ac7f8  000000c000139968 
000000c000e69568:  000000c0035c1fc8  000000c000caefb0 
000000c000e69578:  0000000000000000  0000000000000000 
000000c000e69588:  0000000000bba9a0  0000000000df9f50 
000000c000e69598:  000000c00dcce078  000000c0030e15d8 
000000c000e695a8:  000000c00d8b3758  000000c009d647e0 
000000c000e695b8:  000000c00901eae0  000000c00f5aff38 
000000c000e695c8:  000000c00853ae88  000000c007897a28 
000000c000e695d8:  000000c00de05e78  000000c011ce25a0 
fatal error: unknown caller pc
fatal error: unexpected signal during runtime execution
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000e20dc0, fp:0xc000e20de0} stack=[0xc000e20800,0xc000e21000)
000000c000e20cc0:  0000000000000000  0000000000000000 
000000c000e20cd0:  0000000000000000  0000000000000000 
000000c000e20ce0:  0000000000000000  0000000000000000 
000000c000e20cf0:  0000000000000000  0000000000000000 
000000c000e20d00:  0000000000000000  0000000000000000 
000000c000e20d10:  0000000000000000  0000000000000000 
000000c000e20d20:  0000000000000000  0000000000000000 
000000c000e20d30:  0000000000000000  0000000000000000 
000000c000e20d40:  0000000000000000  0000000000000000 
000000c000e20d50:  0000000000000000  0000000000000000 
000000c000e20d60:  0000000000000000  0000000000000000 
000000c000e20d70:  0000000000000000  0000000000000000 
000000c000e20d80:  0000000000000000  0000000000000000 
000000c000e20d90:  0000000000000000  0000000000000000 
000000c000e20da0:  0000000000000000  0000000000000000 
000000c000e20db0:  0000000000000000  0000000000000000 
000000c000e20dc0: <0000000000000000  0000000000000000 
000000c000e20dd0:  0000000000000000 !0000000000000000 
000000c000e20de0: >0000000000000000  0000000000000000 
000000c000e20df0:  0000000000000000  0000000000000000 
000000c000e20e00:  0000000000000000  0000000000000000 
000000c000e20e10:  0000000000000000  0000000000000000 
000000c000e20e20:  0000000000000000  0000000000000000 
000000c000e20e30:  0000000000000000  0000000000000000 
000000c000e20e40:  0000000000000000  0000000000000000 
000000c000e20e50:  0000000000000000  0000000000000000 
000000c000e20e60:  0000000000000000  0000000000000000 
000000c000e20e70:  0000000000000000  0000000000000000 
000000c000e20e80:  0000000000000000  0000000000000000 
000000c000e20e90:  0000000000000000  0000000000000000 
000000c000e20ea0:  0000000000000000  0000000000000000 
000000c000e20eb0:  0000000000000000  0000000000000000 
000000c000e20ec0:  0000000000000000  0000000000000000 
000000c000e20ed0:  0000000000000000  0000000000000000 
fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000e5d4b8, fp:0xc000e5d4d8} stack=[0xc000e5c000,0xc000e5e000)
000000c000e5d3b8:  0000000000000000  0000000000000000 
000000c000e5d3c8:  0000000000000000  0000000000000000 
000000c000e5d3d8:  0000000000000000  0000000000000000 
000000c000e5d3e8:  0000000000000000  0000000000000000 
000000c000e5d3f8:  0000000000000000  0000000000000000 
000000c000e5d408:  0000000000000000  0000000000000000 
000000c000e5d418:  0000000000000000  0000000000000000 
000000c000e5d428:  0000000000000000  0000000000000000 
000000c000e5d438:  0000000000000000  0000000000000000 
000000c000e5d448:  0000000000000000  0000000000000000 
000000c000e5d458:  0000000000000000  0000000000000000 
000000c000e5d468:  0000000000000000  0000000000000000 
000000c000e5d478:  0000000000000000  0000000000000000 
000000c000e5d488:  0000000000000000  0000000000000000 
000000c000e5d498:  0000000000000000  0000000000000000 
000000c000e5d4a8:  0000000000000000  0000000000000000 
000000c000e5d4b8: <0000000000000000  0000000000000000 
000000c000e5d4c8:  0000000000000000 !0000000000000000 
000000c000e5d4d8: >0000000000000000  0000000000000000 
000000c000e5d4e8:  0000000000000000  0000000000000000 
000000c000e5d4f8:  0000000000000000  0000000000003023 
000000c000e5d508:  0000000000e23e58  000000c000396600 
000000c000e5d518:  0000000000000000  0000000000e1b280 
000000c000e5d528:  000000c01468d350  000000c0013ec520 
000000c000e5d538:  0000000000000000  000000c000a55920 
000000c000e5d548:  000000c002f53a00  000000000000001f 
000000c000e5d558:  0000000000000000  0000000000000000 
000000c000e5d568:  0000000000000000  0000000000000000 
000000c000e5d578:  000000c012edf680  0000000000e09f40 
000000c000e5d588:  000000c01468d380  000000c000623560 
000000c000e5d598:  000000c000b75700  000000c0102fd140 
000000c000e5d5a8:  0000000000000000  0000000000000000 
000000c000e5d5b8:  

[davecheney]

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

[imcom]

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

yes I've tried using the race-detector and there is no complaints from the tests

Just to make sure I merely need to run go test -race <target pkg> right ? Nothing stood out from the test.

[davecheney]

The race detector can only detect races if your test code exercises them. The litmus test is to pass -race to go build and deploy the application.

[imcom]

@davecheney oh, thanks, that's something I missed out, I will deploy the race-enabled version to see if we can catch anything and report back

[imcom]

Unfortunately ... enabled race detector does not give me more information regarding the issue.

runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000767190, fp:0xc0007671b0} stack=[0xc000766000,0xc000768000)
000000c000767090:  000000c0008947c8  0000000000000018
000000c0007670a0:  00000000011acf00  0000000000000000
000000c0007670b0:  0000000000000000  0000000000000000
000000c0007670c0:  0000000000000000  0000000000000000
000000c0007670d0:  0000000000000000  0000000000000000
000000c0007670e0:  000000c000c46000  0000000000000000
000000c0007670f0:  0000000000000000  0000000000000000
000000c000767100:  0000000000000000  0000000000000000
000000c000767110:  0000010100000000  0000000000000000
000000c000767120:  000000c000aa1b60  0000000000000000
000000c000767130:  000000c000b26180  0000000000000000
000000c000767140:  000000c000480600  0000000000000000
000000c000767150:  0000000000000000  0000000000000000
000000c000767160:  0000000000000000  0000000000000000
000000c000767170:  0000010000000000  0000000000000000
000000c000767180:  0000000000000000  0000000000000000
000000c000767190: <0000000000000000  0000000000000000
000000c0007671a0:  000000c000480600 !0000000000000000
000000c0007671b0: >0000000000000000  0000000000000000
000000c0007671c0:  0000000000000000  0000000000000000
000000c0007671d0:  0000000000000000  0000000000000000
000000c0007671e0:  0000000000000000  0000000000000000
000000c0007671f0:  0000000000000000  0000000000000000
000000c000767200:  000000c000480600  0000000000000000
000000c000767210:  0000000000000000  0000000000000000
000000c000767220:  0000000000000000  0000000000000000
000000c000767230:  0000000000000000  0000000000000000
000000c000767240:  0000000000000000  0000000000000000
000000c000767250:  0000000000000000  0000000000000000
000000c000767260:  000000c00037b500  0000000000000000
000000c000767270:  0000000000000000  000000c000a49f88
000000c000767280:  0000000000000000  0000000000000000
000000c000767290:  0000010100000000  0000000000000000
000000c0007672a0:  000000c000c02de0  0000000000000000
fatal error: unknown caller pc
runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unknown pc in defer 0x1
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x1e
stack: frame={sp:0xc00073f190, fp:0xc00073f1b0} stack=[0x5,0x1056580)

fatal error: unknown caller pc

[networkimprov]

Since you can reproduce the issue, can you provide complete instructions to let others do so?

[imcom]

@networkimprov sorry for the late reply, I cannot really reproduce it , I just keep it running in our Staging environment, and the occurrence seems random to me. In addition to enable race on the build, I've enabled core dump this time, hope I can get more info and share here later. Just my feeling, it could be a runtime issue with memory management

[imcom]

Luckily I caught a coredump this time, but the backtrace is mysterious to me also

(gdb) where
#0  0x0000000000462a81 in runtime.sighandler (sig=0, info=0xc000360180, ctxt=0xc0004e9bf0, gp=0xc0004e9ac0) at /usr/local/go/src/runtime/signal_unix.go:593
#1  0x0000000000000000 in ?? ()
(gdb) bt full
#0  0x0000000000462a81 in runtime.sighandler (sig=0, info=0xc000360180, ctxt=0xc0004e9bf0, gp=0xc0004e9ac0) at /usr/local/go/src/runtime/signal_unix.go:593
        flags = 1902473840
        _g_ = 0x7270697271657270
        c = <optimized out>
        level = <optimized out>
#1  0x0000000000000000 in ?? ()
No symbol table info available.

[imcom]

Corresponding stderr goes like this

runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000767190, fp:0xc0007671b0} stack=[0xc000766000,0xc000768000)
000000c000767090:  000000c0008947c8  0000000000000018
000000c0007670a0:  00000000011acf00  0000000000000000
000000c0007670b0:  0000000000000000  0000000000000000
000000c0007670c0:  0000000000000000  0000000000000000
000000c0007670d0:  0000000000000000  0000000000000000
000000c0007670e0:  000000c000c46000  0000000000000000
000000c0007670f0:  0000000000000000  0000000000000000
000000c000767100:  0000000000000000  0000000000000000
000000c000767110:  0000010100000000  0000000000000000
000000c000767120:  000000c000aa1b60  0000000000000000
000000c000767130:  000000c000b26180  0000000000000000
000000c000767140:  000000c000480600  0000000000000000
000000c000767150:  0000000000000000  0000000000000000
000000c000767160:  0000000000000000  0000000000000000
000000c000767170:  0000010000000000  0000000000000000
000000c000767180:  0000000000000000  0000000000000000
000000c000767190: <0000000000000000  0000000000000000
000000c0007671a0:  000000c000480600 !0000000000000000
000000c0007671b0: >0000000000000000  0000000000000000
000000c0007671c0:  0000000000000000  0000000000000000
000000c0007671d0:  0000000000000000  0000000000000000
000000c0007671e0:  0000000000000000  0000000000000000
000000c0007671f0:  0000000000000000  0000000000000000
000000c000767200:  000000c000480600  0000000000000000
000000c000767210:  0000000000000000  0000000000000000
000000c000767220:  0000000000000000  0000000000000000
000000c000767230:  0000000000000000  0000000000000000
000000c000767240:  0000000000000000  0000000000000000
000000c000767250:  0000000000000000  0000000000000000
000000c000767260:  000000c00037b500  0000000000000000
000000c000767270:  0000000000000000  000000c000a49f88
000000c000767280:  0000000000000000  0000000000000000
000000c000767290:  0000010100000000  0000000000000000
000000c0007672a0:  000000c000c02de0  0000000000000000
fatal error: unknown caller pc
runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unknown pc in defer 0x1
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x1e
stack: frame={sp:0xc00073f190, fp:0xc00073f1b0} stack=[0x5,0x1056580)

fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc0006729f0, fp:0xc000672a10} stack=[0xc000670000,0xc000674000)
000000c0006728f0:  0000000000000000  0000000000000000
000000c000672900:  0000000000000000  0000000000000000
000000c000672910:  0000000000000000  ffffffffffffffff
000000c000672920:  0000000000000000  0000000000000000
000000c000672930:  0000000000000000  000000c00a5eff80
000000c000672940:  0000000000454d12 <runtime.goexit1+114>  000000c000c2c900
000000c000672950:  0000000000000000  0000000000000000
000000c000672960:  0000000000000000  000000c00a5effa8
000000c000672970:  0000000000000000  0000000000503865 <syscall.Syscall+5>
000000c000672980:  000000c00a5effb8  0000000000000000
000000c000672990:  0000000000000006  000000000014fe82
000000c0006729a0:  000000c002159800  0000000000000000
000000c0006729b0:  0000000000000000  0000000001000000
000000c0006729c0:  0000000000000000  0000000000000000
000000c0006729d0:  0000000000000000  0000000000000000
000000c0006729e0:  0000000000000000  0000000000000000
000000c0006729f0: <0000000000000000  0000000000000000
000000c000672a00:  0000000000000000 !0000000000000000
000000c000672a10: >0000000000000000  0000000000f6445e <github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer+126>
000000c000672a20:  0000000000000000  0000000000f94f20 <github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1+0>
000000c000672a30:  00007fec4bfa0008  0000000000000000
000000c000672a40:  0000000000000000  0000000000000000
000000c000672a50:  0000000000000000  0000000000000000
000000c000672a60:  0000000000000000  0000000000000000
000000c000672a70:  0000000000000000  0000000000000000
000000c000672a80:  0000000000000000  0000000000000000
000000c000672a90:  000000c00499cba0  ffffffffffffffff
000000c000672aa0:  0000000000000000  0000000000000000
000000c000672ab0:  0000000000000000  000000c00499cfa0
000000c000672ac0:  0000000000454d12 <runtime.goexit1+114>  000000c000c2ca80
000000c000672ad0:  0000000000000000  0000000000000000
000000c000672ae0:  0000000000000000  000000c00499cfc8
000000c000672af0:  0000000000000000  0000000000503865 <syscall.Syscall+5>
000000c000672b00:  000000c00499cfd8  0000000000000000
fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc00066d190, fp:0xc00066d1b0} stack=[0xc00066c000,0xc00066e000)
000000c00066d090:  000000c0000d07e0  000000c000d60a50
000000c00066d0a0:  c03c52e076eedf23  runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000403190, fp:0xc0004031b0} stack=[0xc000402000,0xc000404000)
000000c000403090:  0000000000000000  0000000000000000
000000c0004030a0:  0000000000000000  0000000000000000
000000c0004030b0:  0000000000000000  0000000000000000
000000c0004030c0:  0000000000000000  0000000000000000
000000c0004030d0:  0000000000000000  0000000000000000
000000c0004030e0:  0000000000000000  0000000000000000
000000c0004030f0:  0000000000000000  0000000000000000
000000c000403100:  0000000000000000  0000000000000000
000000c000403110:  0000000000000000  0000000000000000
000000c000403120:  0000000000000000  0000000000000000
000000c000403130:  0000000000000000  0000000000000000
000000c000403140:  0000000000000000  0000000000000000
000000c000403150:  0000000000000000  0000000000000000
000000c000403160:  0000000000000000  0000000000000000
000000c000403170:  0000000000000000  0000000000000000
000000c000403180:  0000000000000000  0000000000000000
000000c000403190: <0000000000000000  0000000000000000
000000c0004031a0:  0000000000000000 !0000000000000000
000000c0004031b0: >0000000000000000  0000000000000000
000000c0004031c0:  0000000000000000  0000000000000000
000000c0004031d0:  0000000000000000  0000000000000000
000000c0004031e0:  0000000000000000  0000000000000000
000000c0004031f0:  0000000000000000  0000000000000000
000000c000403200:  0000000000000000  0000000000000000
000000c000403210:  0000000000000000  0000000000000000
000000c000403220:  0000000000000000  0000000000000000
000000c000403230:  0000000000571080 <context.WithCancel.func1+0>  000000c000e77a00
000000c000403240:  0000000000000000  0000000000000000
000000c000403250:  0000000000000000  0000000000000000
000000c000403260:  0000000000000000  0000000000000000
000000c000403270:  0000000000000000  0000000000000000
000000c000403280:  0000000000000000  0000000000000000
000000c000403290:  0000000000000000  0000000000000000
000000c0004032a0:  0000000000000000  0000000000000000
runtime: pointer 0xc000959200 to unallocated span span.base()=0xc0000fe000 span.limit=0xc0000fff80 span.state=0
runtime: found in object at *(0xc00037ed00+0x2d0)
object=0xc00037ed00 s.base()=0xc00037c000 s.limit=0xc00037ff00 s.spanclass=78 s.elemsize=2304 s.state=mSpanInUse
 *(object+0) = 0x4
 *(object+8) = 0xc000af6ab0
 *(object+16) = 0x4
 *(object+24) = 0x4
 *(object+32) = 0xc000a697e0
 *(object+40) = 0x4
 *(object+48) = 0x4
 *(object+56) = 0xc000af6ac0
 *(object+64) = 0x4
 *(object+72) = 0x4
 *(object+80) = 0xc000a697e0
 *(object+88) = 0x4
 *(object+96) = 0x4
 *(object+104) = 0xc000af6ad0

[imcom]

Output from dlv

0  0x0000000000462a81 in runtime.sighandler
   at /usr/local/go/src/runtime/signal_unix.go:593
(dlv) ls
> runtime.sighandler() /usr/local/go/src/runtime/signal_unix.go:593 (PC: 0x462a81)
Warning: debugging optimized function
   588:
   589:			c.preparePanic(sig, gp)
   590:			return
   591:		}
   592:
=> 593:		if c.sigcode() == _SI_USER || flags&_SigNotify != 0 {
   594:			if sigsend(sig) {
   595:				return
   596:			}
   597:		}
   598:
(dlv)

[imcom]

perhaps related, I got another stack from a user as below:

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x44694b]

runtime stack:
runtime.throw(0xcd4e13, 0x2a)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/signal_unix.go:718 +0x2e5
runtime.preemptall(0x1200828)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/proc.go:5390 +0x4b
runtime.freezetheworld()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/proc.go:815 +0x6f
runtime.startpanic_m(0x1)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1265 +0x155
runtime.fatalthrow.func1()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1170 +0x3d
runtime.fatalthrow()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1169 +0x57
runtime.throw(0xcd4e13, 0x2a)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/signal_unix.go:718 +0x2e5
runtime.memmove(0xc00010db28, 0xc00048bb28, 0xffffffffffffc4d8)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/memmove_amd64.s:507 +0x65f
runtime.syncadjustsudogs(0xc00018a000, 0x4d8, 0xc000227af8, 0x280001eea4)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/stack.go:821 +0x13f
runtime.copystack(0xc00018a000, 0x2000)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/stack.go:885 +0x165
runtime.shrinkstack(0xc00018a000)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/stack.go:1180 +0x13d
runtime.scanstack(0xc00018a000, 0xc000041698)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgcmark.go:720 +0x58e
runtime.markroot.func1()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgcmark.go:233 +0xc6
runtime.markroot(0xc000041698, 0x1e)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgcmark.go:206 +0x33e
runtime.gcDrain(0xc000041698, 0xb)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgcmark.go:1014 +0x118
runtime.gcBgMarkWorker.func2()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgc.go:2001 +0x19a
runtime.systemstack(0xc000219080)
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/proc.go:1246

goroutine 148 [GC worker (idle)]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0xfa56ea02fa56ea02
stack: frame={sp:0xc00004af58, fp:0xc00004af60} stack=[0xc00004a800,0xc00004b000)
000000c00004ae58:  0000000000000000  0000000000040480
000000c00004ae68:  fa56ea02fa56ea02  2e3232322e393231
000000c00004ae78:  00000034392e3432  0000000000000003
000000c00004ae88:  00000020e571de81  fa56ea02fa56ea02
000000c00004ae98:  fa56ea02fa56ea02  2e3232322e393231
000000c00004aea8:  32332f34392e3432  2e3232322e393231
000000c00004aeb8:  00000034392e3432  0000000000000000
000000c00004aec8:  00000000ffff0000  2e302e300c5cde81
000000c00004aed8:  0000000000302e30  2e302e300e6ade81
000000c00004aee8:  0000000000302e30  0000000000010140
000000c00004aef8:  0000000000000000  2e302e303d55de81
000000c00004af08:  0000000000302e30  0000000000040480
000000c00004af18:  fa56ea02fa56ea02  0000000000010140
000000c00004af28:  0000000000000000  0000000000010140
000000c00004af38:  0000000000000000  0000000000040480
000000c00004af48:  fa56ea02fa56ea02  0000000000040480
000000c00004af58: <fa56ea02fa56ea02 >fa56ea02fa56ea02
000000c00004af68:  fa56ea02fa56ea02  0000000000000000
000000c00004af78:  00000000ffff0000  0000000000010140
000000c00004af88:  0000000000000000  0000000000000003
000000c00004af98:  000000208c40de81  0000000000040480
000000c00004afa8:  fa56ea02fa56ea02  0000000000000000
000000c00004afb8:  00000000ffff0000  fa56ea02fa56ea02
000000c00004afc8:  fa56ea02fa56ea02  2e302e30b213de81
000000c00004afd8:  0000000000302e30  d4820a1202080e0a
000000c00004afe8:  0fd2dbd4820fd2db  2e302e304e24de81
000000c00004aff8:  0000000000302e30
runtime.systemstack_switch()
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/asm_amd64.s:339 fp=0xc00004af60 sp=0xc00004af58 pc=0x46ff60
created by runtime.gcBgMarkStartWorkers
/opt/hostedtoolcache/go/1.16.9/x64/src/runtime/mgc.go:1835 +0x37

goroutine 824637448544 [??? (scan), locked to thread]:
runtime: unknown pc 0xc00038e080
stack: frame={sp:0xc00037da90, fp:0x0} stack=[0xc000303a80,0xc0003975f0)
000000c00037d990:  000000000000003a  0000000000000000
000000c00037d9a0:  0000000000000000  0000000000000000
000000c00037d9b0:  0000000000000000  0000000000000000
000000c00037d9c0:  0000000000000000  0000000000000000
000000c00037d9d0:  0000000000000000  0000000000000000
000000c00037d9e0:  0000000000000000  0000000000000000
000000c00037d9f0:  000000c0002e5350  0000000000000000
000000c00037da00:  000000c00071a540  0000000000000004
000000c00037da10:  0000000000000004  0000000000000000
000000c00037da20:  0000000000000000  0000000000000000
000000c00037da30:  000000001dc4962c  0000000000000000
000000c00037da40:  000000c00083e4b0  0000000000000000
000000c00037da50:  000000c00082dc40  0000000000000004
000000c00037da60:  0000000000000004  0000000000000000
000000c00037da70:  0000000000000000

[imcom]

Feels like starting from go 1.16 the new GC may accidentally corrupt some mem area of parked or running goroutines ?

[ianlancetaylor]

@imcom Anything is possible, but as far as I can tell you are the only person reporting this specific problem. That makes it seem more likely to be due to memory corruption in your program rather than in the garbage collector. Memory corruption could come, for example, from uses of unsafe, or calls to C code, or memory races.

Unfortunately it's going to be hard for us to analyze this if we can't reproduce it. Is it possible for you to share the code with us, even if the problem doesn't happen very often?

[imcom]

Hi @ianlancetaylor! Glad to hear from you. Actually I am not the only one who experienced this, there are 3 ppl that I know of. But indeed, we are working with the same program GoBGP. I am pretty sure there is no CGO involved, but since the project uses reflection so unsafe is there. I tried race-detector enabled build but it did not produce any meaningful dump if any. I am also writing a fuzzing tool on gobgp but so far no crashes happened.

Side note on ppl experienced this, we are not in the same organisation and our use cases/setup are different