runtime: fpTracebackPCs SIGSEGV

[mgartner]

Go version

go version go1.23.9 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/marcus_cockroachlabs_com/.cache/go-build'
GOENV='/home/marcus_cockroachlabs_com/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/marcus_cockroachlabs_com/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/marcus_cockroachlabs_com/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/marcus_cockroachlabs_com/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.9.linux-amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/marcus_cockroachlabs_com/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.9.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.9'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/marcus_cockroachlabs_com/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/marcus_cockroachlabs_com/workspace/hll_sigsegv/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2563209848=/tmp/go-build -gno-record-gcc-switches'

What did you do?

At Cockroach Labs we observed a few occurrences of segfaults in CockroachDB when we enabled runtime execution traces. The stacktraces pointed to our use of github.com/axiomhq/hyperloglog and its dependency github.com/dgryski/go-metro. I was able to create a minimal reproduction that causes the same segfault fairly regularly using go-metro directly, see: https://github.com/mgartner/hll_sigsegv. go-metro includes an implementation in assembly for amd64—maybe that is relevant.

I noticed similar issues #73748 and #73664, but it's not obvious to me whether or not this is a duplicate of one of those.

What did you see happen?

The README in the reproduction repository includes the stacktraces for the minimized reproduction.

An abbreviated stacktrace from one of the CockroachDB segfaults is below. Note that github.com/axiomhq/hyperloglog.hashFunc({0xc0e718f140?, 0x1a?, 0x40?}) is calling go-metros's Hash64 function, which the minimized reproduction invokes directly.

SIGSEGV: segmentation violation
PC=0x4d7eac m=7 sigcode=1 addr=0xa2aa0343

goroutine 0 gp=0xc000b041c0 m=7 mp=0xc000b00008 [idle]:
runtime.fpTracebackPCs(...)
        GOROOT/src/runtime/tracestack.go:258
runtime.traceStack(0xc000b00008?, 0xc0000e2508?, 0x3f99)
        GOROOT/src/runtime/tracestack.go:116 +0x2ac fp=0x7ea5c41fc250 sp=0x7ea5c41fbde8 pc=0x4d7eac
runtime.traceLocker.stack(...)
        GOROOT/src/runtime/traceevent.go:176
runtime.traceLocker.GoStop({0xc000b00008?, 0xc000b00008?}, 0x2)
        GOROOT/src/runtime/traceruntime.go:480 +0x85 fp=0x7ea5c41fc2d8 sp=0x7ea5c41fc250 pc=0x4d7085
runtime.traceLocker.GoPreempt(...)
        GOROOT/src/runtime/traceruntime.go:475
runtime.goschedImpl(0xc011d42e00, 0x1?)
        GOROOT/src/runtime/proc.go:4146 +0x8c fp=0x7ea5c41fc368 sp=0x7ea5c41fc2d8 pc=0x4af78c
runtime.gopreempt_m(0xc011d42e00?)
        GOROOT/src/runtime/proc.go:4182 +0x18 fp=0x7ea5c41fc388 sp=0x7ea5c41fc368 pc=0x4b0278
runtime.mcall()
        src/runtime/asm_amd64.s:459 +0x4e fp=0x7ea5c41fc3a0 sp=0x7ea5c41fc388 pc=0x4ef90e

goroutine 7996919 gp=0xc011d42e00 m=7 mp=0xc000b00008 [running]:
runtime.asyncPreempt2()
        GOROOT/src/runtime/preempt.go:308 +0x39 fp=0xc1716937d8 sp=0xc1716937b8 pc=0x4a4af9
runtime.asyncPreempt()
        src/runtime/preempt_amd64.s:53 +0xdb fp=0xc171693960 sp=0xc1716937d8 pc=0x4f2edb
github.com/axiomhq/hyperloglog.hashFunc({0xc0e718f140?, 0x1a?, 0x40?})
        external/com_github_axiomhq_hyperloglog/utils.go:45 +0x41 fp=0xc171693998 sp=0xc171693960 pc=0xf213c1
github.com/axiomhq/hyperloglog.(*Sketch).Insert(...)
        external/com_github_axiomhq_hyperloglog/hyperloglog.go:148
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*sketchInfo).addRow(0xc084091e20, {0x84795e8, 0xc100c40720}, {0xc0c030c008, 0x11, 0x11}, {0xc088d7c600, 0x19, 0x30}, 0xc171693d30)
        pkg/sql/rowexec/sampler.go:607 +0x734 fp=0xc171693af8 sp=0xc171693998 pc=0x2d8d9b4
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).mainLoop(0xc039503c08, {0x84795e8, 0xc100c40720}, {0x844bf68, 0xc0c030c388})
        pkg/sql/rowexec/sampler.go:325 +0x1991 fp=0xc171693f20 sp=0xc171693af8 pc=0x2d8bb11
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).Run(0xc039503c08, {0x84795e8?, 0xc100c40720?}, {0x844bf68, 0xc0c030c388})
        pkg/sql/rowexec/sampler.go:234 +0x9e fp=0xc171693f88 sp=0xc171693f20 pc=0x2d8a05e
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.func1(0x24?)
        pkg/sql/flowinfra/flow.go:508 +0x59 fp=0xc171693fc8 sp=0xc171693f88 pc=0x2ee7999
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.gowrap1()
        pkg/sql/flowinfra/flow.go:510 +0x24 fp=0xc171693fe0 sp=0xc171693fc8 pc=0x2ee7904
runtime.goexit({})
        src/runtime/asm_amd64.s:1700 +0x1 fp=0xc171693fe8 sp=0xc171693fe0 pc=0x4f1941
created by github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal in goroutine 7996916
        pkg/sql/flowinfra/flow.go:507 +0x392

What did you expect to see?

No SIGSEGV.

[nsrip-dd]

See #69629 (comment). If it is indeed due to go-metro, the issue should be fixed in dgryski/go-metro@edb8663. I gave your reproducer a try and it doesn't seem to crash after upgrading to that commit.

[mgartner]

Thanks for the pointers @nsrip-dd! That fix works for me too. I'll close this.