runtime: fatal error: found bad pointer in Go heap (1.19.1)

[mobiusmalware]

I have a parsing program that's been in development for the past few months to pull a variety of information from a large sample set. The current issue is that, at random, I will get a fatal error that will kill the program when running against any given directory:

runtime: pointer 0xc000d8b750 to unused region of span span.base()=0xc000b12000 span.limit=0xc000b13fe0 span.state=1
runtime: found in object at *(0xc00112a040+0x0)
object=0xc00112a040 s.base()=0xc00112a000 s.limit=0xc00112c000 s.spanclass=8 s.elemsize=32 s.state=mSpanInUse
 *(object+0) = 0xc000d8b750 <==
 *(object+8) = 0xc000f52000
 *(object+16) = 0x1000
 *(object+24) = 0x1000
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x76b22e?, 0x6?})
	/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f882d7f9d00 sp=0x7f882d7f9cd0 pc=0x43871d
runtime.badPointer(0x7f8803efa2a8, 0x0?, 0xc00112a040, 0xd590?)
	/usr/local/go/src/runtime/mbitmap.go:373 +0x150 fp=0x7f882d7f9d50 sp=0x7f882d7f9d00 pc=0x415770
runtime.findObject(0xc00112a040?, 0x9d86e0?, 0x40d049?)
	/usr/local/go/src/runtime/mbitmap.go:416 +0xa6 fp=0x7f882d7f9d88 sp=0x7f882d7f9d50 pc=0x415906
runtime.scanobject(0xc00002f740?, 0xc00002f740)
	/usr/local/go/src/runtime/mgcmark.go:1338 +0x190 fp=0x7f882d7f9e20 sp=0x7f882d7f9d88 pc=0x421490
runtime.gcDrain(0xc00002f740, 0x2)
	/usr/local/go/src/runtime/mgcmark.go:1103 +0x1d4 fp=0x7f882d7f9e80 sp=0x7f882d7f9e20 pc=0x420cd4
runtime.gcBgMarkWorker.func2()
	/usr/local/go/src/runtime/mgc.go:1323 +0x154 fp=0x7f882d7f9ed0 sp=0x7f882d7f9e80 pc=0x41d474
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:492 +0x49 fp=0x7f882d7f9ed8 sp=0x7f882d7f9ed0 pc=0x4695a9

The full error can be seen here: https://go.dev/play/p/5m48pLjUsEK

Additionally, this is not intentionally reproducible. Running my program against the same directory again will not yield the error, and it may go through several thousand or several hundred thousand files before it occurs again. I have multiple logs of this activity.

Some things to note:

• The executable is built with CGO_ENABLED=0 due to compatibility issues with the box it's being run on
• All drives being scanned are mounted via mount -o ro /remote/drive/path /local/drive/path
• There is no use of unsafe or cgo in my code; however, there is a dependency for the parser that has a single use. This is located in https://github.com/edsrzf/mmap-go/blob/main/mmap.go (lines 80-82):

func (m *MMap) header() *reflect.SliceHeader {
	return (*reflect.SliceHeader)(unsafe.Pointer(m))
}

• This has not been an issue previous to the last week of testing, but I have not made any changes to the dependency or parser code in that time.

What could be causing this to occur, and is there a way to correct for it?

go version: 1.19.1 linux/amd64
OS: Ubuntu 22.04.1 LTS
Actions: Ran parsing program against mounted drives (which are also running Linux)
Expected: No pointer failures
Observed: Random pointer failures

[mdempsky]

/cc @golang/runtime

FWIW, I spot checked the mentioned use of unsafe.Pointer in mmap-go/blob/main/mmap.go and it seems okay. As far as I can tell, it's only used on Windows anyway.

[randall77]

Have you tried running it under the race detector?

[mobiusmalware]

@randall77 I've tried with the race detector and had no failures, even against files that were parsed immediately before the pointer failure. That also required me to rebuild the executable with CGO enabled, which changed things.

[mknyszek]

Awesome, thanks for confirming. Without a way to reproduce this it's hard to make progress on our end. It could be a real bug in the GC based on the information you shared. Do you have any additional information, or can you point us to some code we could run that at least might reproduce the issue?

[mobiusmalware]

@mknyszek I'd love to be able to point you to some code but I need permission to upload, so I'll have to wait for that. What I do have is a saved callstack from this error that I managed to capture via GoLand while remotely debugging this morning. However, running the program against the files to replicate...failed. It's still humming along, hours later, without issue. Which is the major problem in that I can't just reproduce the error at will.
goroutines.txt
Goroutine 37 is where the pointer issue occurs.