garbage collector found invalid heap pointer: Go 1.4

[HawaiianSpork]

After upgrading from go1.3 to go1.4, I'd get a runtime crash most of the time. It crashes for various reasons (nil tree walker callback, invalid heap pointer...). Here is a scaled down version of the code that demonstrates the problem (runtime crashes most of the time). Run this code with an argument pointing at a git repository like git2go.

package main

import (
    "fmt"
    "github.com/libgit2/git2go"
    "os"
    "strings"
)

type MyRepo struct {
    *git.Repository
}

func (repository *MyRepo) foo(path string, treeEntry *git.TreeEntry) int {
    if git.ObjectBlob != treeEntry.Type {
        return 0 // continue
    }

    if !strings.HasSuffix(treeEntry.Name, ".go") {
        return 0 // continue
    }

    blob, err := repository.Repository.LookupBlob(treeEntry.Id)
    defer blob.Free()
    if err != nil {
        panic(err)
    }

    if blob.Size() > 10000 {
        return 0 // continue
    }

    contents := blob.Contents()
    if contents == nil {
        panic("nil contents")
    }
    fmt.Printf("%q", path, treeEntry.Name, string(contents))
    fmt.Println()

    return 0 // continue
}

func main() {
    filename := os.Args[1]

    repository, err := git.OpenRepository(filename)
    if err != nil {
        panic(err)
    }
    defer repository.Free()

    myRepo := MyRepo{repository}

    revWalk, err := repository.Walk()
    if err != nil {
        panic(err)
    }
    defer revWalk.Free()

    // Start out at the head
    err = revWalk.PushHead()
    if err != nil {
        panic(err)
    }

    err = revWalk.Iterate(func(commit *git.Commit) bool {
        defer commit.Free()

        tree, err := commit.Tree()
        if err != nil {
            panic(err)
        }
        defer tree.Free()

        err = tree.Walk(myRepo.foo)
        if err != nil {
            panic(err)
        }

        return true
    })

    if err != nil {
        panic(err)
    }
}

Some errors that come from the above code:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x40f4299]

goroutine 1 [running]:
github.com/libgit2/git2go.CallbackGitTreeWalk(0x438a520, 0x51008e0, 0xc208071e40, 0x0)
    /Users/maleticm/.gvm/pkgsets/go1.4/jack/src/github.com/libgit2/git2go/tree.go:99 +0x79
github.com/libgit2/git2go._Cfunc__go_git_treewalk(0x5100850, 0x0, 0xc208069e40, 0x0)
    /Users/maleticm/.gvm/pkgsets/go1.4/jack/src/github.com/libgit2/git2go/:402 +0x43
github.com/libgit2/git2go.Tree.Walk(0x5100850, 0xc20802c018, 0x5100850, 0xc208069e80, 0x0, 0x0)

runtime: garbage collector found invalid heap pointer *(0xc208069cd0+0x10)=0xc208083e40 span=0xc20807e000-0xc208083800-0xc208084000 state=0
fatal error: invalid heap pointer

runtime stack:
runtime.throw(0x437df83)
    /Users/maleticm/.gvm/gos/go1.4/src/runtime/panic.go:491 +0xad fp=0x7fff5fbfea18 sp=0x7fff5fbfe9e8
scanblock(0xc208069cd0, 0x20, 0x42c12e4)
    /Users/maleticm/.gvm/gos/go1.4/src/runtime/mgc0.c:378 +0x551 fp=0x7fff5fbfeb58 sp=0x7fff5fbfea18
scanframe(0x7fff5fbfec60, 0x0, 0x1)
    /Users/maleticm/.gvm/gos/go1.4/src/runtime/mgc0.c:740 +0x1c2 fp=0x7fff5fbfebc8 sp=0x7fff5fbfeb58

unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x40f4299]

goroutine 1 [running, locked to thread]:
runtime.gothrow(0x426ec90, 0x5)
    /Users/maleticm/.gvm/gos/go1.4/src/runtime/panic.go:503 +0x8e fp=0xc208069c40 sp=0xc208069c28
runtime.sigpanic()
    /Users/maleticm/.gvm/gos/go1.4/src/runtime/sigpanic_unix.go:29 +0x261 fp=0xc208069c90 sp=0xc208069c40
github.com/libgit2/git2go.CallbackGitTreeWalk(0x438a520, 0x5009680, 0xc208073e40, 0x0)

[carlosmn]

The newer releases of Go (at least that's where I've noticed it) can make the GC free memory from a variable in the middle of a function if the variable isn't in use in the Go code anymore. It's possible that there are some missing defensive copies.

[carlosmn]

It looks like there's a timing aspect to this. Removing the prints lets the program run without crashing.

[kron4eg]

I'm getting same panic, EVEN WITH empty callback, without any prints or anything else. Like this:

WalkCallback := func(dirPath string, ent *git.TreeEntry) int {
    return 0
}

rootTree.Walk(WalkCallback)

But my tracebacks are always pointing to this: https://github.com/libgit2/git2go/blob/master/tree.go#L109

err := C._go_git_treewalk(
    t.cast_ptr,
    C.GIT_TREEWALK_PRE,
    unsafe.Pointer(&callback),
)

[kron4eg]

@carlosmn

It looks like there's a timing aspect to this.

No it's not timing, it's actual problem with git2go code. In 1.4 invalid (contains int for example) go pointers will cause panic during GC. What we are doing here is instructing Go to treat pointer from C (with int) as it was Go's pointer (unsafe.Pointer) which is wrong. We was able get away with this in 1.3 series, but 1.4 is actually have this fixed. This is by design. If we want to operate foreign pointers uintptr should be used.

Here golang/go#9191 (comment) is explained better.

[b35li]

@kron4eg

I got the same issue like your comments. my go version is "go version go1.4.1 linux/amd64"
How shall I fix this?

Thanks in advance, it is urgent!!!!

[kron4eg]

@b35li use go 1.3.3, it's OK there.

[b35li]

@kron4eg
thanks for your comments.
Is there any other way, this module is one part of big project, it is hard to say switch the go version directly for me.

I am confused if the problem is with git2go, why don't we fix in git2go, while we downgrad the go version to suit it?

[kron4eg]

I think the problem is how git2go handle C pointers. Maybe I'm wrong, but I don't have time to fix it myself :) So I just downgraded.

[b35li]

@kron4eg Got it. thanks very much. I will downgrade it and have a try.

@carlosmn Any comments?

[carlosmn]

@kron4eg if we are using ints as pointers, that's definitely a bug, but I don't know which code you're saying that is doing this.

[kron4eg]

@carlosmn I can't say exactly (in other case I'd send PR with fix), but looks like an int pointer returned from libgit2 itself at some point. And Go1.4 runtime see it and crash.

[b35li]

@kron4eg Exactly, I have no idea what root cause it is , however, when I downgrade to 1.3.3 it indeed works. wonderful.

@carlosmn if you need any feedback from me, just @ me. Thanks.

[shinningstar]

I am using 1.4.2 for Tree.Walk testing and it always runtime panic for "nil pointer reference" error.
After printing some log, it shows the callback function pointer(as the payload of git_tree_walk) becomes invalid after a while. It seems that the memory of use provided callback function is recycled by GC.

func CallbackGitTreeWalk(_root *C.char, _entry *C.git_tree_entry, ptr unsafe.Pointer) C.int {
    root := C.GoString(_root)
    entry := _entry
    callback := *(*TreeWalkCallback)(ptr)
    fmt.Println("callback func %v", callback)
    return C.int(callback(root, newTreeEntry(entry)))
}

callback func 0x488c30
ext/pkg/linux_amd64/golang.org/x/tools/refactor/importgraph.a
callback func 0x488c30
ext/pkg/linux_amd64/golang.org/x/tools/refactor/lexical.a
callback func 0x488c30
ext/pkg/linux_amd64/golang.org/x/tools/refactor/rename.a
callback func <nil>
--- FAIL: TestWalk (5.86s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x5251d5]

[shinningstar]

I think the problem is that GC has freed the pointer of callback since it doesn't known that it is referenced by C code. So to prevent GC on the pointer of callback, I've add it into a map and delete it after C._go_git_treewalk call finished. The changed code on tree.go is here:

func (t Tree) Walk(callback TreeWalkCallback) error {
    runtime.LockOSThread()
    defer runtime.UnlockOSThread()
    cbmap := make(map[unsafe.Pointer]interface{}, 1)
    ptr := unsafe.Pointer(&callback)
    cbmap[ptr] = callback
    defer func() { delete(cbmap, ptr) }() 
    err := C._go_git_treewalk(
        t.cast_ptr,
        C.GIT_TREEWALK_PRE,
        ptr,
    )   

    if err < 0 {
        return MakeGitError(err)
    }   

    return nil 
}

@carlosmn Do you have any comments on this?
@kron4eg Could you help to test this code on your situation?

[shinningstar]

By the way, to reproduce the runtime panic on tree_walk, you should walk a larger git repository to make GC happened during walking. Maybe to reduce the percentage of GC by debug.SetGCPercent() is another way.