runtime: OpenReader: Invalid heap pointer in both 1.4rc2 and 1.4rc1

[gopherbot]

by pruest:

Works in all releases prior to 1.4rc1.

Code base installs, but fails at run time (test or regular execution).
The code triggering the problem is attempting to open a binary looking
for an embedded zip file. In this case there is no zip file in the binary. 
In the past an error would have been returned.

    reader, err := zip.OpenReader(programName)  // load.go line 102 in trace below.

Excerpt from full trace below:
--------------------------------
goroutine 5 [garbage collection]:
runtime.switchtoM()
    /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc20805e8b8 sp=0xc20805e8b0
runtime.gogc(0x0)
    /usr/local/go/src/runtime/malloc.go:469 +0x1cf fp=0xc20805e8f0 sp=0xc20805e8b8
runtime.mallocgc(0x12000, 0x4158140, 0x1, 0x403dff3)
    /usr/local/go/src/runtime/malloc.go:341 +0x391 fp=0xc20805e9a0 sp=0xc20805e8f0
runtime.newarray(0x4158140, 0x10400, 0x400)
    /usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc20805e9d8 sp=0xc20805e9a0
runtime.makeslice(0x414f980, 0x10400, 0x10400, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc20805ea20 sp=0xc20805e9d8
archive/zip.readDirectoryEnd(0x43a4d08, 0xc20802a090, 0x30d1c8, 0x0, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:330 +0x10c fp=0xc20805eb68 sp=0xc20805ea20
archive/zip.(*Reader).init(0xc20802e588, 0x43a4d08, 0xc20802a090, 0x30d1c8, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:76 +0x65 fp=0xc20805ec58 sp=0xc20805eb68
archive/zip.OpenReader(0x7fff5fbffa58, 0x5c, 0x0, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:57 +0x1ae fp=0xc20805ecf0 sp=0xc20805ec58
vu/load.newLoader(0x19)
    /Users/rust/code/vu/src/vu/load/load.go:102 +0x28c fp=0xc20805edd0 sp=0xc20805ecf0


------------------------------------------------------------------------------------------------------
Full trace from 1.4rc2
------------------------------------------------------------------------------------------------------
runtime: garbage collector found invalid heap pointer *(0xc20805ee48+0x80)=0x18 s=nil
fatal error: invalid heap pointer

runtime stack:
runtime.throw(0x42c2f63)
    /usr/local/go/src/runtime/panic.go:491 +0xad fp=0x7fff5fbff310 sp=0x7fff5fbff2e0
scanblock(0xc20805ee48, 0x128, 0x422e178)
    /usr/local/go/src/runtime/mgc0.c:378 +0x551 fp=0x7fff5fbff450 sp=0x7fff5fbff310
scanframe(0x7fff5fbff558, 0x0, 0x101)
    /usr/local/go/src/runtime/mgc0.c:719 +0x164 fp=0x7fff5fbff4c0 sp=0x7fff5fbff450
runtime.gentraceback(0x4039df0, 0xc20805e8b0, 0x0, 0xc208000b40, 0x0, 0x0, 0x7fffffff,
0x7fff5fbff608, 0x0, 0x0, ...)
    /usr/local/go/src/runtime/traceback.go:311 +0x7a8 fp=0x7fff5fbff5b0 sp=0x7fff5fbff4c0
scanstack(0xc208000b40)
    /usr/local/go/src/runtime/mgc0.c:777 +0x21c fp=0x7fff5fbff620 sp=0x7fff5fbff5b0
markroot(0xc208010000, 0x9)
    /usr/local/go/src/runtime/mgc0.c:553 +0xe7 fp=0x7fff5fbff680 sp=0x7fff5fbff620
runtime.parfordo(0xc208010000)
    /usr/local/go/src/runtime/parfor.c:76 +0xb2 fp=0x7fff5fbff700 sp=0x7fff5fbff680
gc(0x7fff5fbff838)
    /usr/local/go/src/runtime/mgc0.c:1439 +0x25e fp=0x7fff5fbff818 sp=0x7fff5fbff700
runtime.gc_m()
    /usr/local/go/src/runtime/mgc0.c:1368 +0xe0 fp=0x7fff5fbff850 sp=0x7fff5fbff818
runtime.onM(0x42e1010)
    /usr/local/go/src/runtime/asm_amd64.s:257 +0x68 fp=0x7fff5fbff858 sp=0x7fff5fbff850
runtime.mstart()
    /usr/local/go/src/runtime/proc.c:818 fp=0x7fff5fbff860 sp=0x7fff5fbff858

goroutine 5 [garbage collection]:
runtime.switchtoM()
    /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc20805e8b8 sp=0xc20805e8b0
runtime.gogc(0x0)
    /usr/local/go/src/runtime/malloc.go:469 +0x1cf fp=0xc20805e8f0 sp=0xc20805e8b8
runtime.mallocgc(0x12000, 0x4158140, 0x1, 0x403dff3)
    /usr/local/go/src/runtime/malloc.go:341 +0x391 fp=0xc20805e9a0 sp=0xc20805e8f0
runtime.newarray(0x4158140, 0x10400, 0x400)
    /usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc20805e9d8 sp=0xc20805e9a0
runtime.makeslice(0x414f980, 0x10400, 0x10400, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc20805ea20 sp=0xc20805e9d8
archive/zip.readDirectoryEnd(0x43a4d08, 0xc20802a090, 0x30d1c8, 0x0, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:330 +0x10c fp=0xc20805eb68 sp=0xc20805ea20
archive/zip.(*Reader).init(0xc20802e588, 0x43a4d08, 0xc20802a090, 0x30d1c8, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:76 +0x65 fp=0xc20805ec58 sp=0xc20805eb68
archive/zip.OpenReader(0x7fff5fbffa58, 0x5c, 0x0, 0x0, 0x0)
    /usr/local/go/src/archive/zip/reader.go:57 +0x1ae fp=0xc20805ecf0 sp=0xc20805ec58
vu/load.newLoader(0x19)
    /Users/rust/code/vu/src/vu/load/load.go:102 +0x28c fp=0xc20805edd0 sp=0xc20805ecf0
vu/load.NewLoader(0x0, 0x0)
    /Users/rust/code/vu/src/vu/load/load.go:78 +0x31 fp=0xc20805ee00 sp=0xc20805edd0
vu/audio.TestAudio(0xc208062000)
    /Users/rust/code/vu/src/vu/audio/audio_test.go:15 +0x67 fp=0xc20805ef78 sp=0xc20805ee00
testing.tRunner(0xc208062000, 0x42c73a0)
    /usr/local/go/src/testing/testing.go:447 +0xbf fp=0xc20805efd0 sp=0xc20805ef78
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20805efd8 sp=0xc20805efd0
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:555 +0xa8b

goroutine 1 [chan receive]:
runtime.gopark(0x4032190, 0xc208056238, 0x41ed330, 0xc)
    /usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc208061ba8 sp=0xc208061b78
runtime.goparkunlock(0xc208056238, 0x41ed330, 0xc)
    /usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc208061bd0 sp=0xc208061ba8
runtime.chanrecv(0x4150040, 0xc2080561e0, 0xc208061dc0, 0x4058a01, 0x0)
    /usr/local/go/src/runtime/chan.go:410 +0x452 fp=0xc208061c70 sp=0xc208061bd0
runtime.chanrecv1(0x4150040, 0xc2080561e0, 0xc208061dc0)
    /usr/local/go/src/runtime/chan.go:311 +0x2b fp=0xc208061ca0 sp=0xc208061c70
testing.RunTests(0x4234e10, 0x42c73a0, 0x1, 0x1, 0xee99d9ca48babd01)
    /usr/local/go/src/testing/testing.go:556 +0xad6 fp=0xc208061e48 sp=0xc208061ca0
testing.(*M).Run(0xc2080300f0, 0x42cc8e0)
    /usr/local/go/src/testing/testing.go:485 +0x6c fp=0xc208061ed0 sp=0xc208061e48
main.main()
    vu/audio/_test/_testmain.go:52 +0x1d5 fp=0xc208061f98 sp=0xc208061ed0
runtime.main()
    /usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xc208061fe0 sp=0xc208061f98
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208061fe8 sp=0xc208061fe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x4032190, 0x42e0bb0, 0x41ee7f0, 0xf)
    /usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc20801a798 sp=0xc20801a768
runtime.goparkunlock(0x42e0bb0, 0x41ee7f0, 0xf)
    /usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc20801a7c0 sp=0xc20801a798
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:99 +0xce fp=0xc20801a7e0 sp=0xc20801a7c0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801a7e8 sp=0xc20801a7e0
created by runtime.init·4
    /usr/local/go/src/runtime/proc.go:87 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x4032190, 0x42cccb8, 0x41e8210, 0xd)
    /usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc20801df98 sp=0xc20801df68
runtime.goparkunlock(0x42cccb8, 0x41e8210, 0xd)
    /usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc20801dfc0 sp=0xc20801df98
runtime.bgsweep()
    /usr/local/go/src/runtime/mgc0.go:98 +0xbc fp=0xc20801dfe0 sp=0xc20801dfc0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801dfe8 sp=0xc20801dfe0
created by gc
    /usr/local/go/src/runtime/mgc0.c:1383

goroutine 4 [finalizer wait]:
runtime.gopark(0x4032190, 0x42cccb0, 0x41ee0b0, 0xe)
    /usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc208018f30 sp=0xc208018f00
runtime.goparkunlock(0x42cccb0, 0x41ee0b0, 0xe)
    /usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc208018f58 sp=0xc208018f30
runtime.runfinq()
    /usr/local/go/src/runtime/malloc.go:727 +0xba fp=0xc208018fe0 sp=0xc208018f58
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208018fe8 sp=0xc208018fe0
created by runtime.createfing
    /usr/local/go/src/runtime/malloc.go:707 +0x5e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20805dfe8 sp=0xc20805dfe0

[ianlancetaylor]

Comment 1:

How can we recreate the problem?
What system are you running on?
Does your program use cgo?

Labels changed: added release-go1.4maybe, repo-main.

[gopherbot]

Comment 2 by pruest:

OS X 10.10.1.  Cgo is involved.  Refinement of the problem in the attached file using
standard OpenAL on OS X and the OpenAL go bindings provided provided in the zip. Here is
the program from the zip.
package main
import (
    "al" // from https://github.com/gazed/vu/tree/master/src/vu/audio/al
    "archive/zip"
    "os"
)
// Issue: 9191. All 4 cgo calls and zip.OpenReader are necessary to cause issue.
func main() {
    al.Init()                                 // cgo
    if dev := al.OpenDevice(""); dev != nil { // cgo
        if ctx := al.CreateContext(dev, nil); ctx != nil { // cgo
            defer al.DestroyContext(ctx) // cgo
            programName := os.Args[0]
            zip.OpenReader(programName) // crashes?
        }
    }
}

Attachments:

1. issue9191.zip (10332 bytes)

[ianlancetaylor]

Comment 3:

I doubt this has anything to do with zip.  You would probably get the same crash calling
runtime.GC() instead of zip.OpenReader.
The problem is that something in the Go heap has a pointer type but contains the value
0x18, which is invalid.
I can't recreate the problem on GNU/Linux.  I don't see anything in your code that would
cause it.  This will have to be debugged by somebody with access to a Darwin system.

[gopherbot]

Comment 4 by pruest:

Verified that problem can't be recreated on Windows 8.1 using OpenAL.dll from
http://kcat.strangesoft.net/openal.html.

[minux]

Comment 5:

Reproduced on OS X, but it's not a Go problem.
the *al.Device returned from al.OpenDevice("") is 0x18, which is obviously
an invalid pointer.
If al.OpenDevice could return invalid pointers, the value shouldn't use a
Go pointer type to store.
Because regardless of the platform, the value returned from al.OpenDevice
is not a Go pointer (it can be a C pointer, or even an regular integer), so
Go's GC won't touch it anyway,  you can just use a uintptr to store it.
This situation is similar to Window's HANDLE.
Arguably, this is a bug of OS X OpenAL implementation, but as ALCdevice
is an opaque struct, whether alcOpenDevice returns a real pointer or not
doesn't matter that much to a C programmer.
For Go's new fully precise GC, not only passing Go heap pointer to C poses
problem, even storing a C value (pointer or struct) on Go stack/heap is also
potentially problematic as there might be a field that has a pointer type but
actually stores an integer. Perhaps we should do something here.

Status changed to WorkingAsIntended.

[gopherbot]

Comment 6 by pruest:

Fantastic! Thank you!
Both al.OpenDevice (0x18) and al.CreateContext (0x19) return invalid pointer values on
OS X and regular pointers on Windows. Using uintptr to store them, as you recommend,
solves the problem.