linux/arm: panic: fatal error: malloc deadlock

[CJentzsch]

using the cross compiled binary (https://build.ethdev.com/builds/ARM%20Go%20develop%20branch/geth-ARM-latest.tar.bz2)

./geth version
Geth
Version: 1.4.0-unstable
Protocol Versions: [63 62 61]
Network Id: 1
Go Version: go1.5.1
OS: linux

I got a panic: https://gist.github.com/CJentzsch/31aab5b31fae6805f893

[fjl]

Did this happen only once? Can you reproduce it?

[fjl]

Also, which commit is this exactly? I'm guessing it's e165c2d, but would be nice to be certain.
If you don't know the commit but still have the binary, please provide the SHA-1 hash of geth.

[CJentzsch]

Got the binary about 2 hours ago.
Those are the hashes:

[pi@alarmpi ~]$ sha1sum geth
97fe060e06bb039b808243a8fee8736f69fbd966  geth
[pi@alarmpi ~]$ sha1sum geth-ARM-latest.tar.bz2 
13d252131a7b17f33533524387e21de89a9eec9e  geth-ARM-latest.tar.bz2

I have restarted the client and it is still running. Let you know when it panics again.

[karalabe]

Just a random note here, I have been running an Android build a while ago and noticed a silent crash (i.e. no output, process just stopped being) around the same block height (I remember hitting 90K before dying), and @anthony-cros also reported that running on the Tizen platform, geth failed at 90K blocks. All of these platforms seem to be some variation of ARM code.

[fjl]

@CJentzsch @karalabe please set GOTRACEBACK=crash in the environment and turn on core dump writes (ulimit -c unlimited, etc.)
Getting an actual core would probably help with finding the issue.

[donSchoe]

same here, raspberry pi, archlinux arm and latest binary (downloaded yesterday)

panic log: https://gist.github.com/donSchoe/12f008f036433d5590ff

$ uname -a
Linux ethost 4.1.12-1-ARCH #1 PREEMPT Tue Oct 27 19:16:04 MDT 2015 armv6l GNU/Linux
$ ~/opt/geth version
Geth
Version: 1.4.0-unstable
Protocol Versions: [63 62 61]
Network Id: 1
Go Version: go1.5.1
OS: linux
GOPATH=
GOROOT=/usr/local/go

and can not run it anymore:

$ ~/opt/geth --verbosity 3 --rpc=true --rpcport 13337 --rpccorsdomain '"*"' console 2>> /tmp/eth.p/log
Fatal: Error string IPC: listen unix /home/user/.ethereum/geth.ipc: socket: too many open files
$ lsof | awk '{print $1}' | uniq -c | sort -rn | head     42 bash
     37 systemd
     15 awk
     13 sort
     13 lsof
     11 tail
      9 uniq
      9 head
      8 lsof
      4 sshd

Having the same version as @CJentzsch:

$ sha1sum ~/opt/*
97fe060e06bb039b808243a8fee8736f69fbd966  opt/geth
13d252131a7b17f33533524387e21de89a9eec9e  opt/geth-ARM-20151103142042-1.4.0-dda3bf3.tar.bz2

[donSchoe]

Was able to catch the core dump.

http://chaos.cach.co/dump/hpbinjpd/cltl/core.geth.1001.2549b53633c74295bcfbeb9903c111b9.276.1446675193000000.lz4

$ export GOTRACEBACK=crash
$ ulimit -c unlimited
$ gdb
bash: gdb: command not found
$ ~/opt/geth recover 
I1104 22:13:12.956863     276 database.go:71] Alloted 16MB cache to /home/user/.ethereum/blockchain
panic: runtime error: index out of range

goroutine 1 [running]:
runtime.gopanic(0x7eaa30, 0x96dca008)
        /usr/local/go/src/runtime/panic.go:461 +0x330 fp=0x96ecd784 sp=0x96ecd740
runtime.panicindex()
        /usr/local/go/src/runtime/panic.go:12 +0x48 fp=0x96ecd798 sp=0x96ecd784
main.blockRecovery(0x96eea870)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:559 +0x480 fp=0x96ecd964 sp=0x96ecd798
github.com/codegangsta/cli.Command.Run(0x883358, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x96e5c0, 0x51, 0xa57fe8, ...)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/codegangsta/cli/command.go:131 +0xdf0 fp=0x96ecdbf0 sp=0x96ecd964
github.com/codegangsta/cli.(*App).Run(0x96e1a640, 0x96daa110, 0x2, 0x2, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/codegangsta/cli/app.go:175 +0xdac fp=0x96ecdf40 sp=0x96ecdbf0
main.main()
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:372 +0x8c fp=0x96ecdf94 sp=0x96ecdf40
runtime.main()
        /usr/local/go/src/runtime/proc.go:111 +0x2b4 fp=0x96ecdfbc sp=0x96ecdf94
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96ecdfbc sp=0x96ecdfbc

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96ddafd8 sp=0x96ddafd8

goroutine 2 [force gc (idle)]:
runtime.gopark(0xa4fa04, 0xdea0f8, 0x8cf2a0, 0xf, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc67a8 sp=0x96dc6794
runtime.goparkunlock(0xdea0f8, 0x8cf2a0, 0xf, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc67c4 sp=0x96dc67a8
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:152 +0xa8 fp=0x96dc67dc sp=0x96dc67c4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc67dc sp=0x96dc67dc
created by runtime.init.4
        /usr/local/go/src/runtime/proc.go:141 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0xa4fa04, 0xdea300, 0x8c95f0, 0xd, 0x64914, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc6f9c sp=0x96dc6f88
runtime.goparkunlock(0xdea300, 0x8c95f0, 0xd, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc6fb8 sp=0x96dc6f9c
runtime.bgsweep(0x96dba080)
        /usr/local/go/src/runtime/mgcsweep.go:51 +0x90 fp=0x96dc6fd4 sp=0x96dc6fb8
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc6fd4 sp=0x96dc6fd4
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:206 +0x4c

goroutine 4 [finalizer wait]:
runtime.gopark(0xa4fa04, 0xdff844, 0x8cf070, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc7788 sp=0x96dc7774
runtime.goparkunlock(0xdff844, 0x8cf070, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc77a4 sp=0x96dc7788
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:154 +0xa4 fp=0x96dc77dc sp=0x96dc77a4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc77dc sp=0x96dc77dc
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:135 +0x5c

goroutine 5 [syscall]:
runtime.notetsleepg(0xdffc88, 0xffffffff, 0xffffffff, 0x1)
        /usr/local/go/src/runtime/lock_futex.go:202 +0x4c fp=0x96dc7f98 sp=0x96dc7f80
runtime.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:111 +0x190 fp=0x96dc7fb8 sp=0x96dc7f98
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x14 fp=0x96dc7fdc sp=0x96dc7fb8
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc7fdc sp=0x96dc7fdc
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 6 [chan receive]:
runtime.gopark(0xa4fa04, 0x96dba6f0, 0x8cd8f0, 0xc, 0xdea317, 0x3)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc870c sp=0x96dc86f8
runtime.goparkunlock(0x96dba6f0, 0x8cd8f0, 0xc, 0xdea317, 0x3)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc8728 sp=0x96dc870c
runtime.chanrecv(0x71fd30, 0x96dba6c0, 0x96dc87c4, 0x1, 0x0)
        /usr/local/go/src/runtime/chan.go:505 +0x960 fp=0x96dc8794 sp=0x96dc8728
runtime.chanrecv2(0x71fd30, 0x96dba6c0, 0x96dc87c4, 0x0)
        /usr/local/go/src/runtime/chan.go:354 +0x30 fp=0x96dc87ac sp=0x96dc8794
github.com/ethereum/go-ethereum/logger/glog.(*loggingT).flushDaemon(0xdeab10)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/logger/glog/glog.go:897 +0x60 fp=0x96dc87d4 sp=0x96dc87ac
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc87d4 sp=0x96dc87d4
created by github.com/ethereum/go-ethereum/logger/glog.init.1
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/logger/glog/glog.go:428 +0x64

goroutine 7 [runnable]:
runtime.notetsleepg(0xdea3a4, 0x29defdc0, 0x1, 0x1)
        /usr/local/go/src/runtime/lock_futex.go:202 +0x4c fp=0x96dc8f88 sp=0x96dc8f70
runtime.timerproc()
        /usr/local/go/src/runtime/time.go:209 +0x11c fp=0x96dc8fdc sp=0x96dc8f88
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc8fdc sp=0x96dc8fdc
created by runtime.addtimerLocked
        /usr/local/go/src/runtime/time.go:116 +0x1c4

goroutine 8 [select]:
runtime.gopark(0xa4fa1c, 0x96dc9748, 0x883988, 0x6, 0x18, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc959c sp=0x96dc9588
runtime.selectgoImpl(0x96dc9748, 0x0, 0xc)
        /usr/local/go/src/runtime/select.go:392 +0x14d4 fp=0x96dc9678 sp=0x96dc959c
runtime.selectgo(0x96dc9748)
        /usr/local/go/src/runtime/select.go:212 +0x10 fp=0x96dc9688 sp=0x96dc9678
github.com/ethereum/go-ethereum/logger.dispatchLoop()
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/logger/sys.go:81 +0x4f4 fp=0x96dc97dc sp=0x96dc9688
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc97dc sp=0x96dc97dc
created by github.com/ethereum/go-ethereum/logger.init.1
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/logger/sys.go:60 +0x24

goroutine 11 [chan receive]:
runtime.gopark(0xa4fa04, 0x96f2c930, 0x8cd8f0, 0xc, 0x96f33117, 0x3)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc9e90 sp=0x96dc9e7c
runtime.goparkunlock(0x96f2c930, 0x8cd8f0, 0xc, 0xa4fa17, 0x3)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc9eac sp=0x96dc9e90
runtime.chanrecv(0x720600, 0x96f2c900, 0x96dc9f54, 0x96dc9f01, 0x430000)
        /usr/local/go/src/runtime/chan.go:505 +0x960 fp=0x96dc9f18 sp=0x96dc9eac
runtime.chanrecv1(0x720600, 0x96f2c900, 0x96dc9f54)
        /usr/local/go/src/runtime/chan.go:349 +0x28 fp=0x96dc9f30 sp=0x96dc9f18
github.com/ethereum/go-ethereum/p2p/nat.Any.func1(0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:158 +0xc8 fp=0x96dc9f5c sp=0x96dc9f30
github.com/ethereum/go-ethereum/p2p/nat.(*autodisc).wait.func1()
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:242 +0x4c fp=0x96dc9f74 sp=0x96dc9f5c
sync.(*Once).Do(0x96f0ebd8, 0x96dc9fb8)
        /usr/local/go/src/sync/once.go:44 +0x118 fp=0x96dc9f88 sp=0x96dc9f74
github.com/ethereum/go-ethereum/p2p/nat.(*autodisc).wait(0x96f0ebd0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:244 +0x58 fp=0x96dc9fcc sp=0x96dc9f88
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc9fcc sp=0x96dc9fcc
created by github.com/ethereum/go-ethereum/p2p/nat.startautodisc
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:203 +0xa0

goroutine 12 [select]:
runtime.gopark(0xa4fa1c, 0x96dc2784, 0x883988, 0x6, 0x18, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc2654 sp=0x96dc2640
runtime.selectgoImpl(0x96dc2784, 0x0, 0xc)
        /usr/local/go/src/runtime/select.go:392 +0x14d4 fp=0x96dc2730 sp=0x96dc2654
runtime.selectgo(0x96dc2784)
        /usr/local/go/src/runtime/select.go:212 +0x10 fp=0x96dc2740 sp=0x96dc2730
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x96e02790)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x228 fp=0x96dc27d4 sp=0x96dc2740
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc27d4 sp=0x96dc27d4
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x258

goroutine 13 [runnable]:
runtime.notetsleepg(0xdeac6c, 0xffffffff, 0xffffffff, 0x96dc2d50)
        /usr/local/go/src/runtime/lock_futex.go:202 +0x4c fp=0x96dc2d38 sp=0x96dc2d20
runtime.gcBgMarkStartWorkers()
        /usr/local/go/src/runtime/mgc.go:1240 +0xb0 fp=0x96dc2d58 sp=0x96dc2d38
runtime.gc(0x0)
        /usr/local/go/src/runtime/mgc.go:934 +0x118 fp=0x96dc2fc4 sp=0x96dc2d58
runtime.backgroundgc()
        /usr/local/go/src/runtime/mgc.go:897 +0x34 fp=0x96dc2fdc sp=0x96dc2fc4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc2fdc sp=0x96dc2fdc
created by runtime.startGC
        /usr/local/go/src/runtime/mgc.go:870 +0x178

goroutine 14 [mark worker (idle)]:
runtime.gopark(0xa4f954, 0x96dccdc0, 0x8e4108, 0x12, 0x14, 0x0)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc3794 sp=0x96dc3780
runtime.gcBgMarkWorker(0x96dbea00)
        /usr/local/go/src/runtime/mgc.go:1289 +0xd8 fp=0x96dc37d4 sp=0x96dc3794
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc37d4 sp=0x96dc37d4
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1239 +0x84

goroutine 15 [chan receive]:
runtime.gopark(0xa4fa04, 0x96f2caf0, 0x8cd8f0, 0xc, 0x96dcb917, 0x3)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96dc3ef0 sp=0x96dc3edc
runtime.goparkunlock(0x96f2caf0, 0x8cd8f0, 0xc, 0x17, 0x3)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dc3f0c sp=0x96dc3ef0
runtime.chanrecv(0x720090, 0x96f2cac0, 0x96dc3fb8, 0x96dc3f01, 0x430000)
        /usr/local/go/src/runtime/chan.go:505 +0x960 fp=0x96dc3f78 sp=0x96dc3f0c
runtime.chanrecv1(0x720090, 0x96f2cac0, 0x96dc3fb8)
        /usr/local/go/src/runtime/chan.go:349 +0x28 fp=0x96dc3f90 sp=0x96dc3f78
github.com/ethereum/go-ethereum/p2p/nat.discoverUPnP(0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natupnp.go:129 +0xf4 fp=0x96dc3fbc sp=0x96dc3f90
github.com/ethereum/go-ethereum/p2p/nat.Any.func1.1(0x96f2c900)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:155 +0x14 fp=0x96dc3fd4 sp=0x96dc3fbc
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc3fd4 sp=0x96dc3fd4
created by github.com/ethereum/go-ethereum/p2p/nat.Any.func1
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:155 +0x58

goroutine 16 [runnable]:
runtime.Gosched()
        /usr/local/go/src/runtime/proc.go:166 +0x10 fp=0x96ddbb90 sp=0x96ddbb88
runtime.mallocgc(0x1000, 0x72cef8, 0x1, 0x1)
        /usr/local/go/src/runtime/malloc.go:722 +0x564 fp=0x96ddbbfc sp=0x96ddbb90
runtime.newarray(0x72cef8, 0x1000, 0x11)
        /usr/local/go/src/runtime/malloc.go:777 +0xe4 fp=0x96ddbc1c sp=0x96ddbbfc
runtime.makeslice(0x71fa00, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/slice.go:32 +0x1d8 fp=0x96ddbc44 sp=0x96ddbc1c
syscall.NetlinkRIB(0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/syscall/netlink_linux.go:67 +0x2d4 fp=0x96ddbd58 sp=0x96ddbc44
net.interfaceAddrTable(0x96ddbed4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/interface_linux.go:121 +0x48 fp=0x96ddbdf4 sp=0x96ddbd58
net.(*Interface).Addrs(0x96ddbed4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/interface.go:67 +0x128 fp=0x96ddbe28 sp=0x96ddbdf4
github.com/ethereum/go-ethereum/p2p/nat.potentialGateways(0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natpmp.go:113 +0x128 fp=0x96ddbf14 sp=0x96ddbe28
github.com/ethereum/go-ethereum/p2p/nat.discoverPMP(0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natpmp.go:67 +0x28 fp=0x96ddbfb4 sp=0x96ddbf14
github.com/ethereum/go-ethereum/p2p/nat.Any.func1.2(0x96f2c900)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:156 +0x14 fp=0x96ddbfcc sp=0x96ddbfb4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96ddbfcc sp=0x96ddbfcc
created by github.com/ethereum/go-ethereum/p2p/nat.Any.func1
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:156 +0x74

goroutine 18 [sleep]:
runtime.gopark(0xa4fa04, 0xdea398, 0x883c98, 0x5, 0x13, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96ddda94 sp=0x96ddda80
runtime.goparkunlock(0xdea398, 0x883c98, 0x5, 0x13, 0x2)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96dddab0 sp=0x96ddda94
time.Sleep(0x4c4b40, 0x0)
        /usr/local/go/src/runtime/time.go:59 +0x104 fp=0x96dddad4 sp=0x96dddab0
github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x96f33270, 0x96f309a0, 0x7d2b7500, 0x0, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/httpu/httpu.go:84 +0x9c0 fp=0x96dddcd0 sp=0x96dddad4
github.com/huin/goupnp/ssdp.SSDPRawSearch(0x96f33270, 0x9515f0, 0x31, 0x2, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/ssdp/ssdp.go:53 +0x4d4 fp=0x96dddecc sp=0x96dddcd0
github.com/huin/goupnp.DiscoverDevices(0x9515f0, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/goupnp.go:66 +0xd8 fp=0x96dddf64 sp=0x96dddecc
github.com/ethereum/go-ethereum/p2p/nat.discover(0x96f2cac0, 0x9515f0, 0x31, 0xa4ee64)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natupnp.go:140 +0x24 fp=0x96dddfc4 sp=0x96dddf64
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dddfc4 sp=0x96dddfc4
created by github.com/ethereum/go-ethereum/p2p/nat.discoverUPnP
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natupnp.go:115 +0x70

goroutine 19 [sleep]:
runtime.gopark(0xa4fa04, 0xdea398, 0x883c98, 0x5, 0x13, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x148 fp=0x96ddca94 sp=0x96ddca80
runtime.goparkunlock(0xdea398, 0x883c98, 0x5, 0x13, 0x2)
        /usr/local/go/src/runtime/proc.go:191 +0x44 fp=0x96ddcab0 sp=0x96ddca94
time.Sleep(0x4c4b40, 0x0)
        /usr/local/go/src/runtime/time.go:59 +0x104 fp=0x96ddcad4 sp=0x96ddcab0
github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x96f33230, 0x96f30930, 0x7d2b7500, 0x0, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/httpu/httpu.go:84 +0x9c0 fp=0x96ddccd0 sp=0x96ddcad4
github.com/huin/goupnp/ssdp.SSDPRawSearch(0x96f33230, 0x951628, 0x31, 0x2, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/ssdp/ssdp.go:53 +0x4d4 fp=0x96ddcecc sp=0x96ddccd0
github.com/huin/goupnp.DiscoverDevices(0x951628, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/huin/goupnp/goupnp.go:66 +0xd8 fp=0x96ddcf64 sp=0x96ddcecc
github.com/ethereum/go-ethereum/p2p/nat.discover(0x96f2cac0, 0x951628, 0x31, 0xa4ee68)
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natupnp.go:140 +0x24 fp=0x96ddcfc4 sp=0x96ddcf64
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96ddcfc4 sp=0x96ddcfc4
created by github.com/ethereum/go-ethereum/p2p/nat.discoverUPnP
        /slave/build-go-ethereum-develop-arm/go/src/github.com/ethereum/go-ethereum/p2p/nat/natupnp.go:127 +0xa4

goroutine 20 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x96dad860)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:112 fp=0x96dc47d4 sp=0x96dc47d4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc47d4 sp=0x96dc47d4
created by github.com/syndtr/goleveldb/leveldb.openDB
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:126 +0x710

goroutine 21 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x96dad860)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_state.go:79 fp=0x96dc4fd4 sp=0x96dc4fd4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc4fd4 sp=0x96dc4fd4
created by github.com/syndtr/goleveldb/leveldb.openDB
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:127 +0x72c

goroutine 22 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x96dad860)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:775 fp=0x96dc57d4 sp=0x96dc57d4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc57d4 sp=0x96dc57d4
created by github.com/syndtr/goleveldb/leveldb.openDB
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:130 +0x778

goroutine 23 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x96dad860)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:743 fp=0x96dc5fd4 sp=0x96dc5fd4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x96dc5fd4 sp=0x96dc5fd4
created by github.com/syndtr/goleveldb/leveldb.openDB
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:131 +0x794

goroutine 24 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0x96dad860)
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_write.go:34 fp=0x973587d4 sp=0x973587d4
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x973587d4 sp=0x973587d4
created by github.com/syndtr/goleveldb/leveldb.openDB
        /slave/build-go-ethereum-develop-arm/build/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:132 +0x7b0
Aborted (core dumped)
$ cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e
$ coredumpctl list
TIME                            PID   UID   GID SIG PRESENT EXE
Wed 2015-11-04 22:13:25 UTC     276  1001   100   6 * /home/user/opt/geth
$ coredumpctl info 276
           PID: 276 (geth)
           UID: 1001 (user)
           GID: 100 (users)
        Signal: 6 (ABRT)
     Timestamp: Wed 2015-11-04 22:13:13 UTC (4min 15s ago)
  Command Line: /home/user/opt/geth recover
    Executable: /home/user/opt/geth
 Control Group: /user.slice/user-1001.slice/session-c1.scope
          Unit: session-c1.scope
         Slice: user-1001.slice
       Session: c1
     Owner UID: 1001 (user)
       Boot ID: 2549b53633c74295bcfbeb9903c111b9
    Machine ID: 529280e30d1a42a2b88e3074fb0b13f8
      Hostname: ethost
      Coredump: /var/lib/systemd/coredump/core.geth.1001.2549b53633c74295bcfbeb9903c111b9.276.1446675193000000.lz4
       Message: Process 276 (geth) of user 1001 dumped core.
$ ~/opt/geth upgradedb
I1104 22:21:29.911519     314 chaincmd.go:137] Upgrading blockchain database
I1104 22:21:29.913543     314 database.go:71] Alloted 16MB cache to /home/user/.ethereum/chaindata
I1104 22:21:30.867880     314 blockchain.go:189] Last header: #490325 [624fe106��] TD=2620496232267366942
I1104 22:21:30.870302     314 blockchain.go:190] Last block: #490325 [624fe106��] TD=2620496232267366942
I1104 22:21:30.872666     314 blockchain.go:191] Fast block: #490325 [624fe106��] TD=2620496232267366942
I1104 22:21:30.993273     314 cmd.go:246] Exporting blockchain to /home/user/.ethereum/blockchain_3_20151104_222130.chain
I1104 22:21:30.995069     314 blockchain.go:396] exporting 490326 blocks...
Fatal: Unable to export chain for reimport export failed on #91: not found

[SebastianBoyd]

Same problem

[karalabe]

@donSchoe Please open a new issue with your report. This issue tracks an upstream Go bug. Your crash is related to our implementation. After you open the new issue, please mention @fjl in it too so that he's aware of the trie errors.

@SebastianBoyd Please open a new bug report too. As there are two separate issues being reported here, I'm not sure which one of them occurred to you. Open a new one and we'll link it together with a previous issue if it's the case.

[anthony-cros]

@karalabe I can confirm that I got the same problems 3 times already on an arm binary (created via xgo), geth silently stopped at various blocks (~90K, ~120K, ~150K). I was using the latest release. Any idea what could cause it? Need me to try the GOTRACEBACK procedure as well?

[karalabe]

@anthony-cros Yes please. It's kind of shooting in the dark as we have no idea where to look, but an extra data point cannot hurt. So if you can repro it, all information would be welcome.