Skip to content

runtime: r2 corrupted by mid-stack inlining on ppc64le #30283

New issue
New issue
Closed
Closed
runtime: r2 corrupted by mid-stack inlining on ppc64le#30283
Labels
FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.release-blocker
Milestone
 
Go1.12
@mkumatag

Description

@mkumatag
mkumatag
opened on Feb 17, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12rc1 linux/ppc64le

Does this issue reproduce with the latest release?

Yes, reproduced with 1.12rc1

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build360914728=/tmp/go-build -gno-record-gcc-switches"
#

What did you do?

Steps to reproduce this issues:

root@ip9-114-192-113:~# docker run -ti golang:rc-stretch
root@361515a017ed:/go#
root@361515a017ed:/go# go version
go version go1.12rc1 linux/ppc64le
root@361515a017ed:~# mkdir -p containerd_ws/src/github.com/containerd
root@361515a017ed:~# cd containerd_ws/src/github.com/containerd/
root@361515a017ed:~/containerd_ws/src/github.com/containerd# git clone https://github.com/containerd/containerd.git
Cloning into 'containerd'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 38383 (delta 2), reused 0 (delta 0), pack-reused 38374
Receiving objects: 100% (38383/38383), 49.63 MiB | 24.02 MiB/s, done.
Resolving deltas: 100% (22575/22575), done.
root@361515a017ed:~/containerd_ws/src/github.com/containerd#
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd# cd metadata/
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# export GOPATH=/root/containerd_ws
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# go test -buildmode pie -v
=== RUN   TestContainersList
unexpected fault address 0x135889140
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x135889140 pc=0x135dbd820]

goroutine 19 [running]:
runtime.throw(0x13634f828, 0x5)
	/usr/local/go/src/runtime/panic.go:617 +0x68 fp=0xc0001fb0f0 sp=0xc0001fb0b0 pc=0x135ddff38
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:397 +0x464 fp=0xc0001fb130 sp=0xc0001fb0f0 pc=0x135df8d64
runtime.mapiterinit(0x135889100, 0xc00020e360, 0xc0001fb328)
	/usr/local/go/src/runtime/map.go:823 +0xa0 fp=0xc0001fb180 sp=0xc0001fb150 pc=0x135dbd820
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb400, 0xc00020e300, 0xc0001fb530)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:526 +0x80 fp=0xc0001fb388 sp=0xc0001fb180 pc=0x13600eb40
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb380, 0xc00020e200, 0xc0001fb738)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb590 sp=0xc0001fb388 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb300, 0xc00020e200, 0xc0001fb940)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb798 sp=0xc0001fb590 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb280, 0xc00020e100, 0xc0001fbb48)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb9a0 sp=0xc0001fb798 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0001fe1d8, 0x62c845, 0x136b6a680)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fbba8 sp=0xc0001fb9a0 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Tx).Commit(0xc0001fe1c0, 0x0, 0x0)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/tx.go:160 +0xd0 fp=0xc0001fbce8 sp=0xc0001fbba8 pc=0x13601c1e0
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).Update(0xc00019a000, 0xc0001fbed8, 0x0, 0x0)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:677 +0x108 fp=0xc0001fbd48 sp=0xc0001fbce8 pc=0x136013ef8
github.com/containerd/containerd/metadata.TestContainersList(0xc0000f4400)
	/root/containerd_ws/src/github.com/containerd/containerd/metadata/containers_test.go:74 +0x4e0 fp=0xc0001fbf70 sp=0xc0001fbd48 pc=0x13631b2d0
testing.tRunner(0xc0000f4400, 0x1366ca600)
	/usr/local/go/src/testing/testing.go:865 +0xdc fp=0xc0001fbfb0 sp=0xc0001fbf70 pc=0x135eb1dfc
runtime.goexit()
	/usr/local/go/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0001fbfb0 sp=0xc0001fbfb0 pc=0x135e135f4
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:916 +0x304

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000f4400, 0x136359315, 0x12, 0x1366ca600, 0x136b27f00)
	/usr/local/go/src/testing/testing.go:917 +0x320
testing.runTests.func1(0xc0000f4300)
	/usr/local/go/src/testing/testing.go:1157 +0x8c
testing.tRunner(0xc0000f4300, 0xc0000d3dd8)
	/usr/local/go/src/testing/testing.go:865 +0xdc
testing.runTests(0xc000193660, 0x136b29600, 0x10, 0x10, 0x0)
	/usr/local/go/src/testing/testing.go:1155 +0x2a0
testing.(*M).Run(0xc0000f2280, 0x0)
	/usr/local/go/src/testing/testing.go:1072 +0x174
main.main()
	_testmain.go:74 +0x150
exit status 2
FAIL	github.com/containerd/containerd/metadata	0.013s
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#

root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# uname -a
Linux 361515a017ed 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:33:00 UTC 2018 ppc64le GNU/Linux
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    8
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          2
Model:                 2.1 (pvr 004b 0201)
Model name:            POWER8 (architected), altivec supported
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):     0-15
NUMA node2 CPU(s):
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#

What did you expect to see?

No panic

What did you see instead?

Seen panic with fatal error: fault error

Activity

mkumatag

mkumatag commented on Feb 17, 2019

@mkumatag
mkumatag
on Feb 17, 2019
Author

/cc @laboger

mkumatag
mentioned this on Feb 17, 2019
mvdan

mvdan commented on Feb 17, 2019

@mvdan
mvdan
on Feb 17, 2019
Member

Does this happen with 1.11.5? If not, this is a regression that needs attention soon.

oschwald
mentioned this on Feb 17, 2019
mkumatag

mkumatag commented on Feb 17, 2019

@mkumatag
mkumatag
on Feb 17, 2019
Author

Does this happen with 1.11.5? If not, this is a regression that needs attention soon.

@mvdan tests are passing without any issues in 1.11.5 version, test results are attached here - https://gist.github.com/mkumatag/24f2068a803f36758f938da206312d9f

odeke-em

odeke-em commented on Feb 18, 2019

@odeke-em
odeke-em
on Feb 18, 2019
Member

Thank you for the report @mkumatag and @mvdan for the initial response.

@mkumatag, initially I couldn't reproduce your produce using the Docker container steps as some steps seem to be missing. However, after a plain:

go get github.com/containerd/containerd && \
cd $GOPATH/src/github.com/containerd/containerd/metadata && \
go test -buildmode=pie -v

I cannot get a crash either on the Docker container nor on my Macbook pro, where both are
using Go1.12rc1 and the instructions I've provided can be run on both the Docker container and on your computer.

mkumatag

mkumatag commented on Feb 18, 2019

@mkumatag
mkumatag
on Feb 18, 2019
Author

Thank you for the report @mkumatag and @mvdan for the initial response.

@mkumatag, initially I couldn't reproduce your produce using the Docker container steps as some steps seem to be missing. However, after a plain:

go get github.com/containerd/containerd && \
cd $GOPATH/src/github.com/containerd/containerd/metadata && \
go test -buildmode=pie -v

I cannot get a crash either on the Docker container nor on my Macbook pro, where both are
using Go1.12rc1 and the instructions I've provided can be run on both the Docker container and on your computer.

This is happening in the ppc64le architecture, lemme modify the description

mkumatag
changed the title [-]runtime: panic with fatal error(SIGSEGV)[/-] [+]runtime: panic with fatal error(SIGSEGV) on ppc64le architecture[/+] on Feb 18, 2019
thaJeztah

thaJeztah commented on Feb 18, 2019

@thaJeztah
thaJeztah
on Feb 18, 2019
Contributor

Just from looking at the location where this was failing (see moby/moby#38404 (comment)), these changes could be possible suspects: 05166bf and 020a18c

/cc @martisch

thaJeztah
mentioned this on Feb 18, 2019
laboger

laboger commented on Feb 18, 2019

@laboger
laboger
on Feb 18, 2019
Contributor

I have found that it starts to fail with commit 69c2c56. @randall77

The SEGV occurs because r2 has been corrupted during a call to github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable from github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill.

Before this change, the code at the end of github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable looks like this:

  599bf4:       08 00 05 e9     ld      r8,8(r5)
  599bf8:       08 00 08 e9     ld      r8,8(r8)
  599bfc:       10 00 e7 38     addi    r7,r7,16
  599c00:       80 00 08 e9     ld      r8,128(r8)
  599c04:       76 fe 09 7d     sradi   r9,r8,63
  599c08:       a0 17 29 79     rldicl  r9,r9,2,62
  599c0c:       14 4a 08 7d     add     r8,r8,r9
  599c10:       74 16 08 7d     sradi   r8,r8,2
  599c14:       00 40 27 7c     cmpd    r7,r8
  599c18:       24 00 81 41     bgt     599c3c <github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable+0x16c>
  599c1c:       28 00 01 e9     ld      r8,40(r1)
  599c20:       01 00 08 39     addi    r8,r8,1
  599c24:       00 30 28 7c     cmpd    r8,r6
  599c28:       14 ff 80 41     blt     599b3c <github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable+0x6c>
  599c2c:       01 00 60 38     li      r3,1
  599c30:       e0 00 61 98     stb     r3,224(r1)
  599c34:       b8 00 21 38     addi    r1,r1,184
  599c38:       20 00 80 4e     blr
  599c3c:       e0 00 01 98     stb     r0,224(r1)
  599c40:       b8 00 21 38     addi    r1,r1,184
  599c44:       20 00 80 4e     blr
  599c48:       e0 00 01 98     stb     r0,224(r1)
  599c4c:       b8 00 21 38     addi    r1,r1,184
  599c50:       20 00 80 4e     blr
  599c54:       e0 00 01 98     stb     r0,224(r1)
  599c58:       b8 00 21 38     addi    r1,r1,184
  599c5c:       20 00 80 4e     blr

After this change, the same code looks like this. It is incorrectly loading the value of r2 from 24(r1). That location has never been initialized with the value of r2 so some random value is being loaded and then used by the code when this function returns.

After this change, the same code looks like this:
  5aa1c4:       08 00 05 e9     ld      r8,8(r5)
  5aa1c8:       08 00 08 e9     ld      r8,8(r8)
  5aa1cc:       10 00 e7 38     addi    r7,r7,16
  5aa1d0:       78 03 00 7c     mr      r0,r0
  5aa1d4:       18 00 41 e8     ld      r2,24(r1)  <----- This load is wrong, r2 has never been saved here.
  5aa1d8:       80 00 08 e9     ld      r8,128(r8)
  5aa1dc:       76 fe 09 7d     sradi   r9,r8,63
  5aa1e0:       a0 17 29 79     rldicl  r9,r9,2,62
  5aa1e4:       14 4a 08 7d     add     r8,r8,r9
  5aa1e8:       74 16 08 7d     sradi   r8,r8,2
  5aa1ec:       00 40 27 7c     cmpd    r7,r8
  5aa1f0:       24 00 81 41     bgt     5aa214 <github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable+0x174>
  5aa1f4:       28 00 01 e9     ld      r8,40(r1)
  5aa1f8:       01 00 08 39     addi    r8,r8,1
  5aa1fc:       00 30 28 7c     cmpd    r8,r6
  5aa200:       0c ff 80 41     blt     5aa10c <github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable+0x6c>
  5aa204:       01 00 60 38     li      r3,1
  5aa208:       e0 00 61 98     stb     r3,224(r1)
  5aa20c:       b8 00 21 38     addi    r1,r1,184
  5aa210:       20 00 80 4e     blr
  5aa214:       e0 00 01 98     stb     r0,224(r1)
  5aa218:       b8 00 21 38     addi    r1,r1,184
  5aa21c:       20 00 80 4e     blr
  5aa220:       e0 00 01 98     stb     r0,224(r1)
  5aa224:       b8 00 21 38     addi    r1,r1,184
  5aa228:       20 00 80 4e     blr
  5aa22c:       e0 00 01 98     stb     r0,224(r1)
  5aa230:       b8 00 21 38     addi    r1,r1,184
  5aa234:       20 00 80 4e     blr
laboger

laboger commented on Feb 18, 2019

@laboger
laboger
on Feb 18, 2019
Contributor

FYI... I was able to reproduce the problem without building it in a container.

ianlancetaylor
added this to the Go1.12 milestone on Feb 19, 2019
ianlancetaylor
added
NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
release-blocker
on Feb 19, 2019

1 remaining item

laboger

laboger commented on Feb 19, 2019

@laboger
laboger
on Feb 19, 2019
Contributor

This same error happens in pkg math test Nextafter32 when build with -buildmode=pie

go test -buildmode=pie -test.run=Nextafter32
unexpected fault address 0xbffff3a4f8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xbffff3a4f8 pc=0x108bc53c8]

goroutine 5 [running]:
runtime.throw(0x108c67b5f, 0x5)
/home/boger/golang/baseline/go/src/runtime/panic.go:627 +0x68 fp=0xc00003fdf0 sp=0xc00003fdb0 pc=0x108b73608
runtime.sigpanic()
/home/boger/golang/baseline/go/src/runtime/signal_unix.go:397 +0x464 fp=0xc00003fe30 sp=0xc00003fdf0 pc=0x108b8a344
math.Nextafter32(0x41200000409f5411, 0x0)
/home/boger/golang/baseline/go/src/math/nextafter.go:19 +0x68 fp=0xc00003fe90 sp=0xc00003fe50 pc=0x108bc53c8
math_test.TestNextafter32(0xc0000ca100)
/home/boger/golang/baseline/go/src/math/all_test.go:2738 +0x8c fp=0xc00003ff70 sp=0xc00003fe90 pc=0x108c55a9c
testing.tRunner(0xc0000ca100, 0x108d21df0)
/home/boger/golang/baseline/go/src/testing/testing.go:865 +0xdc fp=0xc00003ffb0 sp=0xc00003ff70 pc=0x108c0b10c
runtime.goexit()
/home/boger/golang/baseline/go/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc00003ffb0 sp=0xc00003ffb0 pc=0x108ba4874
created by testing.(*T).Run
/home/boger/golang/baseline/go/src/testing/testing.go:916 +0x304

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000ca100, 0x108c69cfe, 0xf, 0x108d21df0, 0x108df7f01)
/home/boger/golang/baseline/go/src/testing/testing.go:917 +0x320
testing.runTests.func1(0xc0000ca000)
/home/boger/golang/baseline/go/src/testing/testing.go:1157 +0x8c
testing.tRunner(0xc0000ca000, 0xc0000a7dd8)
/home/boger/golang/baseline/go/src/testing/testing.go:865 +0xdc
testing.runTests(0xc00000e0a0, 0x108df7900, 0x46, 0x46, 0x0)
/home/boger/golang/baseline/go/src/testing/testing.go:1155 +0x2a0
testing.(*M).Run(0xc00000c080, 0x0)
/home/boger/golang/baseline/go/src/testing/testing.go:1072 +0x174
main.main()
_testmain.go:366 +0x150
exit status 2
FAIL math 0.005s

eclipseo

eclipseo commented on Feb 20, 2019

@eclipseo
eclipseo
on Feb 20, 2019

I am a Fedora maintainer for Go and during our latest mass rebuild with 1.12 rc1, we encountered several packages segfaulting on ppc64le during tests.

For example:

+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
unexpected fault address 0xffffffffffd02af8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffd02af8
pc=0x135db8c64]
goroutine 22 [running]:
runtime.throw(0x135dc7cbc, 0x5)
        /usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc0000afe58
sp=0xc0000afe18 pc=0x135b7a4d8
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc0000afe98
sp=0xc0000afe58 pc=0x135b92d74
time.(*Time).unixSec(...)
        /usr/lib/golang/src/time/time.go:176
time.Time.Unix(...)
        /usr/lib/golang/src/time/time.go:1146
github.com/pengsrc/go-shared/convert.TimeToTimestamp(0x0, 0xecf59cff8,
0xc000092600, 0xf)
       
/builddir/build/BUILD/go-shared-1ef04317652833067e47e2ee9815f1f254a7a1da/_build/src/github.com/pengsrc/go-shared/convert/time.go:44
+0x74 fp=0xc0000afef0 sp=0xc0000afeb8 pc=0x135db8c64
github.com/pengsrc/go-shared/convert.TestTimeToTimestamp(0xc000104400)
       
/builddir/build/BUILD/go-shared-1ef04317652833067e47e2ee9815f1f254a7a1da/_build/src/github.com/pengsrc/go-shared/convert/time_test.go:59
+0xe0 fp=0xc0000aff70 sp=0xc0000afef0 pc=0x135dbc530
testing.tRunner(0xc000104400, 0x135f64b28)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc0000affb0
sp=0xc0000aff70 pc=0x135c26fec
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0000affb0
sp=0xc0000affb0 pc=0x135bacf44
created by testing.(*T).Run
        /usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc000104400, 0x135dcbd36, 0x13, 0x135f64b28, 0x136127f01)
        /usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc000104000)
        /usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc000104000, 0xc0000addd8)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00009c380, 0x136127f60, 0x2d, 0x2d, 0x0)
        /usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc000102000, 0x0)
        /usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
        _testmain.go:130 +0x150
exit status 2
FAIL    github.com/pengsrc/go-shared/convert    0.015s

+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
unexpected fault address 0x3b400
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3b400 pc=0x100cdec34]
goroutine 7 [running]:
runtime.throw(0x100d9924f, 0x5)
        /usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc00004ba88
sp=0xc00004ba48 pc=0x100cadc18
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc00004bac8
sp=0xc00004ba88 pc=0x100cc4964
runtime.memmove(0xc00004bdd8, 0x3b400, 0x28)
        /usr/lib/golang/src/runtime/memmove_ppc64x.s:57 +0x64 fp=0xc00004bae8
sp=0xc00004bae8 pc=0x100cdec34
github.com/keybase/go-crypto/ed25519/internal/edwards25519.FeZero(...)
       
/builddir/build/BUILD/go-crypto-670ebd3adf7a737d69ffe83a777a8e34eadc1b32/_build/src/github.com/keybase/go-crypto/ed25519/internal/edwards25519/edwards25519.go:21
github.com/keybase/go-crypto/ed25519/internal/edwards25519.(*ExtendedGroupElement).Zero(...)
       
/builddir/build/BUILD/go-crypto-670ebd3adf7a737d69ffe83a777a8e34eadc1b32/_build/src/github.com/keybase/go-crypto/ed25519/internal/edwards25519/edwards25519.go:683
github.com/keybase/go-crypto/ed25519/internal/edwards25519.GeScalarMultBase(0xc00004bdd8,
0xc00004bd78)
       
/builddir/build/BUILD/go-crypto-670ebd3adf7a737d69ffe83a777a8e34eadc1b32/_build/src/github.com/keybase/go-crypto/ed25519/internal/edwards25519/edwards25519.go:989
+0x140 fp=0xc00004bd00 sp=0xc00004bae8 pc=0x100d92d40
github.com/keybase/go-crypto/ed25519.Sign(0xc000022080, 0x40, 0x40,
0xc000018170, 0xc, 0xc, 0x40, 0x40, 0x0)
       
/builddir/build/BUILD/go-crypto-670ebd3adf7a737d69ffe83a777a8e34eadc1b32/_build/src/github.com/keybase/go-crypto/ed25519/ed25519.go:120
+0x238 fp=0xc00004bea8 sp=0xc00004bd00 pc=0x100d95aa8
github.com/keybase/go-crypto/ed25519.TestSignVerify(0xc0000aa200)
       
/builddir/build/BUILD/go-crypto-670ebd3adf7a737d69ffe83a777a8e34eadc1b32/_build/src/github.com/keybase/go-crypto/ed25519/ed25519_test.go:53
+0xe4 fp=0xc00004bf70 sp=0xc00004bea8 pc=0x100d964d4
testing.tRunner(0xc0000aa200, 0x100e57778)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc00004bfb0
sp=0xc00004bf70 pc=0x100d3c60c
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc00004bfb0
sp=0xc00004bfb0 pc=0x100cde8e4
created by testing.(*T).Run
        /usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000aa200, 0x100d9ab73, 0xe, 0x100e57778, 0x100f27f01)
        /usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc0000aa000)
        /usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc0000aa000, 0xc00009bdd8)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00000e0e0, 0x100f24c00, 0x5, 0x5, 0x0)
        /usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc00000c080, 0x0)
        /usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
        _testmain.go:56 +0x150
exit status 2
FAIL    github.com/keybase/go-crypto/ed25519    0.007s

+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro
-Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
unexpected fault address 0x115912550
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x115912550
pc=0x1155813ec]
goroutine 8 [running]:
runtime.throw(0x1158de6bf, 0x5)
        /usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc00004e908
sp=0xc00004e8c8 pc=0x115581998
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc00004e948
sp=0xc00004e908 pc=0x11559a5d4
panic(0x1155ade94, 0x115623884)
        /usr/lib/golang/src/runtime/panic.go:490 +0xcc fp=0xc00004ea08
sp=0xc00004e968 pc=0x1155813ec
runtime.panicmakeslicelen(...)
        /usr/lib/golang/src/runtime/slice.go:27
runtime.makeslice(0x115587854, 0xc2, 0xc2, 0x8)
        /usr/lib/golang/src/runtime/slice.go:44 +0x1a4 fp=0xc00004ea48
sp=0xc00004ea08 pc=0x11559bab4
github.com/boltdb/bolt.(*Bucket).write(0xc000060580, 0xc00010a901,
0xc00004ec48, 0x115aa0300)
        /usr/share/gocode/src/github.com/boltdb/bolt/bucket.go:619 +0x6c
fp=0xc00004eaa0 sp=0xc00004ea48 pc=0x11569de3c
github.com/boltdb/bolt.(*Bucket).spill(0xc000176398, 0x884a882, 0x115ded400)
        /usr/share/gocode/src/github.com/boltdb/bolt/bucket.go:535 +0xe4
fp=0xc00004eca8 sp=0xc00004eaa0 pc=0x11569d6b4
github.com/boltdb/bolt.(*Tx).Commit(0xc000176380, 0xc000018480, 0x8)
        /usr/share/gocode/src/github.com/boltdb/bolt/tx.go:163 +0xd0
fp=0xc00004ee20 sp=0xc00004eca8 pc=0x1156a9ec0
github.com/hashicorp/raft-boltdb.(*BoltStore).StoreLogs(0xc00000e360,
0xc00004ef58, 0x3, 0x3, 0x0, 0x0)
       
/builddir/build/BUILD/raft-boltdb-d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee/_build/src/github.com/hashicorp/raft-boltdb/bolt_store.go:157
+0x214 fp=0xc00004eec0 sp=0xc00004ee20 pc=0x1158d7cd4
github.com/hashicorp/raft-boltdb.TestBoltStore_FirstIndex(0xc00012a300)
       
/builddir/build/BUILD/raft-boltdb-d1e82c1ec3f15ee991f7cc7ffd5b67ff6f5bbaee/_build/src/github.com/hashicorp/raft-boltdb/bolt_store_test.go:111
+0x300 fp=0xc00004ef70 sp=0xc00004eec0 pc=0x1158d9fb0
testing.tRunner(0xc00012a300, 0x115b1dd08)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc00004efb0
sp=0xc00004ef70 pc=0x11564af7c
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc00004efb0
sp=0xc00004efb0 pc=0x1155b51c4
created by testing.(*T).Run
        /usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc00012a300, 0x1158e9239, 0x18, 0x115b1dd08, 0x115db7f01)
        /usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc00012a000)
        /usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc00012a000, 0xc000119dd8)
        /usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00000e140, 0x115db7400, 0xa, 0xa, 0x0)
        /usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc00000c080, 0x0)
        /usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
        _testmain.go:80 +0x150
goroutine 4 [syscall]:
os/signal.signal_recv(0x0)
        /usr/lib/golang/src/runtime/sigqueue.go:139 +0x10c
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x38
created by os/signal.init.0
        /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x50
exit status 2
FAIL    github.com/hashicorp/raft-boltdb        0.147s

Do you think it could be related to this bug?

laboger

laboger commented on Feb 21, 2019

@laboger
laboger
on Feb 21, 2019
Contributor

Yes, anything built for PIC (i.e., using -buildmode=pie or -shared) could hit this bug. The commit that I noted above is adding the load of r2 from a location that might not have been initialized, and if that is used to generate an address then a SEGV can definitely occur.

It doesn't look like anything has been happening here, I have a fix for it but not sure what the process is at this point. Submit into the go1.12 branch only or upstream too?

gopherbot

gopherbot commented on Feb 21, 2019

@gopherbot
gopherbot
on Feb 21, 2019
Contributor

Change https://golang.org/cl/163337 mentions this issue: cmd/compile: use correct NOP on ppc64le for stacktraces with inlines

laboger

laboger commented on Feb 21, 2019

@laboger
laboger
on Feb 21, 2019
Contributor

@mkumatag Can you test this out to verify it fixes your issue? I verified it on the cases I was aware of.

laboger

laboger commented on Feb 21, 2019

@laboger
laboger
on Feb 21, 2019
Contributor

The headline is misleading, the problem isn't with the inlining but with the insertion of NOPs to fix stacktraces of inlined functions.

ianlancetaylor

ianlancetaylor commented on Feb 21, 2019

@ianlancetaylor
ianlancetaylor
on Feb 21, 2019
Contributor

@laboger Submit your fix to master, and when it is approved and submitted we can cherry pick it to the 1.12 branch.

ianlancetaylor

ianlancetaylor commented on Feb 21, 2019

@ianlancetaylor
ianlancetaylor
on Feb 21, 2019
Contributor

Not that it matters, but the NOPs are a result of adding support for mid-stack inlining, so it seems to me that the subject line isn't completely wrong. But of course feel free to edit it.

mkumatag

mkumatag commented on Feb 22, 2019

@mkumatag
mkumatag
on Feb 22, 2019
Author

@mkumatag Can you test this out to verify it fixes your issue? I verified it on the cases I was aware of.

@laboger I ran the containerd tests(tests, integration) with the submitted patch. I don't see any issues.

test logs: https://gist.github.com/mkumatag/dd0cafcf968700119da401e6a652683f

gopherbot
closed this as completedin 2d34740on Feb 25, 2019
gopherbot

gopherbot commented on Feb 25, 2019

@gopherbot
gopherbot
on Feb 25, 2019
Contributor

Change https://golang.org/cl/163717 mentions this issue: [release-branch.go1.12] cmd/compile: call ginsnop, not ginsnop2 on ppc64le for mid-stack inlining tracebacks

gopherbot
added a commit that references this issue on Feb 25, 2019

[release-branch.go1.12] cmd/compile: call ginsnop, not ginsnop2 on pp…

efc6d86
laboger
mentioned this on Feb 28, 2019
golang
locked and limited conversation to collaborators on Feb 25, 2020
gopherbot
added
FrozenDueToAge
on Feb 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.release-blocker

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @thaJeztah@ianlancetaylor@mvdan@odeke-em@gopherbot

        Issue actions
          runtime: r2 corrupted by mid-stack inlining on ppc64le · Issue #30283 · golang/go