runtime: sweep increased allocation count crash on arm64

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.13.3 linux/arm64
</pre>
we applied some patch  to go1.13.3:
```
2019-08-06 02:42 b25ec50b69  syscall: implement rawVforkSyscall for linux/arm64
2019-09-11 02:26 904f046e2b  runtime: fix crash during VDSO calls on arm
2019-10-23 11:42 758eb020f7  runtime: save/fetch g register during VDSO on ARM and ARM64
2019-10-31 10:32 f07cbc7f88  runtime: don't fetch G from signal stack when using cgo
2019-11-10 13:18 75c839af22  runtime: don't save G during VDSO if we're handling signal
2019-10-28 09:29 0ae9389609  runtime: fix textOff for multiple text sections
```
tested using go1.11.2 and go1.13.3, crash occurs on both golang version.

### Does this issue reproduce with the latest release?

not tested, the go1.13.5 shall reproducible. I'll try master branch soon.

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build956154425=/tmp/go-build -gno-record-gcc-switches"

</pre></details>

### What did you do?


1. build and install docker-engine(we use docker-ce 18.09) using go1.13.3 on arm64 environment. linux kernel 4.19.36. We can reproduce using physical machine, but not easy on virtual machines.
2. run container using follow command, then wait a long time.
```
#!/bin/bash

for ((i=0;i<100;i++)); do
docker run -tid --name=jj-$i --health-interval 1s --health-cmd date ubuntu-arm64 bash
done
```
3. some containers will exit, and core files generated.
![image](https://user-images.githubusercontent.com/51155955/70700244-67b24600-1d05-11ea-8e65-472e8adf833a.png)

we get 2 core files core_containerd-shim_31862_1576112309 core_containerd-shim_53653_1576133943
```
gdb ./containerd-shim core_containerd-shim_31862_1576112309
(gdb) bt
#0  runtime.raise () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:156
#1  0x00000000000517b8 in runtime.dieFromSignal (sig=6) at /usr/lib/golang/src/runtime/signal_unix.go:454
#2  0x0000000000051d70 in runtime.sigfwdgo (sig=6, info=0x4000106da0, ctx=0x4000106e20, ~r3=<optimized out>) at /usr/lib/golang/src/runtime/signal_unix.go:657
#3  0x0000000000050f78 in runtime.sigtrampgo (sig=<optimized out>, info=0x4000106da0, ctx=0x4000106e20) at /usr/lib/golang/src/runtime/signal_unix.go:316
#4  0x0000000000069438 in runtime.sigtramp () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:404
#5  <signal handler called>
#6  runtime.raise () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:156
#7  0x00000000000517b8 in runtime.dieFromSignal (sig=6) at /usr/lib/golang/src/runtime/signal_unix.go:454
#8  0x0000000000051978 in runtime.crash () at /usr/lib/golang/src/runtime/signal_unix.go:546
#9  0x0000000000064988 in runtime.fatalthrow.func1 () at /usr/lib/golang/src/runtime/panic.go:833
#10 0x000000000003c520 in runtime.fatalthrow () at /usr/lib/golang/src/runtime/panic.go:826
#11 0x000000000003c384 in runtime.throw (s=...) at /usr/lib/golang/src/runtime/panic.go:774
#12 0x0000000000031838 in runtime.(*mspan).sweep (s=0xffff86a10b88, preserve=false, ~r1=<optimized out>) at /usr/lib/golang/src/runtime/mgcsweep.go:328
#13 0x0000000000025ab0 in runtime.(*mcentral).uncacheSpan (c=0x5a64f8 <runtime.mheap_+8760>, s=0xffff86a10b88) at /usr/lib/golang/src/runtime/mcentral.go:197
#14 0x0000000000025390 in runtime.(*mcache).releaseAll (c=0xffff86a046d0) at /usr/lib/golang/src/runtime/mcache.go:158
#15 0x000000000002541c in runtime.(*mcache).prepareForSweep (c=0xffff86a046d0) at /usr/lib/golang/src/runtime/mcache.go:185
#16 0x0000000000047c98 in runtime.acquirep (_p_=0x4000022500) at /usr/lib/golang/src/runtime/proc.go:4117
#17 0x0000000000042188 in runtime.stopm () at /usr/lib/golang/src/runtime/proc.go:1930
#18 0x0000000000042bfc in runtime.gcstopm () at /usr/lib/golang/src/runtime/proc.go:2125
#19 0x000000000004428c in runtime.schedule () at /usr/lib/golang/src/runtime/proc.go:2481
#20 0x0000000000044608 in runtime.goschedImpl (gp=0x40001b9c80) at /usr/lib/golang/src/runtime/proc.go:2625
#21 0x0000000000044838 in runtime.gopreempt_m (gp=0x40001b9c80) at /usr/lib/golang/src/runtime/proc.go:2653
#22 0x0000000000054924 in runtime.newstack () at /usr/lib/golang/src/runtime/stack.go:1038
#23 0x0000000000066130 in runtime.morestack () at /usr/lib/golang/src/runtime/asm_arm64.s:310
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

325         nfreed := s.allocCount - nalloc
326         if nalloc > s.allocCount {
327                 print("runtime: nelems=", s.nelems, " nalloc=", nalloc, " previous allocCount=", s.allocCount, " nfreed=", nfreed, "\n")
**328                 throw("sweep increased allocation count")**
329         }


(gdb) x /100 0xffff86a10b88
0xffff86a10b88: 0x00000000      0x00000000      0x00000000      0x00000000
next=0 prev=0
0xffff86a10b98: 0x005a6508      0x00000000      0x00238000      0x00000040
list=0x005a6508 startAddr=0x00000040/0x00238000
0xffff86a10ba8: 0x00000005      0x00000000      0x00000000      0x00000000
npages=5 manualFreeList=0
0xffff86a10bb8: 0x00000001      0x00000000      0x00000002      0x00000000
freeindex=1 nelems=2
0xffff86a10bc8: 0xffffffff      0x7fffffff      0x845d3c18      0x0000ffff
allocCache=0x7fffffff/0xffffffff allocBits=0x0000ffff/0x845d3c18
0xffff86a10bd8: 0x845d3c10      0x0000ffff      0x00069c41      0x00000007
gcmarkBits=0x0000ffff/0x845d3c10 sweepgen=7 divMul=6 baseMask=0x9c41
0xffff86a10be8: 0x017a0001      0x00050c00      0x00005000      0x00000000
allocCount=5 spanClass=0xc state=0 needzero=1 divShift=0x7a divShift2=0 scavenged=true
elemsize=0x5000
0xffff86a10bf8: 0x00242000      0x00000040      0x00000000      0x00000000
limit=0x00000040/0x00242000 speciallock=0
0xffff86a10c08: 0x00000000      0x00000000      0x00000000      0x00000000
specials=0


```

```
gdb ./containerd-shim core_containerd-shim_53653_1576133943
(gdb) bt
#0  runtime.raise () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:156
#1  0x00000000000517b8 in runtime.dieFromSignal (sig=6) at /usr/lib/golang/src/runtime/signal_unix.go:454
#2  0x0000000000051d70 in runtime.sigfwdgo (sig=6, info=0x400015eda0, ctx=0x400015ee20, ~r3=<optimized out>) at /usr/lib/golang/src/runtime/signal_unix.go:657
#3  0x0000000000050f78 in runtime.sigtrampgo (sig=<optimized out>, info=0x400015eda0, ctx=0x400015ee20) at /usr/lib/golang/src/runtime/signal_unix.go:316
#4  0x0000000000069438 in runtime.sigtramp () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:404
#5  <signal handler called>
#6  runtime.raise () at /usr/lib/golang/src/runtime/sys_linux_arm64.s:156
#7  0x00000000000517b8 in runtime.dieFromSignal (sig=6) at /usr/lib/golang/src/runtime/signal_unix.go:454
#8  0x0000000000051978 in runtime.crash () at /usr/lib/golang/src/runtime/signal_unix.go:546
#9  0x0000000000064988 in runtime.fatalthrow.func1 () at /usr/lib/golang/src/runtime/panic.go:833
#10 0x0000000000066080 in runtime.systemstack () at /usr/lib/golang/src/runtime/asm_arm64.s:237
#11 0x0000000000040970 in runtime.startTheWorldWithSema (emitTraceEvent=false, ~r1=<optimized out>) at /usr/lib/golang/src/runtime/proc.go:1080
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

1078 }
1079
**1080 func startTheWorldWithSema(emitTraceEvent bool) int64 {**
1081         mp := acquirem() // disable preemption because it can be holding p in a local var
1082         if netpollinited() {
1083                 list := netpoll(false) // non-blocking
1084                 injectglist(&list)
1085         }

 223         // switch to g0
 224         MOVD    R5, g
 225         BL      runtime·save_g(SB)
 226         MOVD    (g_sched+gobuf_sp)(g), R3
 227         // make it look like mstart called systemstack on g0, to stop traceback
 228         SUB     $16, R3
 229         AND     $~15, R3
 230         MOVD    $runtime·mstart(SB), R4
 231         MOVD    R4, 0(R3)
 232         MOVD    R3, RSP
 233         MOVD    (g_sched+gobuf_bp)(g), R29
 234
 235         // call target function
 236         MOVD    0(R26), R3      // code pointer
 **237         BL      (R3)**

```
### What did you expect to see?

containerd-shim not crash and no core files generated.

### What did you see instead?

containerd-shim crash on arm64.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

runtime: sweep increased allocation count crash on arm64 #36101

What version of Go are you using (`go version`)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

What did you do?

What did you expect to see?

What did you see instead?

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Participants

runtime: sweep increased allocation count crash on arm64 #36101

Description

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Activity

ALTree commented on Dec 12, 2019

jing-rui commented on Dec 13, 2019

ianlancetaylor commented on Dec 13, 2019

jing-rui commented on Dec 13, 2019

ianlancetaylor commented on Dec 14, 2019

jing-rui commented on Dec 17, 2019

jing-rui commented on Dec 18, 2019

jing-rui commented on Dec 19, 2019

jing-rui commented on Dec 19, 2019

jing-rui commented on Dec 23, 2019

jing-rui commented on Dec 26, 2019

cherrymui commented on Dec 27, 2019

19 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Participants

Issue actions

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?