runtime: unexpected return pc with go1.15 and go1.14 but not go1.13

[bigdrum]

We found this bug (random crash) in real world system but managed to reduce a simple example.

The follow sample should not crash but it does with go1.15.7, go.1.14.14:
https://github.com/bigdrum/deferbug/blob/master/deferbug_test.go

There is possibility of this being a bug in the gin framework. But this isn't re-producible in go1.13.15.
So I suspect this is a regression introduced by the new defer optimization introduced in go1.14.

What version of Go are you using (go version)?

$ go version
go version go1.15.7 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/foo/Library/Caches/go-build"

GOENV="/Users/foo/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOINSECURE=""

GOMODCACHE="/Users/foo/go/pkg/mod"

GOOS="darwin"

GOPATH="/Users/foo/go"

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/foo/build/deferbug/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yl/h53cqly57t911sx35j5wbjpm0000gn/T/go-build995529195=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Run the follow test created as a minimum reproducing code:

https://github.com/bigdrum/deferbug

What did you expect to see?

No error.

What did you see instead?

runtime: unexpected return pc for github.com/gin-gonic/gin.RecoveryWithWriter.func1 called from 0x7160f70
stack: frame={sp:0xc0000f7b70, fp:0xc0000f7bb0} stack=[0xc0000f6000,0xc0000f8000)
000000c0000f7a70: 0000000001c3ffff 0000000007160f70
000000c0000f7a80: 0000000007160f70 0000000000000030
000000c0000f7a90: 0000000000000001 000000c0000f7b10
000000c0000f7aa0: 000000000101a625 <runtime.(*mcentral).cacheSpan+517> 0000000007160f70
000000c0000f7ab0: 0000000000000000 000000c0000f7a78
000000c0000f7ac0: 000000000000006c 000000c0000a4060
000000c0000f7ad0: 0000000001464be0 0000000000000000
...

(see https://github.com/bigdrum/deferbug/blob/master/README.md for full output)

[ALTree]

Possibly a dup of #43496 (but this one has a reproducer, that one doesn't).

[ALTree]

cc @danscales

[ALTree]

git bisect points to dad6163 (make defers low-cost through inline code and extra funcdata).

[danscales]

Thanks very much for the report and reproducible case, @bigdrum and @ALTree ! I will investigate in the next day or so.

[mdempsky]

Standalone reproducer: https://play.golang.org/p/vUxBrvpouTu

Further minimized: https://play.golang.org/p/vGyzJCZ2Ssq

(I'm not planning to investigate this any further. I just felt in the mood for minimizing a compiler test case.)

[ianlancetaylor]

This is crash on valid code, but it already fails with 1.14 and 1.15 so not marking as a release blocker for 1.16. Still, if we can fix it for 1.16, that would be nice.

[gopherbot]

Change https://golang.org/cl/286712 mentions this issue: runtime: make sure to remove open-coded defer entries in all cases after a recover

[danscales]

Thanks for the minimized test case, @mdempsky ! I reduced it further and added it as a test case, and now have a fix.

The fix is reasonably small and effects should be minimal, but the runtime handling of open-coded defers is a bit tricky. So, we can decide if we have enough time to test for 1.16, based on reviews, etc.

[dmitshur]

I've updated the milestone to reflect that this fix landed in Go 1.16 after all.