plugin: tls handshake panic: unreachable method called. linker bug?

[oszika]

What version of Go are you using (go version)?

$ go version
go version go1.17.8 linux/amd64

Does this issue reproduce with the latest release?

Reproduced with go1.18rc1.
Not reproduced with go1.16.15

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/oszika/.cache/go-build"

GOENV="/home/oszika/.config/go/env"

GOEXE=""

GOEXPERIMENT=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE="/home/oszika/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/oszika/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/lib/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"

GOVCS=""

GOVERSION="go1.17.8"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/path/to/module/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1435856803=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I build a plugin that calls http.ListenAndServeTLS(). The program that loads the plugin does something with a tls.Config{}.
Then I tried to make a tls connection. In this example, the code about the tls.Config in the main program is not called because the function loaded from the plugin is blocking.

Plugin:

package main

import (
	"log"
	"net/http"
)

func Listen() {
	srv := &http.Server{
		Addr: "127.0.0.1:4443",
	}
	log.Fatal(srv.ListenAndServeTLS("/path/to/cert.pem", "/path/to/key.pem"))
}

Build plugin: go build -buildmode=plugin -o plugin.so

Main file:

package main

import (
	"crypto/tls"
	"fmt"
	"plugin"
)

func main() {
	p, err := plugin.Open("./plugin/plugin.so")
	if err != nil {
		panic(err)
	}

	s, err := p.Lookup("Listen")
	if err != nil {
		panic(err)
	}

	s.(func())() // Blocking Listen call

	fmt.Println(&tls.Config{}) // Do something with tls.Config
}

TLS connection: openssl s_client -connect 127.0.0.1:4443

What did you expect to see?

No error. A successful connection.

What did you see instead?

> go run main.go

fatal error: unreachable method called. linker bug? Trace

goroutine 19 [running]:
runtime.throw({0x64d41d, 0xc00013c340})
	/usr/lib/go/src/runtime/panic.go:1198 +0x71 fp=0xc00012b418 sp=0xc00012b3e8 pc=0x4bf0b1
runtime.unreachableMethod()
	/usr/lib/go/src/runtime/iface.go:561 +0x25 fp=0xc00012b438 sp=0xc00012b418 pc=0x4972c5
crypto/tls.(*halfConn).explicitNonceLen(0xc000119368)
	/usr/lib/go/src/crypto/tls/conn.go:251 +0xc2 fp=0xc00012b480 sp=0xc00012b438 pc=0x7f4c2b4560a2
crypto/tls.(*halfConn).encrypt(0xc000119368, {0xc00017e280, 0x5, 0x80}, {0xc00013e868, 0xc00012b700, 0x8}, {0x682ae0, 0xc000114180})
	/usr/lib/go/src/crypto/tls/conn.go:471 +0x126 fp=0xc00012b670 sp=0xc00012b480 pc=0x7f4c2b456da6
crypto/tls.(*Conn).writeRecordLocked(0xc000119180, 0x16, {0xc00013e868, 0x6, 0x8})
	/usr/lib/go/src/crypto/tls/conn.go:976 +0x32e fp=0xc00012b748 sp=0xc00012b670 pc=0x7f4c2b45b34e
crypto/tls.(*Conn).writeRecord(0xc00017e200, 0x68, {0xc00013e868, 0x8, 0x7f4c2b501458})
	/usr/lib/go/src/crypto/tls/conn.go:1002 +0xe5 fp=0xc00012b7c8 sp=0xc00012b748 pc=0x7f4c2b45b785
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc00012b990)
	/usr/lib/go/src/crypto/tls/handshake_server_tls13.go:562 +0x477 fp=0xc00012b890 sp=0xc00012b7c8 pc=0x7f4c2b484b57
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc00012b990)
	/usr/lib/go/src/crypto/tls/handshake_server_tls13.go:58 +0x72 fp=0xc00012b8b0 sp=0xc00012b890 pc=0x7f4c2b482612
crypto/tls.(*Conn).serverHandshake(0xc000119180, {0x7f4c2b5dc738, 0xc000178200})
	/usr/lib/go/src/crypto/tls/handshake_server.go:54 +0xc5 fp=0xc00012ba88 sp=0xc00012b8b0 pc=0x7f4c2b47d665
crypto/tls.(*Conn).serverHandshake-fm({0x7f4c2b5dc738, 0xc000178200})
	/usr/lib/go/src/crypto/tls/handshake_server.go:42 +0x39 fp=0xc00012bab0 sp=0xc00012ba88 pc=0x7f4c2b490299
crypto/tls.(*Conn).handshakeContext(0xc000119180, {0x7f4c2b5dc7e0, 0xc000115410})
	/usr/lib/go/src/crypto/tls/conn.go:1445 +0x3e2 fp=0xc00012bb80 sp=0xc00012bab0 pc=0x7f4c2b45e322
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/go/src/crypto/tls/conn.go:1395
net/http.(*conn).serve(0xc000124b40, {0x7f4c2b5dc7e0, 0xc000115350})
	/usr/lib/go/src/net/http/server.go:1818 +0x231 fp=0xc00012bfb8 sp=0xc00012bb80 pc=0x7f4c2b4e5cd1
net/http.(*Server).Serve·dwrap·87()
	/usr/lib/go/src/net/http/server.go:3034 +0x2e fp=0xc00012bfe0 sp=0xc00012bfb8 pc=0x7f4c2b4eb42e
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00012bfe8 sp=0xc00012bfe0 pc=0x7f4c2b2e0da1
created by net/http.(*Server).Serve
	/usr/lib/go/src/net/http/server.go:3034 +0x50f
goroutine 1 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c3c3ed7d0, 0x72)

/usr/lib/go/src/runtime/netpoll.go:234 +0x89

internal/poll.(*pollDesc).wait(0xc00019a000, 0x0, 0x0)

/usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

/usr/lib/go/src/internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Accept(0xc00019a000)

/usr/lib/go/src/internal/poll/fd_unix.go:402 +0x23d

net.(*netFD).accept(0xc00019a000)

/usr/lib/go/src/net/fd_unix.go:173 +0x35

net.(*TCPListener).accept(0xc000120120)

/usr/lib/go/src/net/tcpsock_posix.go:140 +0x28

net.(*TCPListener).Accept(0xc000120120)

/usr/lib/go/src/net/tcpsock.go:262 +0x3d

crypto/tls.(*listener).Accept(0xc000120258)

/usr/lib/go/src/crypto/tls/tls.go:66 +0x2d

net/http.(*Server).Serve(0xc000198000, {0x7f4c2b5db278, 0xc000120258})

/usr/lib/go/src/net/http/server.go:3002 +0x3ae

net/http.(*Server).ServeTLS(0xc000198000, {0x7f4c2b5db338, 0xc000120120}, {0x7f4c2b501d6f, 0xe0}, {0x7f4c2b5019bd, 0xe})

/usr/lib/go/src/net/http/server.go:3074 +0x3f0

net/http.(*Server).ListenAndServeTLS(0xc000198000, {0x7f4c2b501d6f, 0xf}, {0x7f4c2b5019bd, 0xe})

/usr/lib/go/src/net/http/server.go:3229 +0x12f

bug/plugin.Listen()

/path/to/main.go:75 +0x54

main.main()

/path/to/main.go:36 +0x127

exit status 2

[zhouguangyuan0718]

A simpler reproduction:
---- go.mod----

module demo2

go 1.17

---- main.go----

package main

import (
	"demo2/pkg"
	"fmt"
	"plugin"
)

func main() {
	p, err := plugin.Open("./plugin.so")
	if err != nil {
		panic(err)
	}

	s, err := p.Lookup("F")
	if err != nil {
		panic(err)
	}

	s.(func())() // Blocking Listen call

	fmt.Println(pkg.T(1)) // Do something with tls.Config
}

----plugin.go----

package main

import "demo2/pkg"

func F() {
	pkg.F()
}

---- pkg/pkg.go----

package pkg

func F() {
	var v interface{} = new(T)
	v.(I).m()

}

type I interface {
	m()
}

type T int

func (*T) m() {}

After using go build -trimpath -buildmode=plugin -o plugin.so and go build -trimpath to build and run it. Will get:

fatal error: unreachable method called. linker bug?

goroutine 1 [running]:
runtime.throw({0x7f25026dea85, 0xc000148000})
        runtime/panic.go:1198 +0x71 fp=0xc000056eb0 sp=0xc000056e80 pc=0x7f2502645fd1
runtime.unreachableMethod()
        runtime/iface.go:563 +0x25 fp=0xc000056ed0 sp=0xc000056eb0 pc=0x7f250261dbe5
demo2/pkg.F(...)
        demo2/pkg/pkg.go:5
demo2.F()
        demo2/plugin.go:6 +0x48 fp=0xc000056f00 sp=0xc000056ed0 pc=0x7f25026d6568
main.main()
        demo2/main.go:20 +0x10d fp=0xc000056f80 sp=0xc000056f00 pc=0x51af6d
runtime.main()
        runtime/proc.go:255 +0x227 fp=0xc000056fe0 sp=0xc000056f80 pc=0x48b5a7
runtime.goexit()
        runtime/asm_amd64.s:1581 +0x1 fp=0xc000056fe8 sp=0xc000056fe0 pc=0x4b6881

The type (*T) in host binary is:

00000000005296c0 :
  5296c0:       08 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00     ................
  5296d0:       ff 0b 21 3b 09 08 08 36 28 57 54 00 00 00 00 00     ..!;...6(WT.....
  5296e0:       f0 14 56 00 00 00 00 00 c8 0a 00 00 00 00 00 00     ..V.............
  5296f0:       00 48 52 00 00 00 00 00 58 17 00 00 01 00 00 00     .HR.....X.......
  529700:       10 00 00 00 00 00 00 00 1d 00 00 00 e0 85 00 00     ................
  529710:       ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00 00     ................

But in the plugin:

00000000001bc120 :
  1bc120:       08 00 00 00 00 00 00 00 08 00 00 00 00 00 00 00     ................
  1bc130:       ff 0b 21 3b 09 08 08 36 00 00 00 00 00 00 00 00     ..!;...6........
                        1bc138: R_X86_64_64     runtime.memequal64·f@@Base
  1bc140:       28 bd 1a 00 00 00 00 00 c0 0a 00 00 00 00 00 00     (...............
                        1bc140: R_X86_64_RELATIVE       *ABS*+0x1abd28
  1bc150:       00 00 00 00 00 00 00 00 20 17 00 00 01 00 00 00     ........ .......
                        1bc150: R_X86_64_64     type.e1XyHGHH@@Base
  1bc160:       10 00 00 00 00 00 00 00 22 00 00 00 a0 84 00 00     ........".......
  1bc170:       40 17 06 00 40 17 06 00 00 00 00 00 00 00 00 00     @...@...........

Seems like the method M of *T in host binary is marked as unreacheble, so the ifn and tfn of it is "ff ff".

[zhouguangyuan0718]

Maybe reloc type R_METHODOFF should be mark as reachable for "DynlinkingGo".
Or we need a new way to deal with the dynamic added type and itab, a little related with #48532 (comment)

[heschi]

cc @golang/runtime

[he7850]

meet same issue.
Also find another example reproducible under go1.17.7/1.18 arm64/x64: https://github.com/zoncoen-sample/go1.17-linker-issue

[aclements]

@cherrymui probably has the most context on this.

[cherrymui]

Thanks for report. When plugin is used, we already mark all exported methods reachable. But an unexported method could also be reachable from both the plugin and the host. We need to mark them as well.

[oszika]

@cherrymui Thank you for the explanations

[gopherbot]

Change https://go.dev/cl/393365 mentions this issue: cmd/link: mark unexported methods for plugins

[zhouguangyuan0718]

Well, it's ingenious. Maybe we need a backport to 1.17 and 1.18?
@gopherbot please open a backport issue.

[gopherbot]

Backport issue(s) opened: #51736 (for 1.17), #51737 (for 1.18).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.