Description
go version devel +f4be767501 Wed Jan 18 16:21:28 2017 -0500 linux/amd64
The fix for #17753 causes signal delivery in Go programs running under TSAN to pass through the TSAN interceptors.
Unfortunately, it appears that the TSAN interceptors defer delivery of asynchronous signals until the next blocking libc call on the same thread. This interacts poorly with the Go runtime's use of direct 'futex' syscalls to park idle threads: if the thread receiving the signal happens to be one that the runtime is about to park, it may never make a libc call and the signal will be lost.
A simple program illustrating this behavior:
src/tsansig/tsansig.go:
package main
import (
"os"
"os/signal"
"syscall"
)
/*
#cgo CFLAGS: -g -fsanitize=thread
#cgo LDFLAGS: -g -fsanitize=thread
*/
import "C"
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGUSR1)
defer signal.Stop(c)
syscall.Kill(syscall.Getpid(), syscall.SIGUSR1)
<-c
}
$ go build tsansig
$ ./tsansig
It should exit with status 0 almost immediately. Instead, it hangs forever.
One might expect that injecting a blocking libc call would help:
package main
import (
"os"
"os/signal"
"syscall"
"time"
)
/*
#cgo CFLAGS: -g -fsanitize=thread
#cgo LDFLAGS: -g -fsanitize=thread
#include <stddef.h>
#include <time.h>
*/
import "C"
func nanosleep(d time.Duration) error {
req := C.struct_timespec{
tv_sec: C.__time_t(d / time.Second),
tv_nsec: C.__syscall_slong_t(d % time.Second),
}
var rem C.struct_timespec
for {
switch _, err := C.nanosleep(&req, &rem); err {
case nil:
return nil
case syscall.EINTR:
req = rem
continue
default:
return err
}
}
}
func main() {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGUSR1)
defer signal.Stop(c)
syscall.Kill(syscall.Getpid(), syscall.SIGUSR1)
for {
select {
case <-c:
return
default:
if err := nanosleep(10 * time.Nanosecond); err != nil {
panic(err)
}
}
}
}
But that attempt at a workaround does not succeed: the thread that receives the signal happens not to be the one the runtime uses for the subsequent cgo calls.
Impact on Go 1.8
This is (unfortunately) a significant regression over Go 1.7. In 1.7, Go programs without significant C signal behavior would exhibit correct Go signal behavior. In the current 1.8 RC, the C signal behavior in the presence of the Go runtime is improved, but the Go signal behavior is broken even for programs with no interesting C signal activity.
Activity
bcmills commentedon Jan 19, 2017
One possible fix would be to have something in the runtime make an explicit call (in cgo-enabled builds) to some no-op libc function (e.g.
sleep(0)
) to trigger signal delivery. (runtime.findrunnable
seems like a likely candidate.)However, it isn't obvious to me that any call to
runtime.futexsleep
with a negativens
parameter can be made safely in the presence of TSAN, since the signal could be delivered at any point between the last libc call and thefutex
syscall. Sincefutexsleep
allows spurious wakeups, perhaps we also need to inject an arbitrary timeout tofutexsleep
when TSAN is enabled to bound the delay on signal delivery.bcmills commentedon Jan 19, 2017
(@ianlancetaylor @dvyukov )
bcmills commentedon Jan 19, 2017
https://go-review.googlesource.com/#/c/35494/ has a potential fix.
dvyukov commentedon Jan 20, 2017
Tsan has a notion of "blocking regions" of code (e.g. pthread_cond_wait). It delivers signals synchronously in such regions as it knows that no activity happens on the thread. We could mark whole Go world as "blocking region", then tsan will deliver signals synchronously while in Go code. However, we call some tsan annotations (__tsan_acquire/release) from Go code, so we will need to somehow mark them as well.
This will require new hooks in tsan runtime, so we will need to handle versioning somehow as old tsan runtime won't have the hooks.
What release is this for?
bcmills commentedon Jan 20, 2017
That seems much more complicated than making periodic libc calls on idle
M
s — we would also need to unblock the regions during cgo calls.I was trying to fix it in 1.8, but the changes are invasive enough that Ian suggested we delay until 1.9.
[-]runtime: async signals not reliably delivered to Go threads under TSAN/MSAN[/-][+]runtime: async signals not reliably delivered to Go threads under TSAN[/+]gopherbot commentedon Jan 24, 2017
CL https://golang.org/cl/35494 mentions this issue.
bcmills commentedon Mar 8, 2017
I've been thinking more about @dvyukov's comments, and I'm starting to see the logic to it.
I agree that marking the whole Go runtime as a "blocking region" for TSAN purposes is the right first-principles solution. It seems that as far as TSAN is concerned, the important property of a "blocking call" is that it does not acquire any libc locks, and Go only acquires libc locks when executing cgo function calls (including the
x_cgo
hooks in place for other runtime calls).That said, the "periodic yielding" strategy does seem to improve responsiveness significantly in practice, so I think that's probably the right fallback for versions of TSAN which do not support the proposed new hooks.
gopherbot commentedon Mar 9, 2017
CL https://golang.org/cl/37867 mentions this issue.
runtime: fix _cgo_yield usage with sysmon and on BSD