Go programs stored on AFS may crash with SIGBUS
#50545
Comments
|
I have the feeling this is just #48997 with a different filesystem. Go uses a lot more signals than other runtimes, which might account for the fact that other runtimes don't exhibit this problem. It would be interesting to try to reproduce using a C program that sends itself lots of signals. That might illuminate matters. |
|
+1 |
bcmills
added
the
WaitingForInfo
|
This looks like a kernel bug. It seems that the kernel is trying to cancel the I/O operation, but you can’t usefully cancel the handling of a page fault! |
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
|
I think this is indeed not a bug in Go, but it is a bug in Linux. As a workaround, could Go register all of its signal handlers with |
|
Go does already register all signal handlers with |
|
That makes this either a kernel bug or an AFS bug, then. Probably the former. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Attempt to run a go binary (including the main
gobinary) from an AFS volume:What did you expect to see?
The application should run normally, i.e. without a
SIGBUSsignal.What did you see instead?
The program (at least one of the tests, in this case) crashes with a
SIGBUS. Note that the problem seems to happen inconsistently, so short workflows seem more likely to succeed without problems, but clearing the caches and retrying seems to reproduce it within a couple of attempts:./go/bin/go test ./go/src/...Outputok archive/tar 0.178s ok archive/zip 41.737s ok bufio 0.236s ok bytes 2.093s ok compress/bzip2 0.113s ok compress/flate 11.861s ok compress/gzip 0.077s ok compress/lzw 0.089s ok compress/zlib 1.319s ok container/heap 0.045s ok container/list 0.057s ok container/ring 0.051s ok context 0.075s ok crypto 0.006s ok crypto/aes 0.033s ok crypto/cipher 0.413s ok crypto/des 0.047s ok crypto/dsa 7.398s ok crypto/ecdsa 3.067s ok crypto/ed25519 0.796s ok crypto/ed25519/internal/edwards25519 5.503s ok crypto/ed25519/internal/edwards25519/field 2.508s ok crypto/elliptic 2.542s ok crypto/elliptic/internal/fiat 0.012s ok crypto/hmac 0.050s ? crypto/internal/randutil [no test files] ok crypto/internal/subtle 0.040s ok crypto/md5 0.028s ok crypto/rand 0.486s ok crypto/rc4 0.047s ok crypto/rsa 0.626s ok crypto/sha1 0.049s ok crypto/sha256 0.068s ok crypto/sha512 0.025s ok crypto/subtle 0.040s ok crypto/tls 6.205s ok crypto/x509 2.017s ? crypto/x509/pkix [no test files] ok database/sql 1.464s ok database/sql/driver 0.056s ok debug/dwarf 0.024s ok debug/elf 0.060s ok debug/gosym 1.033s ok debug/macho 0.016s ok debug/pe 0.040s ok debug/plan9obj 0.030s ? embed [no test files] ok embed/internal/embedtest 0.040s ? encoding [no test files] ok encoding/ascii85 0.043s ok encoding/asn1 0.030s ok encoding/base32 0.027s ok encoding/base64 0.026s ok encoding/binary 0.109s ok encoding/csv 0.010s ok encoding/gob 0.108s ok encoding/hex 0.024s ok encoding/json 2.096s ok encoding/pem 0.017s ok encoding/xml 0.061s ok errors 0.045s ok expvar 0.006s ok flag 0.158s ok fmt 0.134s ok go/ast 0.022s ok go/build 2.402s ok go/build/constraint 0.036s ok go/constant 0.030s ok go/doc 0.062s ok go/format 0.050s ok go/importer 0.501s ok go/internal/gccgoimporter 0.091s ok go/internal/gcimporter 2.486s ok go/internal/srcimporter 9.553s ? go/internal/typeparams [no test files] ok go/parser 0.224s ok go/printer 0.557s ok go/scanner 0.033s ok go/token 0.035s ok go/types 60.487s ok hash 0.012s ok hash/adler32 0.015s ok hash/crc32 0.055s ok hash/crc64 0.109s ok hash/fnv 0.087s ok hash/maphash 165.150s ok html 0.027s ok html/template 0.662s ok image 0.388s ok image/color 0.029s ? image/color/palette [no test files] ok image/draw 0.071s ok image/gif 0.866s ? image/internal/imageutil [no test files] ok image/jpeg 0.357s ok image/png 1.462s ok index/suffixarray 81.850s ok internal/abi 0.217s ? internal/buildcfg [no test files] ? internal/bytealg [no test files] ? internal/cfg [no test files] ok internal/cpu 0.114s ok internal/execabs 0.044s ok internal/fmtsort 0.018s ? internal/goexperiment [no test files] ? internal/goroot [no test files] ? internal/goversion [no test files] ok internal/itoa 0.045s ? internal/lazyregexp [no test files] ? internal/lazytemplate [no test files] ? internal/nettrace [no test files] ? internal/obscuretestdata [no test files] ? internal/oserror [no test files] ok internal/poll 0.833s ok internal/profile 0.023s ? internal/race [no test files] ok internal/reflectlite 0.140s ok internal/singleflight 0.051s ? internal/syscall/execenv [no test files] ? internal/syscall/unix [no test files] ? internal/sysinfo [no test files] ? internal/testenv [no test files] ? internal/testlog [no test files] ok internal/trace 2.279s ok internal/unsafeheader 0.033s ok internal/xcoff 0.047s ok io 0.150s ok io/fs 0.023s ok io/ioutil 0.064s ok log 0.028s ok log/syslog 1.347s ok math 0.025s ok math/big 6.155s ok math/bits 0.012s ok math/cmplx 0.005s ok math/rand 7.547s ok mime 0.052s ok mime/multipart 1.014s ok mime/quotedprintable 0.407s ok net 42.705s ok net/http 85.905s ok net/http/cgi 0.929s ok net/http/cookiejar 0.036s ok net/http/fcgi 0.122s ok net/http/httptest 0.041s ok net/http/httptrace 0.016s ok net/http/httputil 1.417s ok net/http/internal 0.020s ok net/http/internal/ascii 0.083s ? net/http/internal/testcert [no test files] ok net/http/pprof 5.208s ok net/internal/socktest 0.029s ok net/mail 0.009s ok net/rpc 0.176s ok net/rpc/jsonrpc 0.062s ok net/smtp 0.047s ok net/textproto 0.138s ok net/url 0.034s ok os 20.661s ok os/exec 2.766s ok os/signal 8.953s ? os/signal/internal/pty [no test files] ok os/user 0.160s ok path 0.074s ok path/filepath 0.240s ok plugin 0.073s ok reflect 3.289s ? reflect/internal/example1 [no test files] ? reflect/internal/example2 [no test files] ok regexp 24.081s ok regexp/syntax 0.697s --- FAIL: TestRaceProf (2.61s) crash_cgo_test.go:393: building testprogcgo [-race]: exit status 2 # runtime/testdata/testprogcgo fatal error: unexpected signal during runtime execution [signal SIGBUS: bus error code=0x2 addr=0x65f7b4 pc=0x44f4fc]--- FAIL: TestRaceSignal (0.00s)
crash_cgo_test.go:422: building testprogcgo [-race]: exit status 2
# runtime/testdata/testprogcgo
fatal error: unexpected signal during runtime execution
[signal SIGBUS: bus error code=0x2 addr=0x65f7b4 pc=0x44f4fc]
FAIL
FAIL runtime 308.329s
ok runtime/cgo 0.015s
ok runtime/debug 0.189s
ok runtime/internal/atomic 2.970s
ok runtime/internal/math 0.004s
ok runtime/internal/sys 0.067s
ok runtime/metrics 0.079s
ok runtime/pprof 54.341s
? runtime/race [no test files]
ok runtime/trace 14.803s
ok sort 0.702s
ok strconv 1.876s
ok strings 1.406s
ok sync 5.423s
ok sync/atomic 62.616s
ok syscall 0.185s
ok testing 3.597s
ok testing/fstest 0.040s
? testing/internal/testdeps [no test files]
ok testing/iotest 0.091s
ok testing/quick 0.080s
ok text/scanner 0.030s
ok text/tabwriter 0.046s
--- FAIL: TestLinkerGC (1.69s)
link_test.go:50: go build: exit status 2, # command-line-arguments
unexpected fault address 0x7fa404e99200
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fa404e99200 pc=0x567692]
FAIL
FAIL text/template 2.251s
ok text/template/parse 0.017s
ok time 9.976s
? time/tzdata [no test files]
ok unicode 0.017s
ok unicode/utf16 0.013s
ok unicode/utf8 0.074s
? unsafe [no test files]
FAIL
More Information
This does not appear to be specific to the
go testcommand or thegoexecutable in general, that just happens to be an easy example to reproduce the problem. The problem is reproducible (to varying degrees of success) with user applications written in go, although so far I have only managed to trigger the signal with (filesystem) I/O intensive applications. That seems to be true regardless of where the other files are located (and what filesystem they are on), so long as the application binary itself is on an AFS volume, as noted in go-hep/hep#885.I believe this is may be the same problem mentioned in the comment #37310 (comment), which does not appear to have a dedicated issue. This also appears suspiciously similar to an issue with go binaries and gcsfuse, see: #48997. The workaround from the latter issue appears to work here as well: running with
GODEBUG=asyncpreemptoff=1seems to prevent theSIGBUSerror from occuring (for both the maingobinary or for compiled user applications).If I had to guess, I would say that this looks like something is being preempted at a point that's unsafe if the binary happens to be on AFS (or apparently gcsfuse), but which turns out to be safe (in practice) on other filesystems. I don't know if that's a Go problem or an AFS (and gcsfuse) problem, but other binaries do not encounter a
SIGBUSwhen running from the same AFS volume (or on gcsfuse, according to that issue), so my initial suspicious is that the problem is on the Go side.The text was updated successfully, but these errors were encountered: