runtime: unexpected fault address 0xffffffff (windows/386)

[AaronO]

Here's my setup:

• Go version 1.5
• Cross compiling from linux to windows/386
• Linking (maybe not relevant):
  • Static: libgit2
  • Dynamic: -lwinhttp -lws2_32 -lcrypt32 -lole32 -lrpcrt4 -ladvapi32
• Running on Windows 10 x64

When running this program on some Windows 10 x64 machines, this fails with the following error below (it does not fail on all machines but fails consistently and repeatedly on those where it does):

> uhub_windows_386.exe
unexpected fault address 0xffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffff pc=0x52a5bb]

goroutine 1 [running, locked to thread]:
runtime.throw(0x86df74, 0x5)
        /usr/local/go-1.5/src/runtime/panic.go:527 +0x7f fp=0x1271fe60 sp=0x1271fe54
runtime.sigpanic()
        /usr/local/go-1.5/src/runtime/signal_windows.go:164 +0x113 fp=0x1271fe78 sp=0x1271fe60
runtime.aeshashbody()
        /usr/local/go-1.5/src/runtime/asm_386.s:999 +0x1bf fp=0x1271fe7c sp=0x1271fe78
runtime.mapassign1(0x7b2f8c, 0x126f7720, 0x92958c, 0x929594)
        /usr/local/go-1.5/src/runtime/hashmap.go:424 +0x8a fp=0x1271fedc sp=0x1271fe7c
time.init()
        /usr/local/go-1.5/src/time/format.go:1151 +0x182 fp=0x1271ff1c sp=0x1271fedc
os.init()
        /usr/local/go-1.5/src/os/types_windows.go:107 +0x48 fp=0x1271ff40 sp=0x1271ff1c
fmt.init()
        /usr/local/go-1.5/src/fmt/scan.go:1190 +0x57 fp=0x1271ff78 sp=0x1271ff40
log.init()
        /usr/local/go-1.5/src/log/log.go:346 +0x48 fp=0x1271ff9c sp=0x1271ff78
main.init()
        /go/src/github.com/GitbookIO/uhub/server.go:69 +0x3e fp=0x1271ffa0 sp=0x1271ff9c
runtime.main()
        /usr/local/go-1.5/src/runtime/proc.go:100 +0x205 fp=0x1271ffc8 sp=0x1271ffa0
runtime.goexit()
        /usr/local/go-1.5/src/runtime/asm_386.s:1662 +0x1 fp=0x1271ffcc sp=0x1271ffc8

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go-1.5/src/runtime/asm_386.s:1662 +0x1

Any ideas of what could be the cause of this ? Are there any known fixes or workarounds ?

I appreciate any insight you may have, thanks !

[minux]

Have you tried to build your binaries natively on windows? Could you reproduce this crash with simpler examples? (by simpler, I mean something that at least doesn't use unsafe or cgo.)

[AaronO]

@minux I'm writing a simpler test case now, I'll update here once that's done.

I haven't compiled on windows, for this specific project cross-compiling from linux is much easier (because of cmake dependencies and others). BTW the same binary cross compiled for windows/amd64 (GOOS=windows GOARCH=amd64) doesn't crash.

[randall77]

Possibly an alignment issue with the linker? The failing instruction is reading a 16-byte value that needs to be at 16-byte alignment.

[AaronO]

@randall77 I'm cross-compiling using mingw on linux (i686-w64-mingw32).

Do you have a fix or a workaround in mind ?

This has become hard to debug on my side, a colleague of mine gave me an identical copy of his virtual machine (where the binary crashed) but when running the same binaries inside the now imported (but identical) VM it doesn't crash ... (So it's not even repeatable across VMs).

[randall77]

If you can run:

go tool nm uhub_windows_386.exe | grep masks

Look at the alignment of this symbol. If it is not 16-byte aligned, that would be the problem.
It seems unlikely, but worth a try. I don't understand why it would vary with machine, unless there is ASLR or something messing with layouts. Or some machines fault on unaligned reads and some don't?

[AaronO]

@randall77 Here's my output:

> go tool uhub_windows_386.exe | grep masks
  92a96c T masks

So indeed, it doesn't seem to be 16-byte aligned and it's amd64 counterpart seems to be correctly aligned:

> go tool nm uhub_windows_amd64.exe | grep masks
  a6ed30 T masks

How can I change my build process to ensure that masks is always 16-byte aligned ?

[randall77]

Well, at least we know what is wrong. It sounds like a genuine bug in the linker and/or our input to the linker. Are you using external linking? It sounds like it. Maybe we need an additional notation in our compiled output to convince the external linker to align sections correctly. ccing people with more linking knowledge.

@ianlancetaylor @rsc

[alexbrainman]

gcc would be producing executable. Go linker produces object file. But maybe if we mark .text section with IMAGE_SCN_ALIGN_16BYTES, just maybe gcc will keep appropriate address aligned too. I don't know of a way to align a particular symbol in object file. We'll need some test to reproduce this.

Alex

[minux]

I remember seeing problem like this while developing external linking for Windows, i will take a look. Perhaps the real problem is just masked and not fixed.

[minux]

Yes, the problem is because the text section is only aligned to 4 bytes, but data and bss are 32-byte aligned. And mask, being defined in assembly, is included in the text section, even though it's declared DATA.

[minux]

Please give CL 14166 a try and see if it solves the problem.

[AaronO]

@minux Thanks, giving it a try now, I'll report back shortly.

[AaronO]

@minux That seems to work 👍

❯ go tool nm ./build/uhub_windows_386.exe | grep masks
  92a980 T masks

I think you've found the root of the problem, so unless there's anything you need on my behalf I think we can close this issue :)

[minux]

How you tried the new binary on the old VM where old binary uses to crash? I want to verify that it's indeed the correct fix. Thanks!

[AaronO]

It's late here in Europe but I'll try it on my colleague's VM first thing tomorrow morning and I'll report back here.

Thanks for your help !