runtime: nosplit stack overflow when building tests with -N -l

[glycerine]

natively OSX go1.6rc2 darwin amd64
but trying to cross compile to GOOS=linux GOARCH=amd64

I've never seen this; not sure how serious it is.

$ make
go build  # succeeds, however:
go test -c -gcflags "-N -l" -v
# testmain
runtime.typedslicecopy: nosplit stack overflow
        584     assumed on entry to runtime.typedslicecopy (nosplit)
        448     after runtime.typedslicecopy (nosplit) uses 136
        440     on entry to runtime.cgoCheckSliceCopy (nosplit)
        376     after runtime.cgoCheckSliceCopy (nosplit) uses 64
        368     on entry to runtime.cgoCheckTypedBlock (nosplit)
        168     after runtime.cgoCheckTypedBlock (nosplit) uses 200
        160     on entry to runtime.cgoCheckBits (nosplit)
        24      after runtime.cgoCheckBits (nosplit) uses 136
        16      on entry to runtime.cgoIsGoPointer (nosplit)
        -32     after runtime.cgoIsGoPointer (nosplit) uses 48
runtime.sigtrampgo: nosplit stack overflow
        584     assumed on entry to runtime.sigtrampgo (nosplit)
        392     after runtime.sigtrampgo (nosplit) uses 192
        384     on entry to runtime.sigfwdgo (nosplit)
        288     after runtime.sigfwdgo (nosplit) uses 96
        280     on entry to runtime.dieFromSignal (nosplit)
        240     after runtime.dieFromSignal (nosplit) uses 40
        232     on entry to runtime.setsig (nosplit)
        40      after runtime.setsig (nosplit) uses 192
        32      on entry to runtime.funcPC (nosplit)
        0       after runtime.funcPC (nosplit) uses 32
        -8      on entry to runtime.add (nosplit)
runtime.cgocallback_gofunc: nosplit stack overflow
        584     assumed on entry to runtime.cgocallback_gofunc (nosplit)
        576     after runtime.cgocallback_gofunc (nosplit) uses 8
        568     on entry to runtime.cgocallbackg (nosplit)
        480     after runtime.cgocallbackg (nosplit) uses 88
        472     on entry to runtime.exitsyscall (nosplit)
        352     after runtime.exitsyscall (nosplit) uses 120
        344     on entry to runtime.exitsyscallfast (nosplit)
        184     after runtime.exitsyscallfast (nosplit) uses 160
        176     on entry to runtime.writebarrierptr (nosplit)
        128     after runtime.writebarrierptr (nosplit) uses 48
        120     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        56      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        48      on entry to runtime.cgoIsGoPointer (nosplit)
        0       after runtime.cgoIsGoPointer (nosplit) uses 48
        -8      on entry to runtime.cgoInRange (nosplit)
make: *** [testbuild] Error 2
jaten@jatens-MacBook-Pro ~ $ 
jaten@jatens-MacBook-Pro ~ $ env|grep GO
GO15VENDOREXPERIMENT=1
GOARCH=amd64
GOOS=linux
GODEBUG=cgocheck=2
GOPATH=/Users/jaten
jaten@jatens-MacBook-Pro ~ $ 

[ianlancetaylor]

It would be nicer if this didn't happen, but I think it's also OK for us to say "don't do that." In particular, as far as I can tell, it doesn't happen with go build, only with go test.

[lowenna]

I'm seeing the same thing using go version 1.6 (final, not rc), on windows/amd64, but not cross-compiling.

E:\go\src\github.com\docker\containerd\containerd [master +1 ~5 -0 !]> go build -gcflags "-N -l" -v
github.com/Sirupsen/logrus
github.com/docker/containerd
github.com/codegangsta/cli
github.com/golang/protobuf/proto
golang.org/x/net/internal/timeseries
golang.org/x/net/context
google.golang.org/grpc/codes
github.com/rcrowley/go-metrics
google.golang.org/grpc/grpclog
google.golang.org/grpc/naming
golang.org/x/net/http2/hpack
github.com/docker/containerd/specs
github.com/cloudfoundry/gosigar
github.com/docker/containerd/chanotify
golang.org/x/net/trace
google.golang.org/grpc/credentials
google.golang.org/grpc/metadata
golang.org/x/net/http2
github.com/docker/containerd/runtime
github.com/cyberdelia/go-metrics-graphite
google.golang.org/grpc/transport
google.golang.org/grpc
github.com/docker/containerd/api/grpc/types
github.com/docker/containerd/supervisor
github.com/docker/containerd/api/grpc/server
github.com/docker/containerd/containerd
# github.com/docker/containerd/containerd
runtime.typedslicecopy: nosplit stack overflow
        584     assumed on entry to runtime.typedslicecopy (nosplit)
        448     after runtime.typedslicecopy (nosplit) uses 136
        440     on entry to runtime.cgoCheckSliceCopy (nosplit)
        376     after runtime.cgoCheckSliceCopy (nosplit) uses 64
        368     on entry to runtime.cgoCheckTypedBlock (nosplit)
        168     after runtime.cgoCheckTypedBlock (nosplit) uses 200
        160     on entry to runtime.cgoCheckBits (nosplit)
        24      after runtime.cgoCheckBits (nosplit) uses 136
        16      on entry to runtime.cgoIsGoPointer (nosplit)
        -32     after runtime.cgoIsGoPointer (nosplit) uses 48
syscall.loadlibrary: nosplit stack overflow
        584     assumed on entry to syscall.loadlibrary (nosplit)
        536     after syscall.loadlibrary (nosplit) uses 48
        528     on entry to runtime.cgocall (nosplit)
        480     after runtime.cgocall (nosplit) uses 48
        472     on entry to runtime.exitsyscall (nosplit)
        352     after runtime.exitsyscall (nosplit) uses 120
        344     on entry to runtime.exitsyscallfast (nosplit)
        184     after runtime.exitsyscallfast (nosplit) uses 160
        176     on entry to runtime.writebarrierptr (nosplit)
        128     after runtime.writebarrierptr (nosplit) uses 48
        120     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        56      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        48      on entry to runtime.cgoIsGoPointer (nosplit)
        0       after runtime.cgoIsGoPointer (nosplit) uses 48
        -8      on entry to runtime.cgoInRange (nosplit)
syscall.getprocaddress: nosplit stack overflow
        584     assumed on entry to syscall.getprocaddress (nosplit)
        536     after syscall.getprocaddress (nosplit) uses 48
        528     on entry to runtime.cgocall (nosplit)
        480     after runtime.cgocall (nosplit) uses 48
        472     on entry to runtime.exitsyscall (nosplit)
        352     after runtime.exitsyscall (nosplit) uses 120
        344     on entry to runtime.exitsyscallfast (nosplit)
        184     after runtime.exitsyscallfast (nosplit) uses 160
        176     on entry to runtime.writebarrierptr (nosplit)
        128     after runtime.writebarrierptr (nosplit) uses 48
        120     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        56      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        48      on entry to runtime.cgoIsGoPointer (nosplit)
        0       after runtime.cgoIsGoPointer (nosplit) uses 48
        -8      on entry to runtime.cgoInRange (nosplit)
syscall.Syscall: nosplit stack overflow
        584     assumed on entry to syscall.Syscall (nosplit)
        528     after syscall.Syscall (nosplit) uses 56
        520     on entry to runtime.cgocall (nosplit)
        472     after runtime.cgocall (nosplit) uses 48
        464     on entry to runtime.exitsyscall (nosplit)
        344     after runtime.exitsyscall (nosplit) uses 120
        336     on entry to runtime.exitsyscallfast (nosplit)
        176     after runtime.exitsyscallfast (nosplit) uses 160
        168     on entry to runtime.writebarrierptr (nosplit)
        120     after runtime.writebarrierptr (nosplit) uses 48
        112     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        48      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        40      on entry to runtime.cgoIsGoPointer (nosplit)
        -8      after runtime.cgoIsGoPointer (nosplit) uses 48
syscall.Syscall6: nosplit stack overflow
        584     assumed on entry to syscall.Syscall6 (nosplit)
        528     after syscall.Syscall6 (nosplit) uses 56
        520     on entry to runtime.cgocall (nosplit)
        472     after runtime.cgocall (nosplit) uses 48
        464     on entry to runtime.exitsyscall (nosplit)
        344     after runtime.exitsyscall (nosplit) uses 120
        336     on entry to runtime.exitsyscallfast (nosplit)
        176     after runtime.exitsyscallfast (nosplit) uses 160
        168     on entry to runtime.writebarrierptr (nosplit)
        120     after runtime.writebarrierptr (nosplit) uses 48
        112     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        48      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        40      on entry to runtime.cgoIsGoPointer (nosplit)
        -8      after runtime.cgoIsGoPointer (nosplit) uses 48
syscall.Syscall9: nosplit stack overflow
        584     assumed on entry to syscall.Syscall9 (nosplit)
        528     after syscall.Syscall9 (nosplit) uses 56
        520     on entry to runtime.cgocall (nosplit)
        472     after runtime.cgocall (nosplit) uses 48
        464     on entry to runtime.exitsyscall (nosplit)
        344     after runtime.exitsyscall (nosplit) uses 120
        336     on entry to runtime.exitsyscallfast (nosplit)
        176     after runtime.exitsyscallfast (nosplit) uses 160
        168     on entry to runtime.writebarrierptr (nosplit)
        120     after runtime.writebarrierptr (nosplit) uses 48
        112     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        48      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        40      on entry to runtime.cgoIsGoPointer (nosplit)
        -8      after runtime.cgoIsGoPointer (nosplit) uses 48
syscall.Syscall12: nosplit stack overflow
        584     assumed on entry to syscall.Syscall12 (nosplit)
        528     after syscall.Syscall12 (nosplit) uses 56
        520     on entry to runtime.cgocall (nosplit)
        472     after runtime.cgocall (nosplit) uses 48
        464     on entry to runtime.exitsyscall (nosplit)
        344     after runtime.exitsyscall (nosplit) uses 120
        336     on entry to runtime.exitsyscallfast (nosplit)
        176     after runtime.exitsyscallfast (nosplit) uses 160
        168     on entry to runtime.writebarrierptr (nosplit)
        120     after runtime.writebarrierptr (nosplit) uses 48
        112     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        48      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        40      on entry to runtime.cgoIsGoPointer (nosplit)
        -8      after runtime.cgoIsGoPointer (nosplit) uses 48
syscall.Syscall15: nosplit stack overflow
        584     assumed on entry to syscall.Syscall15 (nosplit)
        528     after syscall.Syscall15 (nosplit) uses 56
        520     on entry to runtime.cgocall (nosplit)
        472     after runtime.cgocall (nosplit) uses 48
        464     on entry to runtime.exitsyscall (nosplit)
        344     after runtime.exitsyscall (nosplit) uses 120
        336     on entry to runtime.exitsyscallfast (nosplit)
        176     after runtime.exitsyscallfast (nosplit) uses 160
        168     on entry to runtime.writebarrierptr (nosplit)
        120     after runtime.writebarrierptr (nosplit) uses 48
        112     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        48      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        40      on entry to runtime.cgoIsGoPointer (nosplit)
        -8      after runtime.cgoIsGoPointer (nosplit) uses 48
runtime.cgocallback_gofunc: nosplit stack overflow
        584     assumed on entry to runtime.cgocallback_gofunc (nosplit)
        576     after runtime.cgocallback_gofunc (nosplit) uses 8
        568     on entry to runtime.cgocallbackg (nosplit)
        480     after runtime.cgocallbackg (nosplit) uses 88
        472     on entry to runtime.exitsyscall (nosplit)
        352     after runtime.exitsyscall (nosplit) uses 120
        344     on entry to runtime.exitsyscallfast (nosplit)
        184     after runtime.exitsyscallfast (nosplit) uses 160
        176     on entry to runtime.writebarrierptr (nosplit)
        128     after runtime.writebarrierptr (nosplit) uses 48
        120     on entry to runtime.cgoCheckWriteBarrier (nosplit)
        56      after runtime.cgoCheckWriteBarrier (nosplit) uses 64
        48      on entry to runtime.cgoIsGoPointer (nosplit)
        0       after runtime.cgoIsGoPointer (nosplit) uses 48
        -8      on entry to runtime.cgoInRange (nosplit)
E:\go\src\github.com\docker\containerd\containerd [master +1 ~5 -0 !]>

@ianlancetaylor Is there a workaround to compile with debug symbols (gcflags -N -l) on using 1.6

[randall77]

Note that you don't need to set -N -l to get debug symbols. Debug symbols are always provided. -N and -l just disable some optimizations. Could you explain more about why you use -N -l and what debugging deficiency you're seeing without them?

[ianlancetaylor]

Even if you want to use those flags, it ought to work to install the runtime and syscall packages using the default options, and use the flags only for other packages.

[lowenna]

Sorry, yes, I said "with debug symbols", but I actually meant "without optimizations". Obviously apart from the failing to compile with the above error, what I see is that when attempting to debug docker or containerd on Windows, that gdb64.exe, both directly or indirectly via LiteIDE simply doesn't trigger on breakpoints, or allow any debugging. I used to have this working, circa go 1.4.3 timeframe, maybe even 1.5.1. Note for the compile error, I might have found a workaround in #12055

Set GOROOT_BOOTSTRAP=c:\go1.4.3
Set GO_GCFLAGS=-N
In c:\go\src make.bat

But unfortunately, still no breakpoints trigger.

[randall77]

That seems strange. Breakpoints should still work. Maybe if the breakpoint was in an inlined function or something it might not work, but I don't understand how they could be broken everywhere.

Can you give us some more instructions to reproduce?

[lowenna]

Thanks @randall77. I'm actively trying to track this down. You're right the -N -l is a complete red herring. Apologies for derailing on that.

I've tried various versions of golang - 1.5.1, 1.5.3 and 1.6 (currently reverted back to 1.5.1). This is all on Windows.

Clone https://github.com/docker/docker to c:\go\src\github.com\docker\docker (e: drive in my case, I use a separate drive for the sources from the system drive)

From c:\go\src\github.com\docker\docker\docker, go build. This builds the 'client' binary (not the joint client and daemon binary).

In liteide, set a breakpoint (F9) on the very first line in func main() of c:\go\src\github.com\docker\docker\docker\docker.go, if reexec.Init() {. Hit F5 (run) and the breakpoint triggers.

Repeat exactly the same, but this time, include the daemon code conditionally compiled in using go build -tags daemon. Same breakpoint, but it just doesn't trigger.

[Hopefully that will work, I've got some wrapper scripts which set up some autogen folders needed to compile, and a few utilities such as rsrc installed, so they may be needed. I'd need to sort out instructions for those probably.]

Edit: Oops, wrong screenshot for second one. I did set exactly the same breakpoint, on the very first line executed.

[glycerine]

@randall77 Breakpoints haven't worked reliably since go1.2.3 when Ian specifically repaired them from 1.2.2. This has long been pretty trivial to reproduce by firing up gdb and setting some breakpoints, half of them generally will be missed. (go1.2.3 was the last release where they did work reliably).

[glycerine]

(Moreover because gdb lies to you about variable values as well, c.f. #6913, I've just gone completely over to printf debugging these days.)

[alexbrainman]

@jhowardmsft I tried following your build instructions, but no luck:

c:\>git clone https://github.com/docker/docker c:\go\src\github.com\docker\docker
Cloning into 'c:\go\src\github.com\docker\docker'...
remote: Counting objects: 139081, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 139081 (delta 10), reused 3 (delta 3), pack-reused 139059
Receiving objects: 100% (139081/139081), 80.66 MiB | 311.00 KiB/s, done.
Resolving deltas: 100% (92826/92826), done.
Checking connectivity... done.

c:\>cd c:\go\src\github.com\docker\docker\docker

c:\go\src\github.com\docker\docker\docker>go build
common.go:8:2: cannot find package "github.com/Sirupsen/logrus" in any of:
        c:\users\alex\dev\go\src\github.com\Sirupsen\logrus (from $GOROOT)
        ($GOPATH not set)
docker.go:8:2: cannot find package "github.com/docker/docker/api/client" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\api\client (from $GOROOT)
        ($GOPATH not set)
docker_windows.go:4:2: cannot find package "github.com/docker/docker/autogen/winresources" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\autogen\winresources (from $GOROOT)
        ($GOPATH not set)
client.go:6:2: cannot find package "github.com/docker/docker/cli" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\cli (from $GOROOT)
        ($GOPATH not set)
client.go:7:2: cannot find package "github.com/docker/docker/cliconfig" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\cliconfig (from $GOROOT)
        ($GOPATH not set)
docker.go:10:2: cannot find package "github.com/docker/docker/dockerversion" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\dockerversion (from $GOROOT)
        ($GOPATH not set)
common.go:11:2: cannot find package "github.com/docker/docker/opts" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\opts (from $GOROOT)
        ($GOPATH not set)
client.go:8:2: cannot find package "github.com/docker/docker/pkg/mflag" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\pkg\mflag (from $GOROOT)
        ($GOPATH not set)
docker.go:12:2: cannot find package "github.com/docker/docker/pkg/reexec" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\pkg\reexec (from $GOROOT)
        ($GOPATH not set)
docker.go:13:2: cannot find package "github.com/docker/docker/pkg/term" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\pkg\term (from $GOROOT)
        ($GOPATH not set)
client.go:9:2: cannot find package "github.com/docker/docker/utils" in any of:
        c:\users\alex\dev\go\src\github.com\docker\docker\utils (from $GOROOT)
        ($GOPATH not set)
common.go:13:2: cannot find package "github.com/docker/go-connections/tlsconfig" in any of:
        c:\users\alex\dev\go\src\github.com\docker\go-connections\tlsconfig (from $GOROOT)
        ($GOPATH not set)

c:\go\src\github.com\docker\docker\docker>

What am I doing wrong? Thank you.

Alex

[lowenna]

Hi Alex - thanks for trying. The first issue here seems to be your GOPATH. It should be set to c:\go\src\github.com\docker\docker\vendor;c:\go. But I suspect then you'll hit either an 'autogen' or 'rsrc' issue. Probably autogen. Can you set your GOPATH as above and it should hopefully be fairly straightforward to get to the next step. But it will need you to run a bash script to generate the autogen folder and contents, so will require GitHub for Windows (2.7.x) installed (for msys2) with sh in your path.

From c:\go\src\github.com\docker\docker (only two dockers), run this script, saved as autogen.sh by sh /path/to/autogen.sh:

#!/usr/bin/env bash
set -e
VERSION=$(< ./VERSION)
if command -v git &> /dev/null && git rev-parse &> /dev/null; then
        GITCOMMIT=$(git rev-parse --short HEAD)
        if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
                GITCOMMIT="$GITCOMMIT-dirty"
        fi
        BUILDTIME=$(date -u)
elif [ "$DOCKER_GITCOMMIT" ]; then
        GITCOMMIT="$DOCKER_GITCOMMIT"
fi
source hack/make/.go-autogen

Hopefully then the go build -tags daemon should get further when run from the GitHub.com\docker\docker\docker directory.

Sorry the instructions for building docker on Windows are so complex :(
Thanks,
John.

[lowenna]

I should have added, another way to build is from c:\go\src\github.com\docker\docker, run sh hack/make.sh binary. That will build docker.exe into bundles\1.11.0-dev\docker.exe. And won't need the autogen part. But will need rsrc installed by the equivalent of these lines from Dockerfile.windows: https://github.com/docker/docker/blob/master/Dockerfile.windows#L94-96

[alexbrainman]

Got little further, but still no cigar:

c:\go\src\github.com\docker\docker>sh c:\go\autogen.sh
hack/make/.go-autogen: line 53: rsrc: command not found

c:\go\src\github.com\docker\docker>

Alex