x/build/env/android: gomote debugging with Android builders is painful

[bradfitz]

@griesemer discovered that using gomote to debug things on Android is painful.

We should have a wiki page with some example sessions, including how to build the exec wrapper, and how to run go tests.

[eliasnaur]

What's missing from misc/android/README?

[griesemer]

The go_android_$GOARCH_exec wrapper was not found when I ran gomote. The gomote description didn't point out that I needed that wrapper, and I didn't look in misc/android/README. In general, if one is trying to debug something using gomote, a few more concrete examples would help.

[bradfitz]

That document mentions nothing about gomote.

And both that document and gomote require a bunch of effing around with environment variables and paths.

The combination of the two is super confusing, so cookbook examples (like we have at https://github.com/golang/go/wiki/Gomote#tricks) would help.

[griesemer]

Here's a log of a debugging session I had to fix #31084. Snippets of it may be useful as examples in the wiki.

# create a gomote instance on android-arm64-wikofever
$ gomote create android-arm64-wikofever
user-gri-android-arm64-wikofever-0

# here's a way to check the existing gomotes
$ gomote list
user-gri-android-arm64-wikofever-0	android-arm64-wikofever	host-darwin-amd64-eliasnaur-android	expires in 28m40.657041753s

# store gomote name in env variable for ease of use
$ export MOTE=user-gri-android-arm64-wikofever-0

# I've got a clean branch of master which fails a test in math/big.
# push the entire working directory to the gomote instance (make sure GOROOT is set):
$ gomote push $MOTE
2019/03/27 16:52:21 Remote doesn't have "src/debug/dwarf/testdata/line-gcc-win.bin"
2019/03/27 16:52:21 Remote doesn't have "src/index/suffixarray/suffixarray_test.go"
2019/03/27 16:52:21 Remote doesn't have "test/codegen/issue25378.go"
2019/03/27 16:52:21 Remote doesn't have "test/fixedbugs/bug302.dir/p.go"
2019/03/27 16:52:21 Remote doesn't have "src/cmd/internal/goobj/testdata/mycgo/go2.go"
2019/03/27 16:52:21 Remote doesn't have 8536 files (only showed 5).
2019/03/27 16:52:21 Remote lacks a VERSION file; sending a fake one
2019/03/27 16:52:28 Uploading 8537 new/changed files; 28644987 byte .tar.gz

# verify directory structure
$ gomote ls $MOTE
drwxr-xr-x	go/

# now build the tree on the remote instance
$ gomote run $MOTE go/src/make.bash
Building Go cmd/dist using /Users/elias/go-1.7.
Building Go toolchain1 using /Users/elias/go-1.7.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, darwin/amd64.
Building packages and commands for target, android/arm64.
---
Installed Go for android/arm64 in /private/var/folders/f6/d2bhfqss2716nxm8gkv1fmb80000gn/T/workdir-host-darwin-amd64-eliasnaur-android/go
Installed commands in /private/var/folders/f6/d2bhfqss2716nxm8gkv1fmb80000gn/T/workdir-host-darwin-amd64-eliasnaur-android/go/bin

# verify directory structure again
$ gomote ls $MOTE
drwxr-xr-x	go/
drwxr-xr-x	gocache/
drwxr-xr-x	tmp/

# CAVEAT! To run on Android, we need go_android_arm64_exec, make sure it was built:
# (It should be built by make/bash from the source in misc/android/go_android_exec.go)
$ gomote ls $MOTE go/bin
drwxr-xr-x	android_arm64/
-rwxr-xr-x	go	14729284	2019-03-27T23:55:36Z
-rwxr-xr-x	go_android_arm64_exec	4185976	2019-03-27T23:56:30Z
-rwxr-xr-x	gofmt	3589080	2019-03-27T23:55:28Z

# now we're trying to run the math/big tests: it fails because go_android_arm64_exec
# is not found => we need to add it to the path (see below)
$ gomote run $MOTE go/bin/go test ../src/math/big
fork/exec /var/folders/f6/d2bhfqss2716nxm8gkv1fmb80000gn/T/workdir-host-darwin-amd64-eliasnaur-android/tmp/go-build234509006/b001/big.test: exec format error
FAIL	math/big	0.002s
Error running run: exit status 1

# trying again, this time with the path to go/bin added (make sure to quote properly):
$ gomote run -path '$PATH,$WORKDIR/go/bin' $MOTE go/bin/go test ../src/math/big
ok  	math/big	60.916s

# after making a change in a file on my local machine, to run the changed code
# push the local code again
$ gomote push $MOTE
2019/03/27 18:11:09 Deleting remote files: ["go/test/fixedbugs/issue27836.dir/Äfoo.go" "go/test/fixedbugs/issue27836.dir/Ämain.go"]
2019/03/27 18:11:09 Remote doesn't have "test/fixedbugs/issue27836.dir/Ämain.go"
2019/03/27 18:11:09 Remote doesn't have "test/fixedbugs/issue27836.dir/Äfoo.go"
2019/03/27 18:11:09 Uploading 2 new/changed files; 456 byte .tar.gz

# run the test again, rinse and repeat

# when we're done, we can destroy the gomote explicitly, or let it expire (after 30min, it seems)
$ gomote destroy $MOTE

# and it's gone
$ gomote list

[eliasnaur]

I didn't realize the Android phone gomote would work and assumed you wanted to run tests locally. Thank you for the gomote log, I'll condense it into a wiki entry.

[eliasnaur]

I've added an android entry to https://github.com/golang/go/wiki/Gomote#tricks. Let me know if you need more.

[griesemer]

Thanks for this. If there's one thing that could make the wiki significantly clearer it would be a paragraph about directory structure and the "gomote machine model". It took me a bunch of try-and-error runs to find out. Much of it may be "obvious" in retrospect, but it helps spelling it out right away, especially for first-time gomote users. This would provide newcomers a clearer model of how the gomote looks like from a client's perspective.

Some of the things that would be nice if they were spelled out:

• each instance of a gomote has its own working directory, accessible via $WORKDIR
• gomote push does essentially an rsync of the $GOROOT directory into the remote $WORKDIR; i.e., after a gomote push, the local $GOROOT appears as go directory on the gomote (I had to find this out by doing repeated gomote ls commands)
• when issuing gomote run commands, one needs to specify the full path for commands and files, e.g. go/bin/go tool compile go/src/mypkg/x.go - there appears no notion of a cwd, or perhaps I missed it
• to change individual uploaded files, make the changes locally and use gomote push $MOTE and it will only upload the changes (this was not obvious to me at first, especially because there's also a gomote put command to push single files. I thought I was supposed to use that one for smaller changes)
• I didn't figure out to how to use gomote put to get a file into a specific directory on the gomote instance (I still don't know how to do it, but I resorted to just using gomote push all the time.)
• explain that the gomote self-destructs after 30min of non-use
• I'm unclear on the -system flag

[eliasnaur]

I filed #31118 for expanding the documentation.