x/build/cmd/gomote: `unexpected /connect-ssh response: 502 Bad Gateway` when connecting multiple SSH sessions

[bcmills]

go1.19beta1 version -m $(which gomote)

/usr/local/google/home/bcmills/bin/gomote: go1.19beta1
        path    golang.org/x/build/cmd/gomote
        mod     golang.org/x/build      v0.0.0-20220614223518-59e7a6bb02db      h1:m2l/jsehSm9knEbpJWq+y8CTOTda9SEZoV3MtnaqMv8=
        dep     cloud.google.com/go/compute     v1.3.0  h1:mPL/MzDDYHsh5tHRS9mhmhWlcgClCrCa6ApQCU6wnHI=
        dep     github.com/aws/aws-sdk-go       v1.30.15        h1:Sd8QDVzzE8Sl+xNccmdj0HwMrFowv6uVUx9tGsCE1ZE=
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/googleapis/gax-go/v2 v2.1.1  h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
        dep     github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
        dep     go.opencensus.io        v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
        dep     golang.org/x/net        v0.0.0-20220127200216-cd36cc0744dd      h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
        dep     golang.org/x/oauth2     v0.0.0-20211104180415-d3ed0bb246c8      h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
        dep     golang.org/x/sync       v0.0.0-20210220032951-036812b2e83c      h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
        dep     golang.org/x/sys        v0.0.0-20220209214540-3681064d5158      h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
        dep     golang.org/x/text       v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
        dep     golang.org/x/time       v0.0.0-20210723032227-1f47c861a9ac      h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
        dep     google.golang.org/api   v0.70.0 h1:67zQnAE0T2rB0A3CwLSas0K+SbVzSxP+zTLkQLexeiw=
        dep     google.golang.org/genproto      v0.0.0-20220222213610-43724f9ea8cf      h1:SVYXkUz2yZS9FWb2Gm8ivSlbNQzL2Z/NpPKE3RG2jWk=
        dep     google.golang.org/grpc  v1.44.0 h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
        dep     google.golang.org/protobuf      v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
        dep     gopkg.in/inf.v0 v0.9.1  h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1

What did you do?

In one terminal (in a tmux session):

~$ gomote create linux-amd64
# still creating linux-amd64 after 5s; 0 requests ahead of you
# still creating linux-amd64 after 10s; 0 requests ahead of you
# still creating linux-amd64 after 15s; 0 requests ahead of you
# still creating linux-amd64 after 20s; 0 requests ahead of you
# still creating linux-amd64 after 25s; 0 requests ahead of you
# still creating linux-amd64 after 30s; 0 requests ahead of you
# still creating linux-amd64 after 35s; 0 requests ahead of you
# still creating linux-amd64 after 40s; 0 requests ahead of you
user-bcmills-linux-amd64-0

~$ gomote ssh user-bcmills-linux-amd64-0
$ ssh -p 2222 user-bcmills-linux-amd64-0@farmer.golang.org # auth using https://github.com/bcmills.keys
Enter passphrase for key '/usr/local/google/home/bcmills/.ssh/id_rsa':
# Welcome to the gomote ssh proxy, bcmills.
# Connecting to/starting remote ssh...
#
# `gomote push` and the builders use:
# - workdir: /workdir
# - GOROOT: /workdir/go
# - GOPATH: /workdir/gopath
# - env: GO_BUILDER_NAME=linux-amd64 GOROOT_BOOTSTRAP=/go1.4 GO_DISABLE_OUTBOUND_NETWORK=1
# Happy debugging.
…

Once the first SSH connection is established, attempt another gomote ssh command in a separate terminal:

~$ gomote ssh user-bcmills-linux-amd64-0

What did you expect to see?

Two successful gomote ssh sessions connected to the same VM.

What did you see instead?

$ ssh -p 2222 user-bcmills-linux-amd64-0@farmer.golang.org # auth using https://github.com/bcmills.keys
Enter passphrase for key '/usr/local/google/home/bcmills/.ssh/id_rsa':
# Welcome to the gomote ssh proxy, bcmills.
# Connecting to/starting remote ssh...
#
failed to connect to ssh on user-bcmills-linux-amd64-0: unexpected /connect-ssh response: 502 Bad Gateway, dial tcp [::1]:2200: connect: connection refused

―

To my recollection this used to work — I'm not sure exactly when it regressed.

(CC @golang/release)

[cagedmantis]

I'm investigating if this works via the new gomote path. Once we have that information then we will re-triage the issue.

[cagedmantis]

I've confirmed that this also does not work with the new gomote implementation.

[gopherbot]

Change https://go.dev/cl/541381 mentions this issue: cmd/buildlet: add an SSH server to the buildlet