x/build: using proxy.golang.org directly is incompatible with builder's outbound network restriction

[bradfitz]

Check out the failures on e.g. https://go-review.googlesource.com/c/build/+/191577 ...

https://storage.googleapis.com/go-build-log/0b9b152e/linux-amd64_c339c8eb.log

linux-amd64 at 0b9b152ee3c9a2ff079569d8d5a3a6982b1ae91d building build at dee4382cfa88a26484ad9afc880d5e21ca0c110d

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "test" "-short" "golang.org/x/build/..."] and env ["PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-stretch-rn7c6c4c7" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-amd64" "GOROOT_BOOTSTRAP=/go1.4" "GO_DISABLE_OUTBOUND_NETWORK=1" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://10.240.0.129:30157" "GO111MODULE=on" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache"] in dir /workdir/gopath/src/golang.org/x/build

...

go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding github.com/google/go-querystring v1.0.0
go: finding golang.org/x/sys v0.0.0-20190502175342-a43fa875dd82
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
build golang.org/x/build/app/appengine: cannot load google.golang.org/api/compute/v1: google.golang.org/api@v0.4.0: Get "https://storage.googleapis.com/proxy-golang-org-prod/3a4011dc517dc6d3-google.golang.org:api-v0.4.0.zip?Expires=1567105990&GoogleAccessId=gcs-urlsigner-prod%40golang-modproxy.iam.gserviceaccount.com&Signature=H6dQ47hBsB9eSDZcrmF9B%2F4t2v8NvSDIBVL9jwu%2BHDZHxj%2FdVvXRonTHKbF%2B5%2F5XbYTXV8cU%2BwKhEnx%2BfLmxqfcm3CugvFYWmV%2B9qnCENts4aL%2F4a6t3fd2T8%2FwJY75mo4r85%2FIiM0NzpoDUBaef1qPhnVq3u8eegvh5WcZKcXuQb9PXZHKsds%2FTIL2Gy48lGQsg11Gct1H45ycX6Reb2RnzIduOlM%2BVmbrSKB7lQ4MopftS6rlfzhex09p5Kc3AB0eZoUoT27VUE0kXht88dh2GGSirMDVJgD9zbFWmYNMR8OxgC45lyfcuJOAitfMjWL62ijK3sKynaqcRS%2FjV7A%3D%3D": dial tcp 74.125.126.128:443: connect: no route to host

Error: tests failed: exit status 1

That "connect: no route to host is because most of our builders are firewalled off so they don't accidentally depend on the network more than expected.

Did the Go module proxy start returning redirects to GCS?

/cc @andybons @toothrot @dmitshur @katiehockman @ianlancetaylor

[dmitshur]

Did the Go module proxy start returning redirects to GCS?

It seems to for some module versions:

$ curl -i 'https://proxy.golang.org/github.com/google/go-querystring/@v/v1.0.0.zip'
HTTP/2 200
[...]

[... actual zip bytes ...]

$ curl -i 'https://proxy.golang.org/google.golang.org/api/@v/v0.4.0.zip'
HTTP/2 302
[...]
location: https://storage.googleapis.com/proxy-golang-org-prod/3a4011dc517dc6d3-google.golang.org:api-v0.4.0.zip?Expires=1567161567&GoogleAccessId=gcs-urlsigner-prod%40golang-modproxy.iam.gserviceaccount.com&Signature=Tae2DfFrY2UncTN68%2BLskuY%2Bn1oBmYAs%2Fhj4YuNsxew0bmiZCWVqmwdiZchvTazpiT0lC2H7KZNz8yZOQUbqIhnOUyaB4wO7YBNaf3lQQUgIdvN4E%2BPoGsm5wXf7mBwj0PsjkTv5oCzPBsR%2Fek3fHIVgefOogSLVbsyLDt83bgIN11bGmG4pYVHuk4ZdJ2extai92aLiA4Mra2kfomKdrK3GI4lKybuaINfNi%2FBrqC64aR8WiqiYTWGYrFsDRKjvw8cEJhgmOc79iuUtdbk%2F%2Bx0DEZB%2FRyv2Fglc3dz4WQtlHpDhpoe4H0m%2FD8UKCaLY%2FdYEN%2BJH5tcbAolOIa5wDg%3D%3D
[...]

I think we'll need to open up the firewall just enough for GCS to get through.

[dmitshur]

As far as I can tell, it's intended behavior on the proxy's side, it applies to some larger zips.

I think we'll need to open up the firewall just enough for GCS to get through.

This can be a first order fix. Afterwards, if we want to tighten the firewall more, we can go back to having our own builder-specific intermediate proxy to pass everything through that we can allowlist very narrowly.

[gopherbot]

Change https://golang.org/cl/192317 mentions this issue: dashboard: disable outbound network restriction

[dmitshur]

Sent CL 192317 that implements the first step, the smallest change to return builders to a working state. It will give us more time to work on the next steps.

[dmitshur]

The aforementioned CL is deployed, so this issue is mitigated for now. I'll remove the Soon label.

We can leave this issue open to discuss next steps.

This has caused issue #30612 to get re-opened, so we need to find a solution that will allow us to close that again.