x/playground: deploy may take over 10 minute and timeout

[dmitshur]

I've ran into this when deploying the Go Playground with Go 1.11.1 recently (see CL 140097), where the cloud build failed because it took more than the default 10 minute timeout:

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com Failure status: UNKNOWN: Error Response: [4] DEADLINE_EXCEEDED

In the recent past, some builds have taken around 9~ minutes. They were faster earlier on.

People have run into the 10 minute timeout in other contexts too. E.g., see https://stackoverflow.com/questions/50046545/google-app-engine-build-timed-out-during-deployment.

For now, this issue is to gather data and see if this keeps happening. We can then decide on how to best deal with it. If you run into this, please post data here. /cc @bradfitz @andybons

Workaround

The easiest temporary workaround I'm currently aware of is to increase the timeout before the gcloud app deploy step. For example:

gcloud config set app/cloud_build_timeout 900 # 15 mins

Note that it's a global config value.

[ysmolski]

I wonder if this build command includes building the docker container? I have noticed that building playground docker container became very slow. Specifically fetching packages is slow. Since I was not sure if it was my network, I did not create ticket for that.

[dmitshur]

I wonder if this build command includes building the docker container?

Yes, it does. That's the part that takes up to 8-9 minutes.

Specifically fetching packages is slow.

I didn't notice that to be the slow part, but thanks for pointing out. I'll watch it more closely next time.

[dmitshur]

A data point from today's deploy (for #25462): the cloud build step took 10 min 58 sec. (It was within my 15 minute timeout.)

[dmitshur]

Another data point from today's deploy: the cloud build step took 10 min 43 sec.

Edit: Another data point from a deploy on 2018-10-26: 10 min 40 sec.

[dmitshur]

Two data points for the cloud build step from today's deploy (in #30397), using Go version 1.12:

• 12 min 4 sec
• 11 min 27 sec

By now, it's clear we are reliably beyond the 10 minute mark.

[dmitshur]

I'm going to send a fix for this issue by documenting the command to set app/cloud_build_timeout to a value higher than the default 10 minutes. /cc @bradfitz @andybons

If we discover a better solution in the future, we can improve it then.

[gopherbot]

Change https://golang.org/cl/164577 mentions this issue: README: document the need to increase cloud build timeout

[zigo101]

Is it possible it is a problem of app engine? https://groups.google.com/forum/#!topic/google-appengine-go/fBdzFHZ1D5U

[dmitshur]

This issue was related to the time it takes to build the Docker image using Cloud Build, which grew from 8~ minutes to 12~ minutes over time, and how it intersected with the default timeout of 10 minutes. The linked thread doesn't seem to be directly related.