x/build/cmd/gomote: instances are timing out after 2 hours

[cagedmantis]

There have been reports that instances have been timing sometime after two hours of use.

• linux-amd64-alpine instances timeout after 2 hours.
• Other instances have become unresponsive some time after 2 hours of use (Windows-amd64).

Durring one occurance, we collected the following logs:

2022-08-23 13:47:59.025 EDT2022/08/23 17:47:59 created buildlet userx-windows-amd64-2016-0 for userx (http://10.128.0.20 GCE VM: buildlet-windows-amd64-2016-rna896368)
2022-08-23 15:49:13.486 EDT2022/08/23 19:49:13 deleting VM "buildlet-windows-amd64-2016-rna896368" in zone "us-central1-c"; delete-at expiration ...
2022-08-23 15:49:13.906 EDT2022/08/23 19:49:13 Sent request to delete instance "buildlet-windows-amd64-2016-rna896368" in zone "us-central1-c". Operation ID, Name: 219625075713282518, operation-1661284153488-5e6eddbd6f518-f4db622d-0bbce859
2022-08-23 15:50:21.040 EDT2022/08/23 19:50:21 Buildlet http://10.128.0.20 GCE VM: buildlet-windows-amd64-2016-rna896368 failed three heartbeats; final error: timeout waiting for headers
2022-08-23 16:51:00.891 EDT2022/08/23 20:51:00 10.128.0.20:80: peer dead with Buildlet http://10.128.0.20 GCE VM: buildlet-windows-amd64-2016-rna896368 failed heartbeat after 20.000336477s; marking dead; err=timeout waiting for headers, waiting for headers for /exec
2022-08-23 16:53:51.295 EDT2022/08/23 20:53:51 10.128.0.20:80: peer dead with Buildlet http://10.128.0.20 GCE VM: buildlet-windows-amd64-2016-rna896368 failed heartbeat after 20.000336477s; marking dead; err=timeout waiting for headers,waiting for headers for /exec

@golang/release @prattmic @thanm

[gopherbot]

Change https://go.dev/cl/426015 mentions this issue: cmd/coordinator: check the session pool for buildlets

[dmitshur]

We believe this is fixed.