x/telemetry/upload: decide upload retry logic

[hyangah]

The upload package keeps retrying unconditionally if the POST request ends with non-200 status code. code The upload program will keep retrying...
In case the upload fails due to transient network errors or some internal errors etc, a few retries before giving up completely make sense. But currently telemetry.go.dev upload server also returns http.StatusBadRequest, or other errors. In that case, retrying upload with the same data is not desirable.

Either make the upload package distinguish retryable and non-retryable status codes, or make the upload server never return an error if the upload shouldn't be retried.

cc @golang/tools-team @pjweinb @findleyr

[pjweinb]

Yes, something should be done. None of the alternatives seem perfect:

1. same behavior, but give up after 15 minutes. Then after a few weeks of this the report will be removed.
2. upload server doesn't take the report, but returns no error. Then the report is moved into the local telemetry/upload directory, and looks like it was uploaded.
3. upload server returns errors. What errors? (E.g., how many retryable erros are there for the upload process to think about?) What happens to the report for unretryable errors?

[hyangah]

I am thinking combination of these:

• 4XX status code can indicate client didn't behave as expected and client shouldn't need to retry. Alternatively, upload server can return http OK with a non-empty response (e.g. rejected) so client distinguished rejected upload requests.
• Client will keep retrying (same behavior as now - whenever the uploader runs) and after days of retrying, give up and delete the intermediate reports (.json). This handles 5XX cases.

[hyangah]

From today's discussion:

• 4XX status code - client shouldn't retry.
• Other status code - client should retry. Retry is done by leaving the intermediate json file untouched for 7days. Next upload processes will pick it up.

Not a release blocker. We will pick up in the next sprint.

[findleyr]

Tentatively assigning to @pjweinb. Peter -- does this seem like something you could pick up?

[pjweinb]

Looking at the code...

[pjweinb]

The current behavior of the upload package is to try to upload once, and if that fails, it just leaves the uploadable report in the 'local' directory, for some future invocation to try to upload.

But if the upload package is called many times from different packages, each will try to upload the report. It is likely better that however many upload processes run, to only try to upload a particular report once every (say) 12 hours. If this is the right solution it wold be fairly easy to implement.

[findleyr]

Decision:

The uploader should not retry 4xx error codes, but should retry 5xx. When not retrying, the uploader can just delete the uploadable report in the local directory.
Additionally, uploading should be rate limited, but that is integrated with telemetry.Start as part of #65500.

[gopherbot]

Change https://go.dev/cl/568238 mentions this issue: internal/upload: remove proposed upload on 4xx errors