Epic: Get rid of OTS (One-Time Secret)

[csweichel]

Summary

The one-time secret (OTS) mechanism is used to deliver secrets to the workspace cluster. It's not location-aware which breaks workspace startup across regions.

Context

The one-time secret (OTS) mechanism is used to deliver secrets to the workspace cluster. During workspace startup, server will create up to three OTS:

• one for the SCM token
• one for the Gitpod token
• potentially one for environment variables

A one-time secret is stored in the database and identified by a UUID. Using this UUID it can be downloaded once, after which it's removed from the database. When the OTS is created, server produces a URL from which the OTS can be downloaded. This URL is not location-specific, but uses the load balancer (i.e. gitpod.io/...).

Because the different regions use different databases which are synchronised using db-sync, and because the OTS URL is not region-aware, a workspace created in another region will race db-sync. This can lead to workspace startup failure.

Value

Removing OTS will

• reduce failure modes (see Failed to download OTS in US cluster (possibly happens for prebuilds, only) #8096)
• enable cross-region prebuilds (see Run prebuilds in the storage-bucket's region #6650)
• reduce complexity in webapp

Acceptance Criteria

This work is complete when

• there's a secure way to keep secrets on the workspace side (Kubernetes secrets qualify here)
• the OTS mechanism is no longer in use for shipping secrets
• the OTS mechanism has been removed from the code-base

Measurement

We are successful here when there's no loss of functionality, and no more need for the OTS mechansim.

Tasks

• Add "secret" support to ws-manager, where a StartWorkspace request can carry named secrets
• Ship the SCM token as named secret and pass it to ws-daemon during InitWorkspace
• Ship the Gitpod token as named secret and pass it as environment variable to supervisor
• Ship the user's environment variables as named secrets and pass as environment variables to the workspace
• Unsupported initializer error opening snapshot in GItpod (internal) preview environment (breaks without OTS) #12554
• [Get rid of OTS] Add an integration test for protected_secrets feature flag #11318
• [Get rid of OTS] Load testing #13490
• [installer] Support enabling protected secrets #13484
• https://github.com/gitpod-io/ops/issues/5608
• Enable protected secrets by default in the installer for self-hosted #13632
• Enable protected secrets by default in the saas without using configcat #13633
• Remove old OTS code #13634

[atduarte]

Added to workspace teams' project, even though it's shared work between WebApp and Workspace, to avoid this being forgotten.

[geropl]

@atduarte Added for Team WebApp as well! Please notify when you have the interface changes (w/o impl, maybe). Happy to jump on the changes on our end.