[proxy] Proxy websocket traffic to slow-server deployment according to Sec-WebSocket-Protocol header

[andrew-farries]

Description

As part of #9198 we want to send a subset of websocket traffic from the dashboard to an instance of server that runs with a higher latency connection to the Gitpod database.

This PR changes the proxy Caddyfile to reverse proxy traffic to the new slow-server deployment depending on the value of the Sec-WebSocket-Protocol header that is sent by the browser websocket client during the websocket connection handshake.

#14752 demonstrates an approach that allows to selectively set the value of this header for some users, based on the value of a feature flag.

Related Issue(s)

Part of #9198

How to test

In the browser console, establish a new websocket connection to the Gitpod JSON RPC API:

const sock = new WebSocket('wss://af-use-slofc5943d8fc.preview.gitpod-dev.com/api/gitpod', 'slow-database')

The network tab should show a new websocket connection with these headers sent during the handshake:

Sending and receiving through the new connection should now incur higher than usual latency:

const msg = JSON.stringify({"jsonrpc":"2.0","id":12,"method":"getWorkspaces","params":{"limit":50}})
sock.send(msg)

the network tab shows a gap of ~1s between message send and response.

Tailing the logs of pods in the slow-server deployment reveals that they are handling websocket traffic.

Release Notes

NONE

Documentation

Werft options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-slow-database
• /werft with-large-vm
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh

[werft-gitpod-dev-com]

started the job as gitpod-build-af-proxy-to-slow-server-based-on-ws-header.4 because the annotations in the pull request description changed
(with .werft/ from main)

[geropl]

I'm going to review this PR now.

@andrew-farries I think we should not re-use any of the existing headers like Sec-WebSocket-Protocol that already carry meaning, but rather make up our own like X-Gitpod-SlowServer or so, and inject that via https://addons.mozilla.org/en-US/firefox/addon/modify-header-value/ or similar extensions.

[geropl]

🧡 Love this change - simply on-point 😄

I'd only suggest we user a custom header instead of the websocket one. Also, we should check this on other HTTP routes that are sent against server as well 👇 .

[andrew-farries]

I think we should not re-use any of the existing headers like Sec-WebSocket-Protocol that already carry meaning, but rather make up our own like X-Gitpod-SlowServer or so, and inject that via https://addons.mozilla.org/en-US/firefox/addon/modify-header-value/ or similar extensions.

If we go down the route of using browser extensions to inject extra headers then we eliminate the possibility of being able to control the feature by using a feature flag. By using the Sec-WebSocket-Protocol header we don't need to use browser extensions and can roll this out more widely more easily (eg start with just webapp team members, then rollout to all Gitpodders).

I'd only suggest we user a custom header instead of the websocket one. Also, we should check this on other HTTP routes that are sent against server as well

This is an advantage of using a browser extension; with the Sec-WebSocket-Protocol header we can only check this on the websocket connection handshake. IMO, we should roll out this websocket-only version and then see if we need further validation. It may become clear that the latency is already too high.

[easyCZ]

I'm assuming the formatting is based on the Caddy fmt command

[andrew-farries]

correct 👍

[easyCZ]

LGTM

/hold in case you wanna wait for Gero

[andrew-farries]

Another race between tide and roboquat here 😭