[db-sync] Consider alternatives to the `auth/workspace-cookie` redirect

[andrew-farries]

See internal discussion.

[geropl]

Some prior information why we use a fetch to get that cookie: We want it to be httpOnly, so the browser (JS runtime) never has access to it.

There are two parts of code that are relevant:

• the server handler: /auth/workspace-cookie
• the client code calling it: ensureWorkspaceAuth

I personally observed that during regular workspace runtime, I got redirected to that endpoint, and it would return plain codes of 404 (and maybe 200?) and got stuck there.
Looking at the handler code, there seem to be a couple of potential exits we might be triggering:

• "attempted to fetch workspace cookie without instance ID" (code 400, 0 log entries last 31 days)
• "attempted to fetch workspace cookie for non-existent workspace instance" (code 400, ~1550 log entries)
• "unauthorized attempted to fetch workspace cookie" (code 403, 0 log entries)
• "attempted to fetch workspace cookie, but instance has no owner token" (200 (:exclamation: :question: ), logs are debug only enabled debug logs: 119 entries in 1 hour)

Observations:

• the "non-existent workspace instance" seems to be a bigger issue, beyond the scope of the slow-server experiment. By manually looking at a couple of affected users it feels as if a lot of those are coming from China/East Asia, so maybe we're this is a general problem indeed. Feels like we're looking at a race - question is where it's coming from. The affected workspaces I checked have all been started in the US cluster. But maybe their frontends regularly switch to EU due to changing latency (same DB setup we have with slow-server). This would go a way with a single DB. OR we have a different race, and it will persist
• the client code seems to be very naive, and not matching the handler logic. On error, for instance, the client will simply redirect to the handler's URL, no matter the code. But the handler does not support, that, at all. 🙈 E.g., never does redirect away... ❌

[geropl]

We don't have any insights into client metrics. But from team internal testing it seems this is solved now.