[server] Make account deletion for fault tolerant

[geropl]

Description

#9645 revealed that we might fail during account deletion in face of (rare) bad data cases. This PR fixes that.

Related Issue(s)

Fixes #9645

How to test

• connect to DB
  • create workspace on this branch
  • kubectl port-forward mysql-0 3306
  • mysql -u gitpod -h 127.0.0.1 -p, use gitpod;
• select * from d_b_user; , copy your userId
• INSERT INTO d_b_workspace SET id = < some ws id>, ownerId = < your userId >, creationTime = '2022-04-29 13:53:14.251075', contextUrl = 'http://example.org', description = '', context = '{}', config = '{}';
• INSERT INTO d_b_workspace_instance SET id = < randomstromg >, workspaceId = < your ws id >, creationTime = '2022-04-29 13:53:14.251075', ideUrl = 'http://example.org', workspaceImage = 'some-image', region = 'does-not-exist', status ='{"phase": "pending"}', phasePersisted = 'pending';
• run select phasePersisted from d_b_workspace_instance;
• note how all entries are 'stopped' now ✔️

Release Notes

fix account deletion failing on bad DB state

Documentation

• /werft with-vm=true

[geropl]

/werft run

👍 started the job as gitpod-build-gpl-9645-acc-deletion.2
(with .werft/ from main)

[easyCZ]

Test outlined worked, code looks sensible.
Adding hold in case you want someone else more familiar with this to have a look.
/hold

[geropl]

Adding hold in case you want someone else more familiar with this to have a look.

Thx, but I think the test does cut it.

/unhold