Cordon build nodes if their disk is more than 80% full

[mads-hartmann]

Description

This changes the platform-trigger-werft-cleanup job so that instead of triggering a separate Werft job that is scheduled on each build node which performs a cleanup on the individual nodes, it will now SSH out to the instance and simply cordon it if the disk is more than 80% full.

This approach has two benefits

1. By cordoning the nodes rather than trying to perform cleanup we simplify the implementation and makes it more robust against new things filling up the disk in the future.
2. The previous cleanup job used docker system prune which had the downside of potentially breaking currently running builds. Because of this we couldn't run the job too often. Now we can run the job as often as we'd like. I have changed it to every 4 hours

The relevant changes to the service account has been made here https://github.com/gitpod-io/ops/pull/2375

Related Issue(s)

Fixes https://github.com/gitpod-io/ops/issues/2050
Fixes https://github.com/gitpod-io/ops/issues/1227

How to test

See the comments in the code on how to run this. Here are two examples

• Here's a job which cordoned a node (I ran it with the threshold set to 30) (link). I manually uncordoned the node again after.
• Here's a job where I had set the threshold to 80 so it skipped both nodes (link).

Release Notes

NONE

Documentation

N/A

[meysholdt]

nit: put the "%" after the number to make the output more readable :)

[meysholdt]

🎉 for simplicity and robustness.

Nit:
The new code only checks if the localSSD is full and ignores the root volume.
Not sure if the root volume sometimes runs full, too.
Either way, I think this is out of scope for this PR and just something to keep in mind.

[mads-hartmann]

@meysholdt Thanks for the review! I haven't seen the root disk run full yet but lets keep an eye out for it ☺️