[observability] Get better insight to disk I/O

[kylos101]

Is your feature request related to a problem? Please describe

Our current node overview lacks read/write specs for node disk I/O.. This makes it difficult to tell if we're stressing disks when troubleshooting, but also in general, from a planning standpoint.

Describe the behaviour you'd like

See this existing dashboard for workspace and meta clusters, with the ability to filter by cluster, node pool, and node.. If you watch the video (below), Arthur shows how to update an existing dashboard to work with variables, so it can be used at Gitpod. Add variables so that we can filter by cluster, node pool, and node.

Additional context

This will help, for example, when troubleshooting issues related to heavy I/O like backup failures, or many backups happening at the same time.

[kylos101]

@gitpod-io/platform hey there, can you help me? I want to schedule this work for the workspace team, so that we can have this new dashboard for workspace clusters (which I think will benefit all of our clusters), but have some concerns.

1. Can we consume external dashboards, given our libjsonnet configuration at Gitpod? Given the picture below, I assume yes, we should just have to import it (in a preview environment), alter it, export the JSON, and persist it.
2. It seems like we'll have to modify the dashboard, to support cluster filtering, but it appears to already support node and node pool, which is great!
3. Can you think of anything else we should change with this dashboard, if we decide to use it?

As a test, I did:

[kylos101]

@aledbf I am soliciting feedback from the Platform team.

It seems like this dashboard would help with minimal customization (we could add cluster)! 😍

[mads-hartmann]

@ArthurSens can you follow up with @kylos101 to clarify how to approach this best.

[ArthurSens]

Hey @kylos101, I've recorded a loom trying to explain how I'd get the mentioned dashboard into production:

https://www.loom.com/share/ca15ea6ab12c465f9a72ff75e8f80ad6

We also have some written documentation here, but please ping us if anything isn't clear 🙂

[mads-hartmann]

Thanks @ArthurSens - @kylos101 I'm removing this from the platform project, but please re-add if you need more input from us.

[kylos101]

That's perfect, thank you both! @ArthurSens I appreciate the video and promise to carve out time to look at it soon.