WIP: [kots]: use Helm for the Installer job

[mrsimonemms]

Description

This changes the KOTS Installer job to use Helm for the installation of Gitpod rather than kubectl apply. IMPORTANT this still uses the Gitpod Installer for generating the Kubernetes YAML and is not in any way us going back to using Helm for installing Gitpod.

One thing we have to be aware of with KOTS is that we control what resources third-parties/customers have installed on their clusters under the umbrella of "Gitpod". It is key that we should remove resources when they've been finished with.

An example of this:

• day 1: a user installs Gitpod with in-cluster database/registry/object storage as a way of getting up and running quickly. They use this to get familiar with Gitpod and the installation process
• day 2: once they've decided they're happy, they change their KOTS configuration to migrate to external (and more performant) database/registry/object storage.

If we use kubectl apply, there is no history and any resources that are no longer required (eg, the mysql-0 statefulset) remain in the cluster. As they are no longer being used, they become orphaned from Gitpod installation YAML. More importantly, any persistent volume claims remain and the costs therein.

By switching to helm upgrade, we hook into Helm's history. Any resources that are removed between the previous and current installations are deleted.

Helm can be thought of as fulfilling two jobs:

1. generating the YAML from templates
2. deploying and managing resources

As we are still using the Installer to generate the YAML (ie, part 1), we're only using Helm for part 2.

The alternative way of doing this is to implement our own version of this. In reality, we'd likely end up building something very similar to how Helm works. By doing it this way, we reduce the time involved and benefit from all their testing.

How to test

Deploy via KOTS:

• install with in-cluster dependencies
• upgrade to external dependencies - watch the old resources disappear

Release Notes

[kots]: use Helm for the Installer job

Documentation

[github-actions]

⚠️ Hey reviewer! BE CAREFUL ⚠️
Review the code before opening in your Gitpod. .gitpod.yml was changed and it might be harmful.

[mrsimonemms]

Branch name too long. Replaced with #8491