[installer]: create readme

Author: Simon Emms

installer/README.md

@@ -0,0 +1,374 @@
+<p align="center">
+  <a href="https://www.gitpod.io">
+    <img src="https://github.com/raw/gitpod-io/gitpod/master/components/dashboard/src/icons/gitpod.svg" height="60">
+    <h3 align="center">Gitpod</h3>
+  </a>
+  <p align="center">Always ready-to-code.</p>
+</p>
+
+# Installer
+
+The best way to get started with Gitpod
+
+[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-908a85?logo=gitpod)](https://gitpod.io/from-referrer/)
+
+# Requirements
+
+- A machine running Linux/MacOS and Docker
+    - Windows is not currently supported, but will be in future
+- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed
+- A [Kubernetes cluster configured](https://www.gitpod.io/docs/self-hosted/latest/installation)
+
+Or, [open a Gitpod workspace](https://gitpod.io/from-referer/)
+
+The process to install Gitpod is:
+1. generate a base config
+2. amend the config for your own use-case
+3. validate
+4. render the Kubernetes YAML
+5. `kubectl apply`
+
+# Quickstart
+
+## Create a gitpod-installer script
+
+> This script is available in [/scripts/gitpod-installer.sh](../scripts/gitpod-installer.sh)
+
+```shell
+#!/bin/bash
+
+VERSION="${VERSION:-main.1844}" # Build ID taken from https://werft.gitpod-dev.com/
+
+docker run -it --rm \
+  -v $PWD:/gitpod \
+  -v $HOME/.kube:/root/.kube \
+  eu.gcr.io/gitpod-core-dev/build/installer:${VERSION} \
+  $@
+```
+
+Save this in your `$PATH` as `gitpod-installer` and make executable
+
+## Generate the base config
+
+```shell
+gitpod-installer init > gitpod.config.yaml
+```
+
+## Customise your config
+
+There are many things you can change in your config, which can be found in
+the [Config Guide](#config).
+
+For the purposes of a quickstart, just change the `domain` to one of your
+own.
+
+## Validate
+
+```shell
+# Checks the validity of the configuration YAML
+gitpod-installer validate config --config /gitpod/gitpod.config.yaml
+```
+
+Any errors here must be fixed before deploying. See [Config](#config) for
+more details.
+
+```shell
+# Checks that your cluster is ready to install Gitpod
+gitpod-installer validate cluster --kubeconfig /root/.kube/config --config /gitpod/gitpod.config.yaml
+```
+
+Any errors here must be fixed before deploying. See [Cluster Dependencies](#cluster-dependencies)
+for more details.
+
+## Render the YAML
+
+```shell
+gitpod-installer render --config /gitpod/gitpod.config.yaml > gitpod.yaml
+```
+
+## Deploy
+
+```shell
+kubectl apply -f gitpod.yaml
+```
+
+After a few minutes, your Gitpod installation will be available on the
+specified `domain`.
+
+## Uninstallation
+
+The Installer generates a ConfigMap with the metadata of every Kubernetes
+object generated by the Installer. This can be retrieved to remove Gitpod
+from your cluster.
+
+```shell
+kubectl get configmaps gitpod-app -o jsonpath='{.data.app\.yaml}' \
+  | kubectl delete -f - # Piping to this will delete automatically
+```
+
+**Important**. This may leave certain objects still in your Kubernetes
+cluster. This will include `Secrets` generated from internal `Certificates`
+and `PersistentVolumeClaims`. These will need to be manually deleted.
+
+---
+
+# What is installed
+
+- All Gitpod components
+- Container registry*
+- MySQL database*
+- Jaeger operator*
+- RabbitMQ
+- Minio object storage*
+
+\* By default, these dependencies are installed if the `inCluster` setting
+is `true`. External dependencies can be used in their place
+
+# Config
+
+> Not every parameter is discussed in this table, just ones that are likely
+> to need changing. The full config structure is available in [config.go](/installer/pkg/config/v1/config.go).
+
+| Property | Required | Description | Notes |
+| --- | --- | --- | --- |
+| `domain` | Y | The domain to deploy to | This will need to be changed on every deployment |
+| `kind` | Y | Installation type to run - for most users, this will be `Full` | Can be `Full`, `Meta` or `Workspace` |
+| `metadata.region` | Y | Location for your `objectStorage` provider | If using Minio, set to `local` |
+| `workspace.runtime.containerdRuntimeDir` | Y | The location of containerd on host machine | Common values are: <ul><li>`/run/containerd/io.containerd.runtime.v2.task/k8s.io` (K3s)</li><li>`/var/lib/containerd/io.containerd.runtime.v2.task/k8s.io` (AWS/Azure/GCP)</li><li>`/run/containerd/io.containerd.runtime.v1.linux/k8s.io`</li><li>`/run/containerd/io.containerd.runtime.v1.linux/moby`</li></ul> |
+| `workspace.runtime.containerdSocket` | Y | The location of containerd socket on the host machine |
+| `workspace.runtime.fsShiftMethod` | Y | File system | Can be either `fuse` (fuse-overlayfs) or `shiftfs`. This depending upon your host OS/distribution. If unsure, use `fuse`. |
+
+## Auth Providers
+
+> This may move to a secret in future [#6867](https://github.com/gitpod-io/gitpod/issues/6867)
+
+Gitpod must be connected to a Git provider. This can be done via the
+dashboard on first load, or by providing `authProviders` configuration.
+
+```yaml
+authProviders:
+  - id: Public-GitHub
+    host: github.com
+    type: GitHub
+    oauth:
+      clientId: xxx
+      clientSecret: xxx
+      callBackUrl: https://$DOMAIN/auth/github.com/callback
+      settingsUrl: xxx
+```
+
+## In-cluster vs External Dependencies
+
+Gitpod requires certain services for it to function correctly. The Installer
+provides all of these in-cluster, but they can be configured to use services
+external to the cluster.
+
+To use the in-cluster dependency, set `inCluster` to be `true`.
+
+## Container Registry
+
+```yaml
+containerRegistry:
+  inCluster: false
+  external:
+    url: <url of registry>
+    certificate:
+      kind: secret
+      name: container-registry-token
+```
+
+The `container-registry-token` secret must contain a `.dockerconfigjson`
+key - this can be created by using the `kubectl create secret docker-registry`
+[command](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line).
+
+## Database
+
+Gitpod requires an instance of MySQL 5.7 for data storage.
+
+The default encryption keys are `[{"name":"general","version":1,"primary":true,"material":"4uGh1q8y2DYryJwrVMHs0kWXJlqvHWWt/KJuNi04edI="}]`
+
+### Google Cloud SQL Proxy
+
+If using a GCP SQL instance, a [Cloud SQL Proxy](https://cloud.google.com/sql/docs/mysql/sql-proxy)
+connection can be used.
+
+```yaml
+database:
+  inCluster: false
+  cloudSQL:
+    instance: <PROJECT_ID>:<REGION>:<INSTANCE>
+    serviceAccount:
+      kind: secret
+      name: cloudsql-token
+```
+
+The `cloudsql-token` secret must contain the following key/value pairs:
+ - `credentials.json` - GCP Service Account key with `roles/cloudsql.client` role
+ - `encryptionKeys` - database encryption key. Use default value as above if unsure
+ - `password` - database password
+ - `username` - database username
+
+### External Database
+
+For all other connections, use an external database configuration.
+
+```yaml
+database:
+  inCluster: false
+  external:
+    certificate:
+      kind: secret
+      name: database-token
+```
+
+The `database-token` secret must contain the following key/value pairs:
+- `encryptionKeys` - database encryption key. Use default value as above if unsure
+- `host` - IP or URL of the database
+- `password` - database password
+- `port` - database port, usually `3306`
+- `username` - database username
+
+## Object Storage
+
+Gitpod supports the following object storage providers:
+
+### GCP
+
+```yaml
+metadata:
+  region: <gcp-region-code, eg europe-west2>
+objectStorage:
+  inCluster: false
+  cloudStorage:
+    project: <PROJECT_ID>
+    serviceAccount:
+      kind: secret
+      name: gcp-storage-token
+```
+
+The `gcp-storage-token` secret must contain the following key/value pairs:
+- `service-account.json` - GCP Service Account key with `roles/storage.admin` and `roles/storage.objectAdmin` roles
+
+### Azure
+
+> Azure Blob Storage is not S3 compatible. This uses Minio Gateway to
+> provide an S3 interface
+
+```yaml
+metadata:
+  region: <azure-region-code, eg northeurope>
+objectStorage:
+  inCluster: false
+  azure:
+    certificate:
+      kind: secret
+      name: az-storage-token
+```
+
+The `az-storage-token` secret must contain the following key/value pairs:
+- `accountName` - the globally-unique storage account name
+- `accountKey` - access key for the storage account
+
+### S3
+
+> This is currently only tested with AWS. Other S3-compatible providers should
+> work but there may compatability issues - please raise a ticket if you have
+> issues with other providers
+
+```yaml
+metadata:
+  region: <aws-region-code, eg eu-west-2>
+objectStorage:
+  inCluster: false
+  s3:
+    endpoint: s3.amazonaws.com
+    credentials:
+      kind: secret
+      name: s3-storage-token
+```
+
+The `s3-storage-token` secret must contain the following key/value pairs:
+- `username` - username that has access to S3 account
+- `password` - password that has access to S3 account
+
+> In AWS, the username/password are an IAM user's credentials with
+> `AmazonS3FullAccess` policy
+
+# Cluster Dependencies
+
+In order for the deployment to work successfully, there are certain
+dependencies that need to be installed.
+
+## Kernel and Runtime
+
+Your Kubernetes nodes must have the Linux kernel v5.4.0 or above and
+have a containerd runtime.
+
+## Affinity Labels
+
+Your Kubernetes nodes must have the following labels applied to them:
+- `gitpod.io/workload_meta`
+- `gitpod.io/workload_ide`
+- `gitpod.io/workload_workspace_services`
+- `gitpod.io/workload_workspace_regular`
+- `gitpod.io/workload_workspace_headless`
+
+It is recommended to have a minimum of two node pools, grouping the `meta`
+and `ide` nodes together and the `workspace` nodes together.
+
+## TLS certificates
+
+It is a requirement that a certificate secret exists, named as per
+`certificate.name` in your config YAML with `tls.crt` and `tls.key`
+in the secret data. How this certificate is generated is entirely your
+choice - we suggest [Cert Manager](https://cert-manager.io/) for
+simplicity, however any certificate authority can be used by creating a
+Kubernetes secret.
+
+The certificate must be associated with the following domains (where
+`$DOMAIN` is the value in config `domain`):
+ - `$DOMAIN`
+ - `*.$DOMAIN`
+ - `*.ws.$DOMAIN`
+
+### Cert Manager
+
+Cert Manager **MUST** be installed to your cluster. In order to secure
+communication between the various components, the application creates
+internally which are created using the Cert Manager `Certificate` and
+`Issuer` Custom Resource Definitions.
+
+```shell
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm upgrade \
+    --atomic \
+    --cleanup-on-fail \
+    --create-namespace \
+    --install \
+    --namespace='cert-manager' \
+    --reset-values \
+    --set installCRDs=true \
+    --set 'extraArgs={--dns01-recursive-nameservers-only=true,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}' \
+    --wait \
+    cert-manager \
+    jetstack/cert-manager
+```
+
+# FAQs
+
+## Why are you writing your own Installer instead of using Helm/Kustomize/etc?
+
+The Installer is a complete replacement for our Helm charts. Over time,
+this had grown to be too complex to effectively support and was a barrier
+to entry for new users - the base config was many hundreds of lines long
+and did not have effective validation on it. By contrast, the Installer's
+config is < 50 lines long and can be fully validated before running.
+
+# Todo
+
+PRs/comments welcome
+
+- [ ] [Improve distribution of gitpod-installer binaries](https://github.com/gitpod-io/gitpod/issues/6766)
+- [ ] [Configure and document S3 object storage](https://github.com/gitpod-io/gitpod/issues/6772)
+- [ ] [Create a Gitpod workspace that can be used for installing a Gitpod instance](https://github.com/gitpod-io/gitpod/issues/6801)

scripts/gitpod-installer.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+VERSION="${VERSION:-main.1844}" # Build ID taken from https://werft.gitpod-dev.com/
+
+docker run -it --rm \
+  -v $PWD:/gitpod \
+  -v $HOME/.kube:/root/.kube \
+  eu.gcr.io/gitpod-core-dev/build/installer:${VERSION} \
+  $@