[installer]: make repo mirroring easier

[mrsimonemms]

Repository mirroring is something that is likely to be used by customers in an air-gapped environment. Currently, we have repository as a parameter in the config, which shouldn't need to change.

Create a command in the installer that outputs all images (including Helm ones) to Stdout in a JSON format to allow air-gapped users to easily mirror these images.

We cannot rely on the repository property when generating the image paths as Gitpod images will be in the eu.gcr.io/gitpod-core-dev/build repo and the Helm dependencies will be in their configured path

[SirLemyDanger]

The problem might me docker. I am on a k3s cluster and I have configured my cluster to transparently pull images from eu.gcr.io via a private registry. The resulting image name, although not downloaded from eu.gcr.io directly still starts with eu.gcr.io.

So an alternative could be to run your installer in the cluster and not use docker at all.

[mrsimonemms]

@SirLemyDanger thanks for the input. Would that work for non-Gitpod repos as well - for example, we pull ubuntu, alpine quay.io/brancz/kube-rbac-proxy and bitnami/rabbitmq among others

[SirLemyDanger]

I had to setup a mirror for each of the used repositories. At the moment I mirror docker.io, gcr.io, eu.gcr.io and quay.io. However I'm not sure which of these I needed for gitpod.
So for each of the 4 mirrors I have one entry in the registries.yaml in /etc/rancher/k3s/

Can be tested with crictl pull alpine

[mrsimonemms]

This script works quite nicely. Generate your installer config using the default repository: eu.gcr.io/gitpod-core-dev/build value (eg, installer render --config config.yaml > default-repo.yaml) and then run this script:

#!/bin/bash

set -e

TARGET_FILE=default-repo.yaml # Set to your file name
TARGET_REPO=my-private-registry.com/gitpod # Set to your private registry
GITPOD_REPO=eu.gcr.io/gitpod-core-dev/build

# Get the image tags, removing any whitespace, tabs or array markers from start of line
k8s_images=$(cat "${TARGET_FILE}" | grep "image:" | sed 's/^[ \t-]*image\:[ \t]*//' | sed 's/"//g')

# Get any strings that start with GITPOD_REPO - these are supervisor and workspace images
generic_images=$(cat "${TARGET_FILE}" | grep "${GITPOD_REPO}" | sed 's/"//g' | sed 's/^[ \ta-zA-Z-]*:[ \t]*//' | sed 's/,//g')

images="${k8s_images} ${generic_images}"

for image in $images
do
  echo "======"
  echo "New image to pull: ${image}"

  if [[ $image = $GITPOD_REPO* ]];
  then
    # Gitpod image - replace old repo with new repo
    TARGET_IMAGE="${image/$GITPOD_REPO/$TARGET_REPO}"
  else
    # Third party image - convention is to remove URL and keep owner name
    TARGET_IMAGE=$(echo "${image}" | sed -e 's/^[a-zA-Z\.]*\//TARGET_REPO\//g')
    TARGET_IMAGE="${TARGET_IMAGE/TARGET_REPO/$TARGET_REPO}"
  fi

  # Pull the image - if this fails, ignore
  docker pull "${image}" || continue

  # Tag the image
  docker tag "${image}" "${TARGET_IMAGE}"

  echo "New tag created: ${TARGET_IMAGE}"

  # Push the image
  docker push "${TARGET_IMAGE}"

  echo "End of ${image}"
  echo "======"
done

When this ticket gets pulled in, this will likely be the basis of how it's actually achieved but inside the Golang binary