[Installer]: simplify container image mirroring #7156

mrsimonemms · 2021-12-09T15:37:05Z

Description

Image mirroring is something that is crucial to air-gapped installations. As part of that, repository mirroring is necessary to allow users to pull images into their own container registries.

This adds a gitpod-installer render mirror command. It reads the current gitpod-config.yaml, generates the YAML with the default Gitpod registry, finds any image: tags in the YAML and changes the URLs to the registry that the user chooses.

The data is output in the format:

type Output struct {
	Original string `json:"original"`
	Target   string `json:"target"`
}

It also gives the option to output in either JSON or YAML format.

Related Issue(s)

Fixes #6756

How to test

Follow the instructions in gitpod-installer mirror list -h

To mirror a repo, can use jq to iterate through:

for row in $(gitpod-installer mirror list --config ./config.yaml ./versions.yaml -o json | jq -c '.[]'); do
    original=$(echo $row | jq -r '.original')
    target=$(echo $row | jq -r '.target')
    docker pull $original
    docker tag $original $target
    docker push $target
done

Then deploy a cluster as-per #6236

Release Notes

[Installer]: simplify container image mirroring

Documentation

codecov · 2021-12-09T15:37:25Z

Codecov Report

Merging #7156 (ad5937e) into main (2c8b59b) will decrease coverage by 13.28%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #7156       +/-   ##
==========================================
- Coverage   19.04%   5.76%   -13.29%     
==========================================
  Files           2      13       +11     
  Lines         168    1162      +994     
==========================================
+ Hits           32      67       +35     
- Misses        134    1094      +960     
+ Partials        2       1        -1

Flag	Coverage Δ
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
installer-raw-app	`5.76% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/local-app/pkg/auth/pkce.go
components/local-app/pkg/auth/auth.go
installer/pkg/common/common.go	`4.64% <0.00%> (ø)`
installer/pkg/common/storage.go	`0.00% <0.00%> (ø)`
installer/pkg/components/ws-manager/role.go	`0.00% <0.00%> (ø)`
...staller/pkg/components/ws-manager/networkpolicy.go	`0.00% <0.00%> (ø)`
installer/pkg/common/display.go	`0.00% <0.00%> (ø)`
installer/pkg/components/ws-manager/configmap.go	`29.71% <0.00%> (ø)`
...components/ws-manager/unpriviledged-rolebinding.go	`0.00% <0.00%> (ø)`
installer/pkg/components/ws-manager/tlssecret.go	`0.00% <0.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c8b59b...ad5937e. Read the comment docs.

mrsimonemms · 2021-12-10T15:16:10Z

/werft run

👍 started the job as gitpod-build-sje-installer-repo-mirror.2

mrsimonemms · 2021-12-15T09:45:50Z

/verify-owners

mrsimonemms · 2021-12-15T09:46:38Z

/werft run

👍 started the job as gitpod-build-sje-installer-repo-mirror.4

corneliusludmann · 2021-12-15T12:04:38Z

Like it! 🤩

Here is the result of my first try. Didn't debug it but probably we could get a better error message (minor)!? 😇

$ cd installer
$ go build
$ docker run --rm eu.gcr.io/gitpod-core-dev/build/versions:sje-installer-repo-mirror.4 cat versions.yaml > versions.yaml
$ ./installer render mirror --config ./example-config.yaml ./versions.yaml
panic: cannot parse image ref eu.gcr.io/gitpod-core-dev/build/content-service:: invalid reference format

goroutine 1 [running]:
github.com/gitpod-io/gitpod/installer/pkg/common.ImageName({0x2cd35a3, 0x1}, {0x2cafbdc, 0x0}, {0x0, 0x0})
        /workspace/gitpod/installer/pkg/common/common.go:343 +0x1fb
github.com/gitpod-io/gitpod/installer/pkg/components/content-service.deployment(0xc000727200)
        /workspace/gitpod/installer/pkg/components/content-service/deployment.go:44 +0x1ee
github.com/gitpod-io/gitpod/installer/pkg/common.CompositeRenderFunc.func1(0x0)
        /workspace/gitpod/installer/pkg/common/render.go:29 +0xc6
github.com/gitpod-io/gitpod/installer/pkg/common.CompositeRenderFunc.func1(0x0)
        /workspace/gitpod/installer/pkg/common/render.go:29 +0xc6
github.com/gitpod-io/gitpod/installer/pkg/common.CompositeRenderFunc.func1(0x29e6240)
        /workspace/gitpod/installer/pkg/common/render.go:29 +0xc6
github.com/gitpod-io/gitpod/installer/pkg/common.CompositeRenderFunc.func1(0xc0008dfc30)
        /workspace/gitpod/installer/pkg/common/render.go:29 +0xc6
github.com/gitpod-io/gitpod/installer/cmd.renderKubernetesObjects({0xc000542c40, 0x2}, 0xc000b06600)
        /workspace/gitpod/installer/cmd/render.go:117 +0x352
github.com/gitpod-io/gitpod/installer/cmd.glob..func2(0x4188100, {0x2c94865, 0x3, 0x3})
        /workspace/gitpod/installer/cmd/render-mirror.go:80 +0x114
github.com/spf13/cobra.(*Command).execute(0x4188100, {0xc0006239b0, 0x3, 0x3})
        /workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x4187e80)
        /workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/gitpod-io/gitpod/installer/cmd.Execute()
        /workspace/gitpod/installer/cmd/root.go:22 +0x25
main.main()
        /workspace/gitpod/installer/main.go:10 +0x17

Will have a closer look after lunch. 🥣

mrsimonemms · 2021-12-15T12:07:04Z

@corneliusludmann when building your Installer from source, you have to inject the versions file (handled by werft normally, and part of #6766). Can show you how to do that after lunch

That error message is written by the Docker image formatter

corneliusludmann

Left a few comments. Please let me know what you think about it. 🙏

installer/cmd/render-mirror.go

corneliusludmann · 2021-12-15T14:24:05Z

installer/cmd/render-mirror.go

+		rawImages := make([]string, 0)
+		for _, item := range k8s {
+			rawImages = append(rawImages, getPodImages(item)...)
+			rawImages = append(rawImages, getGenericImages(item)...)


Just to understand what happens: We are looking in the generated k8s output for everything that looks like an image, right?

When we install Gitpod, is there a place where we replace all these images with our mirrored ones? Or is that still to come?

Yes, this has been done. The common.ImageName function is used for images where we specify the container path as part of the called - this is the case for all Gitpod images, but also a few others (eg, the kube-rbac-proxy image).

There is also the ThirdPartyContainerRepo function if it's something which may be a non-Gitpod registry or uses the registry if it's mirrored. The default value can legitimately be "" as it will default to docker.io, such as here

I've tested all this with a third-party registry and it works ok - happy to demo it to you if you'd like to see it

OK, I see when I change repository in the config, the third-party images are fetched from this repos as well.

However, I wonder what the use case for the --exclude-third-party flag would look like? This flag would only make sense when the users are able to tell the installer that all Gitpod images should be fetched from their private repo but the third-party images from the original repo, wouldn't it? Or is it more that the users have a mirror for the third-party images already in place but need to add the Gitpod images to their repo as well? Is that actually a possible use case?

Yes, talking with some of the community, it appears that there is a desire to only mirror the Gitpod images as they already mirror public registries (quay.io, docker.io etc). You're right that it will likely be an edge case, but there is a legitimate use case for it (and repo mirroring is an edge case anyway)

mrsimonemms · 2021-12-15T21:56:01Z

/werft run

👍 started the job as gitpod-build-sje-installer-repo-mirror.9

mrsimonemms · 2021-12-20T15:51:59Z

/werft run

👍 started the job as gitpod-build-sje-installer-repo-mirror.12

mrsimonemms · 2021-12-20T16:30:24Z

/werft run

👍 started the job as gitpod-build-sje-installer-repo-mirror.13

csweichel · 2021-12-20T17:10:29Z

At this stage, I see this ending here. I'm not against putting it into a root mirror command if you think this would be preferable. I can see it does give us future wiggle room

Considering that the CLI is basically API, this might be a worthwhile bit of future-proofing. Note: it would just be mirror but rather mirror list

meysholdt · 2021-12-23T09:29:57Z

I tried this PR by running

#!/bin/bash

./installer init > c.yaml

./installer validate config --config c.yaml

(docker run eu.gcr.io/gitpod-core-dev/build/versions:main.2087 cat versions.yaml) > versions.yaml

for row in $(./installer render mirror --config ./c.yaml --debug-version-file versions.yaml | jq -c '.[]'); do
    original=$(echo $row | jq -r '.original')
    target=$(echo $row | jq -r '.target')
    docker pull $original
    docker tag $original $target
    docker push $target
done

However, I got the error

{
  "valid": true
}
parse error: Invalid numeric literal at line 2, column 0

The output of installer render mirror seems to be the full k8s yaml rather than a list of images. Was I too early to test this? :)

mrsimonemms · 2021-12-23T11:14:02Z

My bad @meysholdt. I updated the command as per @csweichel's request so that the command is mirror list and didn't update the PR.

Should be working ok for you now. Apologies and good catch

stale · 2022-01-02T13:05:10Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

The output gives the original image name and tag and generates the new image name to be used

mrsimonemms · 2022-01-04T08:51:49Z

/unhold

csweichel · 2022-01-04T16:36:07Z

/lgtm

roboquat · 2022-01-04T16:36:11Z

LGTM label has been added.

Git tree hash: 16ec954fcaad74d1e725572fb798a7cd73401e78

roboquat · 2022-01-04T16:36:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: corneliusludmann, csweichel, laushinka

Associated issue: #6756

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~installer/OWNERS~~ [corneliusludmann]
~~installer/pkg/components/image-builder-mk3/OWNERS~~ [csweichel]
~~installer/pkg/components/server/OWNERS~~ [laushinka]
~~installer/pkg/components/ws-scheduler/OWNERS~~ [csweichel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

roboquat added release-note do-not-merge/work-in-progress team: workspace Issue belongs to the Workspace team size/XL labels Dec 9, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 20d5614 to 475604a Compare December 10, 2021 15:12

mrsimonemms marked this pull request as ready for review December 10, 2021 15:16

roboquat removed the do-not-merge/work-in-progress label Dec 10, 2021

mrsimonemms requested a review from a team December 10, 2021 15:16

mrsimonemms added the team: delivery Issue belongs to the self-hosted team label Dec 13, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 475604a to 66035ee Compare December 14, 2021 17:07

mrsimonemms requested a review from a team December 15, 2021 09:46

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 66035ee to 4a0ece1 Compare December 15, 2021 13:18

corneliusludmann reviewed Dec 15, 2021

View reviewed changes

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 4a0ece1 to 0917019 Compare December 15, 2021 21:03

roboquat added the team: webapp Issue belongs to the WebApp team label Dec 15, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch 2 times, most recently from 93af1c2 to 66e411e Compare December 15, 2021 21:29

mrsimonemms requested a review from corneliusludmann December 15, 2021 21:31

mrsimonemms marked this pull request as draft December 16, 2021 11:15

roboquat added the do-not-merge/work-in-progress label Dec 16, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 66e411e to d4da07e Compare December 16, 2021 11:31

mrsimonemms marked this pull request as ready for review December 16, 2021 11:43

roboquat removed the do-not-merge/work-in-progress label Dec 16, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from d9c2109 to 7d9d2ff Compare December 21, 2021 10:08

roboquat removed the lgtm label Dec 21, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 7d9d2ff to fb25b91 Compare December 21, 2021 10:12

mrsimonemms requested a review from csweichel December 21, 2021 11:00

mrsimonemms added the installer: needs interface change Change required to input, output or configuration file(s) label Dec 21, 2021

mrsimonemms force-pushed the sje/installer-repo-mirror branch from fb25b91 to 7b9b139 Compare December 23, 2021 11:13

stale bot added the meta: stale This issue/PR is stale and will be closed soon label Jan 2, 2022

Simon Emms added 3 commits January 4, 2022 08:42

[installer]: refactor render function into a callable function

3eafd8c

[installer]: allow workspace image to be mirrored

4951f52

[installer]: output all images to allow easy mirroring of containers

ad5937e

The output gives the original image name and tag and generates the new image name to be used

mrsimonemms force-pushed the sje/installer-repo-mirror branch from 7b9b139 to ad5937e Compare January 4, 2022 08:43

stale bot removed the meta: stale This issue/PR is stale and will be closed soon label Jan 4, 2022

roboquat removed the do-not-merge/hold label Jan 4, 2022

roboquat assigned csweichel Jan 4, 2022

roboquat added the lgtm label Jan 4, 2022

roboquat added the approved label Jan 4, 2022

roboquat merged commit 6da5600 into main Jan 4, 2022

roboquat deleted the sje/installer-repo-mirror branch January 4, 2022 16:37

roboquat added deployed: webapp Meta team change is running in production deployed: workspace Workspace team change is running in production labels Jan 6, 2022

[Installer]: simplify container image mirroring #7156

[Installer]: simplify container image mirroring #7156

Uh oh!

Conversation

mrsimonemms commented Dec 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

How to test

Release Notes

Documentation

Uh oh!

codecov bot commented Dec 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mrsimonemms commented Dec 10, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrsimonemms commented Dec 15, 2021

Uh oh!

mrsimonemms commented Dec 15, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corneliusludmann commented Dec 15, 2021

Uh oh!

mrsimonemms commented Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corneliusludmann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

corneliusludmann Dec 15, 2021

Choose a reason for hiding this comment

Uh oh!

mrsimonemms Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corneliusludmann Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

mrsimonemms Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

mrsimonemms commented Dec 15, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrsimonemms commented Dec 20, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrsimonemms commented Dec 20, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

csweichel commented Dec 20, 2021

Uh oh!

meysholdt commented Dec 23, 2021

Uh oh!

mrsimonemms commented Dec 23, 2021

Uh oh!

stale bot commented Jan 2, 2022

Uh oh!

mrsimonemms commented Jan 4, 2022

Uh oh!

csweichel commented Jan 4, 2022

Uh oh!

roboquat commented Jan 4, 2022

Uh oh!

roboquat commented Jan 4, 2022

Uh oh!

Uh oh!

mrsimonemms commented Dec 9, 2021 •

edited

Loading

codecov bot commented Dec 9, 2021 •

edited

Loading

mrsimonemms commented Dec 10, 2021 •

edited by werft-gitpod-dev-com bot

Loading

mrsimonemms commented Dec 15, 2021 •

edited by werft-gitpod-dev-com bot

Loading

mrsimonemms commented Dec 15, 2021 •

edited

Loading

mrsimonemms Dec 15, 2021 •

edited

Loading

mrsimonemms commented Dec 15, 2021 •

edited by werft-gitpod-dev-com bot

Loading

mrsimonemms commented Dec 20, 2021 •

edited by werft-gitpod-dev-com bot

Loading

mrsimonemms commented Dec 20, 2021 •

edited by werft-gitpod-dev-com bot

Loading