Determine image building process for release tools

[justaugustus]

Something that has been on my mind and I'm realizing now that we may not have had an issue for it before.

From kubernetes/test-infra#13890...
@tpepper:

I think we should be moving more towards distroless plus specific, explicit dependencies. In the near term changes over the past months have driven the tooling to a place where it tends to expect a general and recent-ish distro base. Picking a base Ubuntu or debian might more imply "this is a catch all linux dev/test env" versus using kube-cross?

Either way, merging an update to this image to ungate things should also come with an issue to clean up, minimize, make explicit the image base and add-on content requirements and properly lifecycle it moving forward.

@hoegaarden:

I don't think distroless is the right choice for most of the things we want to run as part of the build & test. I agree that one should think about using distroless for running things. But for building and testing, esp. for the sh tests, we literally need a distro or tools you'd expect to be part of a distro.

We should strive for lightweight images that have only the dependencies we require to build, test, utilize release tooling.

Adding my opinion to the discussion, I also don't think distroless is the right choice at the moment, as some of our process requires package-building tools for debs and rpms.

Suggesting:

• lightweight Debian/Ubuntu image
• git
• curl
• jq
• python
• go
• bazel
• Google Cloud utilities
• rpm
• docker(?)

Additionally, we need an automated process for bumping these images when dependencies are updated.
k/test-infra has some process for this, though I don't know what it is.

Should we host there instead?
We also need to figure out which image repository to host in and investigate leveraging the container-image-promoter.

cc: @kubernetes/sig-release-admins @kubernetes/release-engineering @hoegaarden @spiffxp @BenTheElder
/help
/area release-eng
/kind feature
/milestone v1.16
/priority important-longterm

[justaugustus]

(This is a good one for a Release Manager Associate to pick up.)

[justaugustus]

@hoegaarden has some initial work here that I still need to review: #825

[hoegaarden]

@hoegaarden has some initial work here that I still need to review: #825

I just took what was already there and made it hopefully easier to discover, nicer to use, and better documented.

Should we host there instead?

As far as I can see for some of those they also use GCB to bake the images and push them into a registry. I have no idea how the promotion of their images out of the staging registry works, what criteria they use to consider an image as good and ready for promotion ...

In any case, I think none of this should block us in the short term. We should be able to add new images (e.g. for testing, ...) or update images if we need to. If we/someone finds the time to figure out how test-infra does this and their flow works for us, I think it is rel. easy for us to move to that.

[alexeldeib]

This sounds right up my alley, and I'm an associate looking for places to jump in! 🎉 I also have some very outstanding test-infra PRs to add kubebuilder/controller-runtime to Prow -- which happened to require a new image with various deps. I can circle back there to close the loop and see if maybe the same solution can apply to both.

If i'm hearing this correctly, the immediate solution is the two parts you mentioned:

• Create a Docker image from a reasonable debian/ubuntu base, with the necessary deps for release tooling (above) included.
• Formalize a process for the hosting of the image, updates, etc.

For the first point, I'm wondering how to go about validation? Create a new prow job using the images to test against/similar?

[hoegaarden]

If i'm hearing this correctly, the immediate solution is the two parts you mentioned:

Create a Docker image from a reasonable debian/ubuntu base, with the necessary deps for release tooling (above) included.

I don't think that is needed. For the release itself, we have https://github.com/kubernetes/release/tree/master/images (specifically https://github.com/kubernetes/release/tree/master/images/k8s-cloud-builder).
For the images we use for our (unit) tests, we for now can steal use other images like the kubekins-e2e.

Formalize a process for the hosting of the image, updates, etc.
For the first point, I'm wondering how to go about validation? Create a new prow job using the images to test against/similar?

Right now I have no idea how images are generally build, maintained, promoted, ... and I have the feeling this is not unified across the k-org. I "just" like to have process to introduce, update, promote, ... images and understand that process (documented). Right now I don't really have that for the k/release images, which is not good.

[justaugustus]

For the images we use for our (unit) tests, we for now can steal use other images like the kubekins-e2e.

@hoegaarden -- Let's not conflate the immediate fix (using kubekins-e2e) with the long-term goal (this issue). We still want this.

---

@alexeldeib -- Feel free to proceed with this.

For the first point, I'm wondering how to go about validation? Create a new prow job using the images to test against/similar?

In terms of validation, as a baseline, the resultant image should be able to successfully run all of the following jobs:

• pull-release-unit, periodic-release-verify-packages-debs, periodic-release-verify-packages-rpms: https://github.com/kubernetes/test-infra/blob/66944a3aa86ac7cbce7cad808306cb4fb0898b02/config/jobs/kubernetes/release/release-config.yaml#L41-L118
• ci-release-build-packages-debs, ci-release-build-packages-rpms: https://github.com/kubernetes/test-infra/blob/66944a3aa86ac7cbce7cad808306cb4fb0898b02/config/jobs/kubernetes/sig-release/kubernetes-builds.yaml#L62-L120

We should be able to use the existing prow jobs to validate the images, without having to create new ones.

Hosting, updating, promotion, and policy can be discussed once we can clear that first hurdle.

As golang is one of our most important dependencies, please use golang:1.12.9-buster as your base.

[hoegaarden]

For the images we use for our (unit) tests, we for now can steal use other images like the kubekins-e2e.

@hoegaarden -- Let's not conflate the immediate fix (using kubekins-e2e) with the long-term goal (this issue). We still want this.

We absolutely want that.

I think that came across differently than intended, what I am trying to say is:

We need to figure out how to handle images in general, how and where to bake them, how to test and how to promote them. It should be a process that allows me (as a member of the release management or release engineering team) to change the image at any time or add a new one, and (given all the tests go green and so forth) that will all just work and the new or updated image will be available for me and the release tooling. That is a process we don't have in place yet or I don't understand it. However, having it in place would add value.

I don't see much value in just swapping out the test images (even if they are lighter or what not) without fixing the above.

I think we agree on that anyway, you and @alexeldeib seem to have a ton of context there (and how test-infra and others handle that) anyway, so ✅!

[BenTheElder]

If you're going to run our scripts or bazel they need a full typical userspace, the standard Ubuntu image is relatively small for that. Everything else in the dependency list is explicitly added to kubekins-e2e which just happens to also have some other junk.

FWIW test-infra is going to need to replace that image with a similar cleaned up one for testing.

[BenTheElder]

As for how test-infra handles it, look at the postsubmits on that repo, you can trace it from there. In short prow postsubmit => GCB matrix via a tool from Katharine. 🙃