Clarification on new SIG Release GCP projects and image promotion

[justaugustus]

As we work through building out the build/stage/release process on K8s Infra (as opposed to Google Infra), a few things come to mind:

New SIG Release GCP projects need to be consistently named

The current projects are:

• k8s-staging-release-test - Staging
• k8s-release-test-prod - Fake prod

We need to create consistent names for these projects.
Suggesting:

• k8s-release-staging - Staging
• k8s-release - Prod

We also need a project for CI assets, which should likely have shared ownership between SIG Release and Testing.

Additionally, all Release Engineering projects should be considered "production/near-prod" and not be subject to the same retention policies and IAM configurations as the other K8s Infra GCP projects.

Understanding image promotion for Release Engineering

We're starting to create new images which are currently getting pushed to the k8s-release-test project.
Those are wired to push on post-submit within k/release.

This means they're landing in the new infra, but there are instances where we would want to promote them in a prod location e.g., k8s-cloud-builder, which is the underlying image using to stage/release Kubernetes.

I'd also like us to get to the point where we can do image promotion of some core images, like:

• debian-*
• kube-cross

In order to do that, we need to get those images pushed to some K8s Infra staging area (k8s-release-staging).

I'm not sure if the ability exists today to promote from a K8s Infra project into Google Infra project, but that could be a stopgap.

Let me know what you think!

From @kubernetes/release-engineering:
/assign

WG K8s Infra:
/assign @dims @cblecker @listx @thockin

ref: #852

/sig release
/area release-eng
/wg k8s-infra
/priority critical-urgent
/kind feature

[dims]

cc @spiffxp

[thockin]

One point at a time:

The current projects are:

k8s-staging-release-test - Staging
k8s-release-test-prod - Fake prod

We need to create consistent names for these projects.

They are consistent - with the norms k8s-infra-wg has established for staging and prod-archetype projects. What do you mean by consistent? Consistent with what?

Suggesting:

k8s-release-staging - Staging
k8s-release - Prod

I don't object to those names except that we have ample tooling that assumes the existing format. To change the names as you suggest will require reworking the tooling. Frankly, this needs to happen anyway, but name changes were not in scope.

We also need a project for CI assets, which should likely have shared ownership between SIG
Release and Testing.

Define assets?

Additionally, all Release Engineering projects should be considered "production/near-prod" and not
be subject to the same retention policies and IAM configurations as the other K8s Infra GCP
projects.

Explain? Are you concerned about holding on to things too long or not long enough?

We're starting to create new images which are currently getting pushed to the k8s-release-test
project.
Those are wired to push on post-submit within k/release.

Note that release-test was intended for you to figure out the tooling and process and structure AND THEN GO AWAY once we have real names.

This means they're landing in the new infra, but there are instances where we would want to
promote them in a prod location e.g., k8s-cloud-builder, which is the underlying image using to
stage/release Kubernetes.

I'm unclear on what "them" is.

I'd also like us to get to the point where we can do image promotion of some core images, like:

debian-*
kube-cross
In order to do that, we need to get those images pushed to some K8s Infra staging area (k8s-
release-staging).

This is what the existing staging structure is for - for example k8s-staging-build-image is supposed to house kube-cross.

I'm not sure if the ability exists today to promote from a K8s Infra project into Google Infra project,
but that could be a stopgap.

I am missing the point here, I think.

[justaugustus]

Suggesting:
k8s-release-staging - Staging
k8s-release - Prod

I don't object to those names except that we have ample tooling that assumes the existing format. To change the names as you suggest will require reworking the tooling. Frankly, this needs to happen anyway, but name changes were not in scope.

To your point in kubernetes/k8s.io#434 (review):

Mostly I question the overall factoring - these don't feel like "staging" and "prod" projects (in the sense that we have scripts that exist) any more. Maybe they need to be totally distinct (removed from other scripts) and even renamed?

The SIG Release projects are more prod-like and I would expect to diverge somewhat from the way we configure staging ones e.g., addition of IAM roles for Release Managers, service accounts to lift/shift artifacts like release tarballs into prod.

We also need a project for CI assets, which should likely have shared ownership between SIG
Release and Testing.

Define assets?

For CI: build artifacts (tarballs, SHANNNSUMS, version markers) that are dropped in https://gcsweb.k8s.io/gcs/kubernetes-release-dev/ci/ as a result of running push-build.sh from within the ci-kubernetes-build* prow jobs.

Do we have a project for that purpose yet?

For staging --> prod, we have this somewhat enumerated: https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md

Additionally, all Release Engineering projects should be considered "production/near-prod" and not
be subject to the same retention policies and IAM configurations as the other K8s Infra GCP
projects.

Explain? Are you concerned about holding on to things too long or not long enough?

Not long enough? Our fake prod would need an indefinite policy on the bucket it's landing things in.

We're starting to create new images which are currently getting pushed to the k8s-release-test
project.
Those are wired to push on post-submit within k/release.

Note that release-test was intended for you to figure out the tooling and process and structure AND THEN GO AWAY once we have real names.

To clarify, I had opened this issue because we have made progress and I wanted a gut check into what's been done so far to make sure we're not diverging from anything K8s Infra had in mind/have the order of operations right.

I tagged K8s Infra on the umbrella issue for that work and would love a review.

This means they're landing in the new infra, but there are instances where we would want to
promote them in a prod location e.g., k8s-cloud-builder, which is the underlying image using to
stage/release Kubernetes.

I'm unclear on what "them" is.

In this case, container images produced from running anago.

Today, the process of building those images is entangled within anago, meaning you have to run anago against the project you intend the images to be pushed on.

I'd like to get us to the point where we're building all artifacts (GCS + GCR) against staging and then promoting into prod, but that is likely to be further entangled with our work to refactor the tools in this repo to Go.

I'd also like us to get to the point where we can do image promotion of some core images, like:
debian-*
kube-cross
In order to do that, we need to get those images pushed to some K8s Infra staging area (k8s-
release-staging).

This is what the existing staging structure is for - for example k8s-staging-build-image is supposed to house kube-cross.

It kind of is and isn't.
Maybe kube-cross is a bad example, consider kube-apiserver.

That's an image that would have to land in new prod/new k8s.gcr.io and is currently owned by SIG Release by way of anago.

I'm not sure if the ability exists today to promote from a K8s Infra project into Google Infra project,
but that could be a stopgap.

I am missing the point here, I think.

If I was to set up image promotion for something like the kube-apiserver image, would I be able to stage something in new K8s Infra and promote it across to old prod k8s.gcr.io?

I ask because I have no idea about the plans to backfill images from the old registries to the new.

[thockin]

@justaugustus and I spoke today.

1. Anything that produces container images can start moving to the new promoter process. Granularity TBD.

2. For each image under k8s-staging-foo, we promote to k8s.gcr.io/foo/... We discussed maybe a special-case for release-team to promote to the "root dir" for release artifacts. I am not against it, per se.

3. GCS promotion is hypothetical. In the interim, we can grant a select set of users the abilityt o manually promote.

4. TBD whether promoting to gs://k8s-artifacts-prod is sufficient or whether we want a more vanity name.