[Umbrella] Implement improvements to protect CI and releases from changes to release tooling

[justaugustus]

(Also sent to k-dev, SIG Release MLs, and SIG Testing: https://groups.google.com/d/topic/kubernetes-dev/OuLMzqZkdtw/discussion)

Following an attempt to improve the semantics of the release tooling via shellcheck (#726), we found that we were unable to stage releases.

Multiple fixes were merged in an attempt to bring us to a usable state.

An unintended and unexpected side effect of this was a cascading failure of multiple release-blocking jobs. A few for example:

• Multiple GCE test suites in master-blocking are failing kubernetes#79652
• [Failing Test] Uses kubeadm/kinder to run kubeadm-e2e and e2e tests  kubernetes#79668
• [Failing Test] Uses kubeadm/kinder to run kubeadm-e2e and e2e tests against a cluster created using kinder's skew-master-on-stable workflow kubernetes#79669

Ultimately, it was decided that the right course of action was to revert back to a known good state in the repo (#814) to stop the bleeding.

This implies that, in our current state, it is inadvisable to make any changes to the tooling in this repo.

As such, I'm advising the following course of action (h/t to @nikhita, @liggitt, and @BenTheElder for being a sounding board):

• (sig-release: Blockade changes to critical k/release tooling test-infra#13328) Add a blockade for files that have the potential to impact releasing and CI signal
  (this will require repo admins to explicitly approve and override the blockade to merge changes to critical tooling)
• (in progress below) Examine and document exactly why these release-blocking jobs failed
  (they are using something in k/release; we need to figure out what those somethings are)
• Tag the repo after executing a successful release of Kubernetes
  (this locks in a known good state of k/release that doesn't need to be master)
• (sig-release: Add ci-kubernetes-build-canary to canary release tooling test-infra#13340) Setup a periodic/presubmit job that can emulate one of the existing jobs that broke recently
• Refactor release tooling/jobs that depend on tooling to accept pulling a tag of k/release instead of master

At this point, we will have gotten to a place where we can safely make changes to k/release without impacting CI. We will then:

• Write tests around the specific pieces of the tooling that caused job failure (maybe https://github.com/sstephenson/bats ?)

For longer term goals, we should seek to:

• Write go tooling (and tests!) to replace the shell libraries (lib/{common,gitlib,releaselib}) and call these new tools in the existing release tooling
  (this allows us to get some immediate benefit of a more robust language w/o having to completely refactor)
• Full refactor of existing tools (shell --> go)

(Some historical references: kubernetes/kubernetes#28922, kubernetes/kubernetes#16529, kubernetes/kubernetes#15560, kubernetes/kubernetes#8686)

Please take this an initial assessment of the situation and feel free to provide feedback. :)

/assign
/milestone v1.16
/area release-eng
/sig release
/kind bug
/priority critical-urgent

cc @kubernetes/sig-release-admins @kubernetes/release-engineering @dims @neolit123 @pswica

[hoegaarden]

FWIW, what you can do today is set RELEASE_TOOL_REPO and/or RELEASE_TOOL_BRANCH when submitting the cloud builder job via gcbmgr, e.g.:

RELEASE_TOOL_REPO='https://github.com/other-org/other-release-repo'  RELEASE_TOOL_BRANCH='feature-foo' \
  ./gcbmgr stage release-1.12 --official --build-at-head

This & mock releases should be safe enough and a way forward to start backfilling tests, refactoring, ...

Just to be clear: I am not advocating to keep using this and a mock stage & release as "the test". We need way faster feedback. But we can start with that.

[neolit123]

Examine and document exactly why these release-blocking jobs failed
(they are using something in k/release; we need to figure out what those somethings are)

on the side of kubeadm release-master-informing jobs the failures stared happening because the kubeadm jobs could not find the CI artifacts they needs such as:

https://storage.googleapis.com/kubernetes-release-dev/ci/<CI-TAG>/bin/linux/amd64/kubelet
https://storage.googleapis.com/kubernetes-release-dev/ci/<CI-TAG>/bin/linux/amd64/kubeadm
...

examining the bucket:

gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/
gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/extra/
# ^ the bin directory is missing

as @dims outlined, the prow job ci-kubernetes-build even if passing/green, had errors:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-build/1146386381002510337/

W0703 12:54:56.715] tar: Substituting `.' for empty member name
W0703 12:54:56.715] tar: : Cannot stat: No such file or directory
W0703 12:54:56.715] tar: Exiting with failure status due to previous errors

this is the first kubeadm job failure we saw:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-master-on-stable/1145990882009288707

i couldn't not map the time to a problematic commit in k/release.

[neolit123]

Setup a presubmit job that can emulate one of the existing jobs that broke recently

i think this is a great idea and such an e2e can compensate for the lack of unit tests.

the problems i saw - e.g.

gsutil ls gs://kubernetes-release-dev/ci/v1.16.0-alpha.0.1789+a807cb625b08c7/

indicate issues in the staging process.
so such an e2e has to stage somewhere and verify if all the artifacts are up.

[justaugustus]

@neolit123 -- that's super useful; thanks for the data!

FWIW, what you can do today is set RELEASE_TOOL_REPO and/or RELEASE_TOOL_BRANCH when submitting the cloud builder job via gcbmgr, e.g.:

@hoegaarden -- I didn't know you could do that. Do we have it documented somewhere?

[BenTheElder]

There is one such presubmit now (pull-release-cluster-up) but that doesn't cover everything used by EG the kubeadm job.

[dims]

@hoegaarden @justaugustus that gcbmgr script needs access to release-test-dev bucket too

dims@rex:~/go/src/k8s.io/release$ RELEASE_TOOL_REPO='https://github.com/kubernetes/release' RELEASE_TOOL_BRANCH=master ./gcbmgr stage master --build-at-head
gcbmgr: BEGIN main on rex Fri Jul  5 02:17:21 UTC 2019

Checking required system packages: OK
Checking/setting cloud tools: OK

FAILED: Job was not submitted.  Details:
ERROR: (gcloud.builds.submit) User [davanum@gmail.com] does not have permission to access project [kubernetes-release-test] (or it may not exist): Caller
does not have required permission to use project project:kubernetes-release-test. Grant the caller the Owner or Editor role, or a custom role with the
serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=project:kubernetes-release-test
and then retry (propagation of new permission may take a few minutes).  - '@type': type.googleapis.com/google.rpc.Help
  links: - description: Google developers console
    url: https://console.developers.google.com
- '@type': type.googleapis.com/google.rpc.Help
  links: - description: Google developer console IAM admin
    url: https://console.developers.google.com/iam-admin/iam/project?project=project:kubernetes-release-test

[justaugustus]

@dims -- Could you clarify what you mean by access to release-test-dev?

That snippet only shows that you don't have access to the kubernetes-release-test project on GCP:

augustus@auggievmw01:~/go/src/k8s.io/release$ RELEASE_TOOL_REPO='https://github.com/kubernetes/release' RELEASE_TOOL_BRANCH=master ./gcbmgr stage master --build-at-head
gcbmgr: BEGIN main on auggievmw01 Thu Jul  4 22:45:51 EDT 2019

Checking required system packages: OK
Checking /home/augustus/go/src/k8s.io/release state: OK
Checking/setting cloud tools: OK

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
STAGE 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c submitted successfully.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

To view last build:
$ gcbmgr tail

To view this specific build:
$ gcbmgr tail 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c
-OR-
$ /usr/bin/gcloud --project kubernetes-release-test builds log --stream 92f24742-f1fe-4ca8-b33d-ffbca2e1c43c
-OR-
https://console.cloud.google.com/gcr/builds/92f24742-f1fe-4ca8-b33d-ffbca2e1c43c?project=648026197307

gcbmgr: DONE main on auggievmw01 Thu Jul  4 22:45:55 EDT 2019 in 4s

[hoegaarden]

@hoegaarden -- I didn't know you could do that. Do we have it documented somewhere?

It is not documented, it came in somewhat recently with 636269f.