[Failing Test] post-release-push-image-kube-cross (post-release-push-image-kube-cross)

[droslean]

Which jobs are failing:
post-release-push-image-kube-cross (post-release-push-image-kube-cross)

Testgrid link:
https://testgrid.k8s.io/sig-release-releng-informing#post-release-push-image-kube-cross

Reason for failure:

 2020/03/20 08:17:43 Running build jobs...
2020/03/20 08:17:43 No variants.yaml, starting single build job...
Created [https://cloudbuild.googleapis.com/v1/projects/k8s-staging-build-image/builds/a7330d25-3678-40f5-a069-4edcde4a9933].
Logs are available at [https://console.cloud.google.com/cloud-build/builds/a7330d25-3678-40f5-a069-4edcde4a9933?project=960211007710].
----------------------------- REMOTE BUILD OUTPUT ------------------------------
ERROR: (gcloud.builds.submit) build a7330d25-3678-40f5-a069-4edcde4a9933 completed with status "FAILURE"
starting build "a7330d25-3678-40f5-a069-4edcde4a9933"
FETCHSOURCE
Fetching storage object: gs://k8s-staging-build-image-gcb/source/1584692264.34-c9d74aed627e48e1baaa718c563e079d.tgz#1584692264511810
--------------------------------------------------------------------------------
2020/03/20 08:27:36 Failed to run some build jobs: [error running [gcloud builds submit --verbosity info --config /home/prow/go/src/github.com/kubernetes/release/images/build/cross/cloudbuild.yaml --substitutions _GIT_TAG=v20200320-v0.2.6-28-g7e2a7ef --project k8s-staging-build-image --gcs-log-dir gs://k8s-staging-build-image-gcb/logs --gcs-source-staging-dir gs://k8s-staging-build-image-gcb/source gs://k8s-staging-build-image-gcb/source/73101c49-a0ee-415c-a640-463cfaa8f97c.tgz]: exit status 1] 

Anything else we need to know:
/cc @kubernetes/release-managers
/sig release
/priority important-soon
/milestone v1.19

[justaugustus]

I'm aware of this one and already working on a fix.
/assign

[justaugustus]

Just to explain the failures:

1

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-release-push-image-kube-cross/1240797966474678274 --> https://console.cloud.google.com/cloud-build/builds/4a51bc16-2bd3-4bc9-b9b6-ec4f7be6ef50?project=k8s-staging-build-image

fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
docker build \
	-t gcr.io/k8s-staging-build-image/kube-cross-amd64:kubernetes- \
	-t gcr.io/k8s-staging-build-image/kube-cross-amd64:v1.13.9-1 \
	-t us.gcr.io/k8s-artifacts-prod/build-image/kube-cross-amd64:v1.13.9-1 \
	.
Sending build context to Docker daemon  9.728kB

Step 1/12 : FROM golang:1.13.9
no matching manifest for unknown in the manifest list entries
make: *** [Makefile:33: build] Error 1
1.13.9: Pulling from library/golang
ERROR
ERROR: build step 0 "gcr.io/k8s-testimages/gcb-docker-gcloud:v20191019-6567e5c" failed: step exited with non-zero status: 2

This failed because the upstream golang:1.13.9 image didn't exist yet.

2

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-release-push-image-kube-cross/1240799857229172736

Running...
2020/03/20 00:38:40 Build directory: images/build/cross
2020/03/20 00:38:40 Config directory: /home/prow/go/src/github.com/kubernetes/release/images/build/cross
2020/03/20 00:38:40 cd-ing to build directory: images/build/cross
2020/03/20 00:38:40 Creating source tarball at /tmp/215057757...
2020/03/20 00:38:40 Uploading /tmp/215057757 to gs://k8s-staging-build-image-gcb/source/cdc19149-5d63-447c-8b43-59557b4ded9f.tgz...
ERROR: (gsutil) timed out
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.
2020/03/20 00:38:48 Failed to run some build jobs: [failed to upload source: failed to upload files: exit status 1]

Appears to be an intermittent network failure.

3

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-release-push-image-kube-cross/1240915379610456064 --> https://console.cloud.google.com/cloud-build/builds/a7330d25-3678-40f5-a069-4edcde4a9933?project=k8s-staging-build-image

docker manifest create --amend gcr.io/k8s-staging-build-image/kube-cross:v1.13.9-1 \
	gcr.io/k8s-staging-build-image/kube-cross-amd64:v1.13.9-1
Created manifest list gcr.io/k8s-staging-build-image/kube-cross:v1.13.9-1
docker manifest annotate gcr.io/k8s-staging-build-image/kube-cross:v1.13.9-1 \
	gcr.io/k8s-staging-build-image/kube-cross-amd64:v1.13.9-1 \
	--arch amd64
docker manifest push -p gcr.io/k8s-staging-build-image/kube-cross:v1.13.9-1
Pushed ref gcr.io/k8s-staging-build-image/kube-cross@sha256:60e119bc59d8e6c77ddf159af0a682d0634249e3baca69d5be2972d00942ff51 with digest: sha256:60e119bc59d8e6c77ddf159af0a682d0634249e3baca69d5be2972d00942ff51
sha256:3b963befb3b660a859f51bc8212ff72a9ded9ff1495829c4d9cc4bbf676c80d8
ERROR
ERROR: failed to find one or more images after execution of build steps: ["gcr.io/k8s-staging-build-image/kube-cross:kubernetes-v20200320-v0.2.6-28-g7e2a7ef"]

Misconfiguration of the cloudbuild.yaml.
We're attempting to validate that gcr.io/k8s-staging-build-image/kube-cross:kubernetes-v20200320-v0.2.6-28-g7e2a7ef exists, but we actually pushed gcr.io/k8s-staging-build-image/kube-cross-amd64:kubernetes-.

If we look earlier in that log, we can see:

Successfully tagged gcr.io/k8s-staging-build-image/kube-cross-amd64:kubernetes-
Successfully tagged gcr.io/k8s-staging-build-image/kube-cross-amd64:v1.13.9-1
Successfully tagged us.gcr.io/k8s-artifacts-prod/build-image/kube-cross-amd64:v1.13.9-1
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
docker push gcr.io/k8s-staging-build-image/kube-cross-amd64:kubernetes-

As we're using a .tgz of images/build/cross as the source for the GCB run, it doesn't have a .git and therefore fails on the git describe command in the Makefile:

release/images/build/cross/Makefile

Line 21 in 3e3e14a

TAG?=kubernetes-$(shell git describe --tags --match='v*' --abbrev=14)

Fixed in #1202.

[justaugustus]

Moving this to k/release...

[justaugustus]

/area release-eng
/milestone v1.18

[justaugustus]

Closed via #1202.
The latest image push job has passed now that that PR has merged: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-release-push-image-kube-cross/1241732100298641408

/close