Ensure specific golang version everywhere in CI based on Makefile #5095

medyagh · 2019-08-15T20:47:31Z

Single source of truth for go version based on Makefile
Installs right version golang (if not already) on jenkins build agent
Bump the golang version to 1.12.8
Also addresses multiple golang CVEs closes Due to multiple CVEs and fds leak, need to upgrade golang to 1.12.8+ #5087 also closes CI: install specific version of golang for cross build #4895

Before this PR run on the jenkins box:

jenkins@jenkins:~$ go version
go version go1.12.5 linux/

on the first run of this PR if wrong verison of golang is installed, it will warn and then install the right version:

when the golang version is not same as Makefile jenkins will warn and then install the right version

14:36:06 WARNING: expected go version to be 1.12.8 but got 1.12.5

here is the output:

15:01:27 ++ install_golang 1.12.8 /usr/local
15:01:27 ++ echo 'Installing golang version: 1.12.8 on /usr/local'
15:01:27 Installing golang version: 1.12.8 on /usr/local
15:01:27 ++ pushd /tmp
15:01:27 ++ sudo curl -qL -O https://storage.googleapis.com/golang/go1.12.8.linux-amd64.tar.gz
15:01:27   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
15:01:27                                  Dload  Upload   Total   Spent    Left  Speed
15:01:27 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 88  122M   88  107M    0     0   190M      0 --:--:-- --:--:-- --:--:--  190M
100  122M  100  122M    0     0   196M      0 --:--:-- --:--:-- --:--:--  196M
15:01:28 ++ sudo tar xfa go1.12.8.linux-amd64.tar.gz
15:01:32 ++ sudo rm -rf /usr/local/go
15:01:32 ++ sudo mv go /usr/local/
15:01:32 +++ whoami
15:01:32 ++ sudo chown -R jenkins: /usr/local/go
15:01:32 ++ popd
15:01:32 ++ return

after the first run I can see in the jenkins box ssh:

jenkins@jenkins:~$ go version
go version go1.12.8 linux/amd64
jenkins@jenkins:~$ which go
/usr/local/go/bin/go
jenkins@jenkins:~$ ls -lah /usr/local/go/bin/go
-rwxr-xr-x 1 jenkins jenkins 14M Aug 13 09:59 /usr/local/go/bin/go
jenkins@jenkins:~$ 


$ ls -lah /usr/local/go/bin/go
-rwxr-xr-x 1 jenkins jenkins 14M Aug 13 09:59 /usr/local/go/bin/go
jenkins@jenkins:~$ 
jenkins@jenkins:~$ 
jenkins@jenkins:~$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/var/lib/jenkins/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/var/lib/jenkins/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build458013956=/tmp/go-build -gno-rec
ord-gcc-switches"

and also when the right version is already installed it won't re-install but lets you know it is good !

15:13:27 ++ check_and_install_golang
15:13:27 ++ echo 'go version on the host looks good : 1.12.8 '
15:13:27 go version on the host looks good : 1.12.8

k8s-ci-robot · 2019-08-15T20:47:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

josedonizetti

it updated my golang locally from 1.12.5 to 1.12.8 :)

hack/jenkins/release_build_and_upload.sh

hack/jenkins/installers/check_install_golang.sh

afbjorklund

Version number seems duplicated ?

Probably should be 1.12.9 now, too

Makefile

afbjorklund · 2019-08-16T15:59:38Z

Makefile

@@ -22,6 +22,9 @@ ISO_VERSION ?= v$(VERSION_MAJOR).$(VERSION_MINOR).0
 VERSION ?= v$(VERSION_MAJOR).$(VERSION_MINOR).$(VERSION_BUILD)
 DEB_VERSION ?= $(VERSION_MAJOR).$(VERSION_MINOR).$(VERSION_BUILD)
 RPM_VERSION ?= $(VERSION_MAJOR).$(VERSION_MINOR).$(VERSION_BUILD)
+# used by hack/jenkins/release_build_and_upload.sh and KVM_BUILD_IMAGE, see also BUILD_IMAGE bellow
+GO_VERSION ?= 1.12.8


Make this 1.12.9

I was thinking to make it 1.12.9 but the kubec-cross image is not available in that version yet ! I rather we build everything with same version.

Makes perfect sense, and it is version 1.12.8 that has the needed security fix anyway

I saw the commit updating the version, and naively thought the image was built too...

kubernetes/kubernetes@54c6744

alternatively we can make everything build in host golang and not use any docker at all.

hack/jenkins/installers/check_install_golang.sh

hack/jenkins/minikube_cross_build_and_upload.sh

hack/jenkins/release_build_and_upload.sh

tstromberg

Looks wonderful. Thank you for doing this!

hack/jenkins/installers/check_install_golang.sh

tstromberg · 2019-08-16T15:50:41Z

hack/jenkins/installers/check_install_golang.sh

+function install_golang {
+    echo "Installing golang version: $1 on $2"  
+    pushd /tmp >/dev/null
+    sudo curl -qL -O "https://storage.googleapis.com/golang/go${1}.linux-amd64.tar.gz" &&


Could this be installed to the users home directory instead? It seems awkward to have an automated system upgrading system binaries as a non-system user, only to chown it back to themselves.

Interestingly, nothing in the home directory can be assumed to be in the PATH everywhere

Might as well use the standard location ? And have it owned by root, as per documentation

hack/jenkins/installers/check_install_golang.sh

tstromberg · 2019-08-16T15:57:41Z

hack/jenkins/installers/check_install_golang.sh

+    sudo curl -qL -O "https://storage.googleapis.com/golang/go${1}.linux-amd64.tar.gz" &&
+    sudo tar xfa go${1}.linux-amd64.tar.gz &&
+    sudo rm -rf "${2}/go" &&
+    sudo mv go "${2}/" && sudo chown -R $(whoami): ${2}/go


is the chown step here necessary?

Only because doing tar as root ?

But it also seems wrong, files under /usr/local should be owned by root (this is not brew)

the reason I did this is because, the previous version of golang was installed by a user other than "jenkins" . the /usr/local/go was owned by "tstromberg"

so when jenkins was trying to delete that folder and re-install it again, it had permission error.

so I deleted and downloded with sudo and then later changed back to jenkins !

this was the only automated way I could find!

let me know what you guys think ! I personally think this was best way given the previous manual installation situation.

I could have provided a comment on why I was doing sudo tar

tstromberg · 2019-08-16T16:00:00Z

hack/jenkins/installers/check_install_golang.sh

+# install_golang takes two parameters version and path to install.
+function install_golang {
+    echo "Installing golang version: $1 on $2"  
+    pushd /tmp >/dev/null


mktemp -d is preferable.

reminds me of this poem http://blog.medyagh.com/2017/07/thats-more-preferable.html

hack/jenkins/minikube_cross_build_and_upload.sh

hack/jenkins/installers/check_install_golang.sh

hack/jenkins/release_build_and_upload.sh

Ensure specific golang version is installed on build agents

dba5520

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 15, 2019

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 15, 2019

k8s-ci-robot requested review from RA489 and tstromberg August 15, 2019 20:47

medyagh changed the title ~~Ensure specific golang version is installed on build agents~~ Ensure specific golang version everywhere based on Makefile Aug 15, 2019

medyagh changed the title ~~Ensure specific golang version everywhere based on Makefile~~ Ensure specific golang version everywhere in CI based on Makefile Aug 15, 2019

medyagh added 2 commits August 15, 2019 15:07

add go install to both release and cross build

6ace05d

added chown

f63208b

medyagh force-pushed the update_golang_ver branch from 41ab354 to f63208b Compare August 15, 2019 22:07

Fix string compare whitespace

b6c8346

josedonizetti approved these changes Aug 15, 2019

View reviewed changes

josedonizetti reviewed Aug 15, 2019

View reviewed changes

hack/jenkins/release_build_and_upload.sh Outdated Show resolved Hide resolved

medyagh and others added 2 commits August 15, 2019 15:44

Added comment

51dcf92

Update release_build_and_upload.sh

c02ea8e

josedonizetti reviewed Aug 15, 2019

View reviewed changes

hack/jenkins/installers/check_install_golang.sh Show resolved Hide resolved

medyagh added 2 commits August 15, 2019 17:00

update boilerplate year

be0dc2d

Update release_build_and_upload.sh

8b7b06f

afbjorklund reviewed Aug 16, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

afbjorklund requested changes Aug 16, 2019

View reviewed changes

tstromberg suggested changes Aug 16, 2019

View reviewed changes

medyagh added 2 commits August 16, 2019 10:21

remove default, fix indent, set pipefail

fda017d

Fix boilerplate

3599e8f

medyagh merged commit 434ca47 into kubernetes:master Aug 18, 2019

medyagh deleted the update_golang_ver branch August 18, 2019 02:46

warmchang mentioned this pull request Aug 19, 2019

bump to go1.12.9 #5128

Merged

Ensure specific golang version everywhere in CI based on Makefile #5095

Ensure specific golang version everywhere in CI based on Makefile #5095

Uh oh!

Conversation

medyagh commented Aug 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before this PR run on the jenkins box:

Uh oh!

k8s-ci-robot commented Aug 15, 2019

Uh oh!

josedonizetti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

afbjorklund left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

afbjorklund Aug 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tstromberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

medyagh commented Aug 15, 2019 •

edited

Loading

afbjorklund Aug 16, 2019 •

edited

Loading