kubepkg tests fail

[hoegaarden]

Running go test ./... fails for the package k8s.io/release/pkg/kubepkg over here.

time="2020-01-08T14:22:29Z" level=info msg="Setting version for  package..."
time="2020-01-08T14:22:29Z" level=info msg="Using Kubernetes version for  package"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for  package..."
time="2020-01-08T14:22:29Z" level=info msg="Using Kubernetes version for  package"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for kubernetes-cni package..."
time="2020-01-08T14:22:29Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:29Z" level=info msg="Setting CNI version to 0.8.3"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for cri-tools package..."
time="2020-01-08T14:22:29Z" level=info msg="Getting CRI version..."
--- FAIL: TestGetPackageVersionSuccess (0.66s)
    kubepkg_test.go:68: did not expect an error: GET https://github.com/api/repos/kubernetes-sigs/cri-tools/releases: 403 API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 22m17s]
time="2020-01-08T14:22:30Z" level=info msg="Using Kubernetes version (1.17.0) for  package"
time="2020-01-08T14:22:31Z" level=info msg="Retrieved Kubernetes version: 1.14.10"
time="2020-01-08T14:22:31Z" level=info msg="Retrieved Kubernetes version: 1.14.11-beta.1"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Kubernetes version earlier than 1.17 must use CNI version <= 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.8.3"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Getting CRI version..."
--- FAIL: TestGetCRIToolsVersionSuccess (0.14s)
    kubepkg_test.go:252: did not expect an error: GET https://github.com/api/repos/kubernetes-sigs/cri-tools/releases: 403 API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 22m16s]
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of kubelet check"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of release check"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of testing check"
time="2020-01-08T14:22:31Z" level=info msg="Adding 'fakearch' (type: string) to not supported"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of amd64 check"
time="2020-01-08T14:22:31Z" level=info msg="The following options are not supported: [fakearch]"
FAIL
FAIL    k8s.io/release/pkg/kubepkg      2.197s

This indicates IMHO that we should take more care about our tests and split them into different types (static, unit, integration, e2e).

[hoegaarden]

/kind failing-test
/priority important-soon
/area release-eng

[justaugustus]

@hoegaarden -- Here's what I got, FWIW:

$ go test ./...
?   	k8s.io/release/cmd/blocking-testgrid-tests	[no test files]
?   	k8s.io/release/cmd/krel	[no test files]
?   	k8s.io/release/cmd/krel/cmd	[no test files]
?   	k8s.io/release/cmd/kubepkg	[no test files]
?   	k8s.io/release/cmd/kubepkg/cmd	[no test files]
?   	k8s.io/release/cmd/release-notes	[no test files]
ok  	k8s.io/release/pkg/command	0.020s
ok  	k8s.io/release/pkg/git	0.990s
ok  	k8s.io/release/pkg/kubepkg	1.754s
ok  	k8s.io/release/pkg/notes	0.356s
ok  	k8s.io/release/pkg/notes/internal	0.004s
?   	k8s.io/release/pkg/notes/notesfakes	[no test files]
ok  	k8s.io/release/pkg/release	1.011s
ok  	k8s.io/release/pkg/util	1.003s

How would you propose organizing the tests?

[hoegaarden]

How would you propose organizing the tests?

Not sure :) I have seen different approaches to separate different test types. Wanted to bring them up and compare them and start a discussion. Stay tuned.

[justaugustus]

SGTM! I've mentioned on a few calls that testing methodologies is not a strong suit of mine, so happy to have you to steer the convo.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[saschagrunert]

/remove-lifecycle rotten

[justaugustus]

/assign @saschagrunert
/milestone v1.19