Skip to content

Commit ab0cc03

Browse files
committed
Makefile: Reduce flake attempts when running the e2e suite in CI
Update the `e2e-local` Makefile target and reduce the number of flake attempts that will be attempted to a single test failure. This should help weed out any tests that are flake-prone and need to be updated in order to increase signal during individual CI runs, along with any real problematic tests that masking regressions due to re-running any failed test multiple times. Signed-off-by: timflannagan <[email protected]>
1 parent 5812e0b commit ab0cc03

File tree

1 file changed

+5
-2
lines changed

1 file changed

+5
-2
lines changed

Makefile

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ setup-bare: clean e2e.namespace
119119
. ./scripts/package_release.sh 1.0.0 test/e2e/resources test/e2e/e2e-bare-values.yaml
120120
. ./scripts/install_bare.sh $(shell cat ./e2e.namespace) test/e2e/resources
121121

122-
GINKGO_OPTS := -flakeAttempts 3 -randomizeAllSpecs -v --timeout 120m
122+
GINKGO_OPTS := -flakeAttempts 1 -randomizeAllSpecs -v --timeout 120m
123123

124124
# TODO(tflannag): Remove this target entirely and move downstream
125125
e2e:
@@ -138,10 +138,13 @@ e2e:
138138
.PHONY: FORCE
139139
FORCE:
140140

141+
# TODO(tflannag): Temporarily decrement flake attempts to 1 to weed out any flakes
142+
# or problematic tests that are masked by the flake attempts. Determine an appropriate
143+
# value long term that balances out flake tolerance with contributor experience.
141144
# main entry point for running end to end tests. used by .github/workflows/e2e-tests.yml See test/e2e/README.md for details
142145
.PHONY: e2e-local
143146
e2e-local: bin/e2e-local.test test/e2e-local.image.tar
144-
$(GINKGO) -nodes $(or $(NODES),1) -flakeAttempts 3 -randomizeAllSpecs $(if $(TEST),-focus '$(TEST)') -v -timeout 90m $< -- -namespace=operators -olmNamespace=operator-lifecycle-manager -dummyImage=bitnami/nginx:latest -kind.images=../test/e2e-local.image.tar
147+
$(GINKGO) -nodes $(or $(NODES),1) -flakeAttempts 1 -randomizeAllSpecs $(if $(TEST),-focus '$(TEST)') -v -timeout 90m $< -- -namespace=operators -olmNamespace=operator-lifecycle-manager -dummyImage=bitnami/nginx:latest -kind.images=../test/e2e-local.image.tar
145148

146149
# this target updates the zz_chart.go file with files found in deploy/chart
147150
# this will always fire since it has been marked as phony

0 commit comments

Comments
 (0)