[integration tests] provide a means to run tests from PR via opt-in, non-blocking mechanism

[kylos101]

Is your feature request related to a problem? Please describe

Integration tests run nightly, or ad-hoc.

Describe the behaviour you'd like

As an initial skateboard, give developers an opt-in, non-blocking way to easily run the tests. Perhaps as an additional check/item in PRs, such as:

• /werft with-integration-tests

For a follow-on skateboard, consider something like

• /werft with-integration-tests workspace

Where we could also do

• /werft with-integration-tests workspace ide (but not webapp, as an example)

Additional context

The thought is having an easier way to run tests will help teams at Gitpod more easily adopt and rely on the tests.

The preview environment for these test environments should not be the same as the live one used for preview. In other words, it should be possible to leave with-preview unchecked but check with-integration-tests.

Failed tests should not block the PR from merging.

[kylos101]

@meysholdt may we ask for your help in adding to Platform groundwork?

[mads-hartmann]

@kylos101 what's the motivation for this requirement:

The preview environment for these test environments should not be the same as the live one used for preview. In other words, it should be possible to leave with-preview unchecked but check with-integration-tests.

I ask because it complicates the solution a bit. We currently have a 1:1 mapping between branches and preview environments, so if we extend this with 1:* mapping we have to extend our GC of preview environments, for example ☺️

If the goal is to be able to run integration tests in a fresh environment, could we have an initial skateboard of having people use with-clean-slate-deployment when they want to ensure that they get a clean VM and Gitpod installation? E.g. you'd use with-integration-tests with-clean-slate-deployment (then with-preview is implied if you use with-integration-tests)

[kylos101]

@mads-hartmann good question! @meysholdt and I were worried that the integration tests would be extra load on the 1 preview environment. But, in hindsight, that may be premature optimization. Amazing what a night of rest will do.

Therefore, using the existing should be fine. And, since this is opt-in, it won't be disruptive. 👋

[mads-hartmann]

@meysholdt I'm assigning myself and adding it to next week as I'd like to look at this - I think we can get a skateboard out quite quickly ☺️ Please reach out if you'd prefer otherwise

[mads-hartmann]

A quick update. with-integration-tests already is an existing Werft flag, and it does indeed try to run all integration tests. However, in the last 60 days it has only been used 7 times across 3 branches (source) - and they were all by me 🤣 The job doesn't work as it was never updated to work with preview environments in Harvester (when I used with-integration-tests it was only to verify that Werft was still allowed to trigger other Werft jobs, as we were rolling out auth changes).

So the MVP is to make with-integration-tests work again. But I also want to extend it to be able to run team-specific tests as suggested in the issue description.

I might also consider doing a bit of clean-up because right now we have 3 different ways of running integration tests, that all do things slightly differently

1. We have with-integration-tests in our build job which trigger a new run-integration-tests.yaml job
2. IDE has ide-integration-tests-startup.yaml
3. Workspace has workspace-run-integration-tests.yaml

The IDE and Workspace jobs are very similar. They start a new preview environment by creating a new branch and triggering a Werft job. Waits for the preview environment to be happy and then uses go test -v ./... to run the tests.

The run-integration-tests.yaml job on the other hand expects a preview environment to exist, but the image for the job is our integration tests docker image eu.gcr.io/gitpod-core-dev/build/integration-tests:{{ .Annotations.version }} so it starts the jobs using the entrypoint.sh script.

[mads-hartmann]

Our README also has two other manual options from running integration tests during development.

It would be nice to consolidate some of this so there's one easy way to run the tests. Preferably the underlying mechanism should be the same so that it doesn't matter if you run it in Werft or in a Gitpod workspace as you develop.

[mads-hartmann]

From the PR

My intention is to use test/run.sh in both the Workspace and IDE integration test jobs too, so that we have a single way to run integration tests which is used in both Werft and Gitpod workspaces - but I'll look into this in a follow up PR.

We can follow up on this later if we feel like it. For now I'm considering this issue done ☺️