Integration Tests

[csweichel]

This PR adds an integration test framework and a first set of actual tests.

The star of the show is an integration framework that integrates with Go unit tests. It provides

• access to Gitpod's internal API,
• access to the database,
• easy start and observation of workspaces,
• instrumentation of existing components.

Because integration tests are Go unit tests, they can be compiled to a binary and shipped in a Docker image. There is excellent tool support and loads of examples.

Access to Gitpod's internal API

Integration tests may need to interact with any component of Gitpod directly. The test framework can access

• ws-manager
• image-builder
• supervisor
• server

For all API access, we create a port-forward using the Kubernetes API and produce a client that access the API. For the Gitpod server, we'll create a fully privileged API token for the first non-builtin user we find. If there is no such user the test fails (see open points).

Examples:

• gitpod/test/tests/examples/server_test.go

  Lines 16 to 29 in 3b8b428

  	func TestServerAccess(t *testing.T) {
  	it := integration.NewTest(t)
  	defer it.Done()
  	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
  	defer cancel()
  	server := it.API().GitpodServer()
  	res, err := server.GetLoggedInUser(ctx)
  	if err != nil {
  	t.Fatal(err)
  	}
  	t.Log(res.Name)
  	}

• gitpod/test/tests/examples/wsmanager_test.go

  Lines 16 to 27 in 3b8b428

  	func TestGetWorkspaces(t *testing.T) {
  	it := integration.NewTest(t)
  	defer it.Done()
  	wsman := it.API().WorkspaceManager()
  	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
  	defer cancel()
  	_, err := wsman.GetWorkspaces(ctx, &api.GetWorkspacesRequest{})
  	if err != nil {
  	t.Fatal(err)
  	}
  	}

Access to the database

Inspecting or even modifying is yet another common task. The integration test framework provides easy database access. To this end, it inspects the db service and port-forwards to the backing pod (the same that kubectl port-forward service/db would do). It then grabs the DB password from the corresponding secret and opens a sql.DB connection.

Examples:

• gitpod/test/tests/examples/db_test.go

  Lines 13 to 35 in 3b8b428

  	func TestBuiltinUserExists(t *testing.T) {
  	it := integration.NewTest(t)
  	defer it.Done()
  	db := it.API().DB()
  	rows, err := db.Query(`SELECT count(1) AS count FROM d_b_user WHERE id ="builtin-user-workspace-probe-0000000"`)
  	if err != nil {
  	t.Fatal(err)
  	}
  	defer rows.Close()
  	if !rows.Next() {
  	t.Fatal("no rows selected - should not happen")
  	}
  	var count int
  	err = rows.Scan(&count)
  	if err != nil {
  	t.Fatal(err)
  	}
  	if count != 1 {
  	t.Fatalf("expected a single builtin-user-workspace-probe-0000000, but found %d", count)
  	}
  	}

Easy start and observation of workspaces

Starting workspaces and interacting with them is a common task. The integration test framework provides a simple API for waiting until a workspace is running.

There are two main ways how a workspace can be started:

1. using the server which follows the regular path that workspaces would take. I.e. things go through the context parser, workspace config factory and workspace starter. There is a convenience function that goes this route and then uses ws-manager to wait for the workspace startup.
2. using ws-manager which provides greater control over a workspace's configuration and has fewer external prerequisites (e.g. existing repositories, pre-existing users in the system). There is a convenience function that goes this route and then uses ws-manager to wait for the workspace startup.

When in doubt the ws-manager route is preferable, because it requires fewer external moving parts.

Examples:

• server route

  gitpod/test/tests/examples/server_test.go

  Lines 38 to 45 in 3b8b428

  	server := it.API().GitpodServer()
  	resp, err := server.CreateWorkspace(ctx, &protocol.CreateWorkspaceOptions{
  	ContextURL: "github.com/gitpod-io/gitpod",
  	Mode: "force-new",
  	})
  	if err != nil {
  	t.Fatalf("cannot start workspace: %q", err)
  	}

• ws-manager route

  gitpod/test/tests/workspace/tasks_test.go

  Lines 62 to 64 in 3b8b428

  	nfo := integration.LaunchWorkspaceDirectly(it,
  	integration.WithRequestModifier(addInitTask),
  	)

Instrumentation of existing components

To interact within the context of existing pods, the integration test framework supports "agents". Agents are tiny Go programs which are compiled on-demand (can also be precompiled and added to the path), uploaded to the pod, executed and port-fowarded to the test.
This mechanism allows for the inspection of component configuration (e.g. checking if ws-daemon can create a bucket with its configuration) and workspaces.

Example:

gitpod/test/tests/workspace/tasks_test.go

Lines 91 to 103 in 3b8b428

	rsa, err := it.Instrument(integration.ComponentWorkspace, "workspace", integration.WithInstanceID(nfo.Req.Id))
	if err != nil {
	t.Fatal(err)
	}
	defer rsa.Close()
	var ls agent.ListDirResponse
	err = rsa.Call("WorkspaceAgent.ListDir", &agent.ListDirRequest{
	Dir: "/workspace",
	}, &ls)
	if err != nil {
	t.Fatal(err)
	}

The example above will look for gitpod-integration-test-workspace-agent in the $PATH or try to compile workspace_agent/main.go. Those agents are expected to communicate using net/rpc, and there's a convenience function available for building them.

Example:
https://github.com/gitpod-io/gitpod/blob/cw/integration-tests/test/tests/workspace/workspace_agent/main.go#L14-L34

How can I run tests?

The integration tests are built, not run, as part of the components:all package, and result in a Docker image. This Docker image can be

• deployed to a Kubernetes cluster because the integration test framework supports in-Kubernetes config, using -kubeconfig in-cluster
• run outside a cluster e.g. docker run --rm -it -v $HOME/.kube/config:/config eu.gcr.io/gitpod-core-dev/build/integration-tests:cw-integration-tests.18 -test.v -kubeconfig /config

Because the integration tests are Go unit tests, you can also run them from the command line, e.g.:

cd test
go test -v ./...

or from within the editor:

Open Points

• Gitpod user: when accessing the Gitpod server API interface, we need a user to exist in the system already. Clearly this severely limits the tests usefulness in a fully automated (self-hosted) scenario. We need to figure out a way how we can add a user to the system. Possible ways are:
  • add a user directly to the database using the it.API().DB() functionality
  • register and provide a custom auth provider that works "headless"
  • try to work the login flow using puppeteer
• End-to-end tests: with the flexibility afforded by this framework I'd hope we rarely need actual end-to-end tests that simulate/operate a browser. Those kinds of tests are usually fickle and hard to maintain. That said, I reckon we will need those kinds of tests, too. chromedp might prove rather useful here.

[akosyakov]

For end-to-end we could look into https://playwright.dev/. It does not have issues with flakiness since tests are running next to app code, not remote communications and so on as with Selenium.

[csweichel]

https://playwright.dev/

Thanks for the pointer :) There also is https://github.com/mxschmitt/playwright-go, which uses Playwright but could be integrated into this framework.

That said, I'd hope we won't need those kinds of tests in the foreseeable future, due to their complexity and brittlenes. The more we can cover using other means, the better.

[geropl]

We need to figure out a way how we can add a user to the system. Possible ways are:

Another way @akosyakov and I explored back when we tried to make the integration tests work: https://github.com/gitpod-com/gitpod/pull/3757/files#diff-844058ea0e3f3ad9600eaa77711088f83e185cc2d9b9b66550198930b759d067
It's a not-exposed endpoint in server that creates a user + session for you.

[geropl]

I'm really looking forward to writing tests with this! Looks really nice from the outside.

(comment are questions/nits only)

[geropl]

connect to

[geropl]

Just to clarify: The value of this test lies in the fact that we run the same functionality in the same context.
But why not talk to ws-deamon directly instead? And: What is the benefit value in using agents for this compared to accessing the service directly? (Injecting agents into workspaces is another story altogether, there it's just awesome to have a functionality like this!)

[csweichel]

Indeed, that's the value of this test.

The trade-off/decision is exactly the one you outline: either embed the testing functionality into the components and make them available their gRPC interface, or we keep this logic out of the components and use agents instead. In this particular case I'm really really on the fence, because arguably trying to create a bucket is a self-test that ws-daemon could certainly run at start-up. Maybe this use of agents just isn't a good one.

My reasoning was that we have agents already, might as well use them rather than extend other components. I reckon there are other cases which are not as close a call, e.g. meddling with the on-disk state of ws-daemon. Certainly this could be done in a TestService, but it would be super intrusive to the component and spread the test out across several components (test and ws-daemon).

[geropl]

In this particular case I'm really really on the fenc

I reckon there are other cases which are not as close a call,

Thank you for elaborating; completely agree!