Werft can no longer build my Pull Request (with-helm, with-payment)

[jankeromnes]

Bug description

In my Pull Request #8041, I use the following Werft options:

• /werft with-helm=true
• /werft with-payment=true

That's because I need a working Chargebee integration to exercise our paid plans code, and that only works with Helm currently.

It used to work quite well, until I needed to work on something else for a week or two, and coming back to the Pull Request (and rebasing it), the Werft build now always fails with:

Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition

Error Error: /usr/local/bin/helm3 upgrade --install --timeout 10m -f ../.werft/jobs/build/helm/values.nodeAffinities_5.yaml [...] --set components.registryFacade.ports.registry.servicePort=30203 gitpod . exit with non-zero status code
    at handleResult (/workspace/.werft/util/shell.ts:34:19)
    at exec (/workspace/.werft/util/shell.ts:52:9)
    at installGitpod (/workspace/.werft/jobs/build/deploy-to-preview-environment.ts:648:13)
    at /workspace/.werft/jobs/build/deploy-to-preview-environment.ts:614:9

E.g. https://werft.gitpod-dev.com/job/gitpod-build-jx-better-team-subscriptions.33/raw

Please help me figure out why it broke, and how to fix it again, so that I can simplify Gitpod's Team Plans.

Steps to reproduce

1. Try to build Implement a new Team Billing where Owners can conveniently manage a paid plan for their Team #8041 into a working preview env

Workspace affected

No response

Expected behavior

The build should succeed (and still have a payment integration).

Example repository

https://github.com/gitpod-io/gitpod

Anything else?

See also my initial report on Slack (internal)

[fullmetalrooster]

Working on another issue I encountered the problem that the secret proxy-config-certificates is not present and proxy, ws-proxy and registry-facade were not running. Creating it manually solved the problem. Maybe we introduced a bug during the process of issuing a new certificate?

[mads-hartmann]

@wulfthimm I'm assigning this to you and moving it to in-progress as it sounds like you're currently debugging this.

[fullmetalrooster]

@mads-hartmann Thx, I was about to do that.

[mads-hartmann]

@jankeromnes and I paired on this. The installation failed with Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition. To get the logs of the post-upgrade job we used the Logs Explorer in gitpod-core-dev with the following query

resource.labels.container_name="db-migrations"
resource.labels.namespace_name="staging-jx-better-8e4ec195b8"

And found the following log line (see link): Error: Environment variable undefined or empty: DB_ENCRYPTION_KEYS - see screenshot

Our theory is that this only happens when you use with-helm and have a migration (we haven't verified this though). Jan will look into why the ENV is missing. I'm closing this issue now as this is outside our scope ☺️

[meysholdt]

Re-Opening, because the error is still there and it affects users of every with-helm=true preview environment because db-migrations initialize the db when spinning up a preview environment.

Error:

Migration PWSUIndexes1643879235654 has been executed successfully.
query: SELECT COUNT(1) AS cnt FROM INFORMATION_SCHEMA.COLUMNS
            WHERE   table_schema = 'gitpod'
                AND table_name = 'd_b_auth_provider_entry'
                AND column_name = 'oauthRevision'
query: ALTER TABLE d_b_auth_provider_entry ADD COLUMN oauthRevision varchar(128) NOT NULL DEFAULT ''
query: SELECT COUNT(1) AS cnt FROM INFORMATION_SCHEMA.STATISTICS
            WHERE   table_schema = 'gitpod'
                AND table_name = 'd_b_auth_provider_entry'
                AND index_name = 'ind_oauthRevision'
query: CREATE INDEX ind_oauthRevision ON d_b_auth_provider_entry (oauthRevision)
Loading TypeORM entities and migrations from /app/node_modules/@gitpod/gitpod-db/lib/typeorm
Using DB: db:3306/gitpod
query: ROLLBACK
Error during migration run:
Error: Environment variable undefined or empty: DB_ENCRYPTION_KEYS
    at getEnvVar (/app/node_modules/@gitpod/gitpod-protocol/lib/env.js:44:15)
    at Config.get dbEncryptionKeys [as dbEncryptionKeys] (/app/node_modules/@gitpod/gitpod-db/lib/config.js:42:36)
    at Binding.dynamicValue (/app/node_modules/@gitpod/gitpod-db/lib/container-module.js:103:85)
    at /app/node_modules/inversify/lib/resolution/resolver.js:66:118
    at invokeFactory (/app/node_modules/inversify/lib/resolution/resolver.js:11:16)
    at /app/node_modules/inversify/lib/resolution/resolver.js:66:26
    at Array.map (<anonymous>)
    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:46:67)
    at /app/node_modules/inversify/lib/resolution/resolver.js:75:42
    at Array.map (<anonymous>)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[meysholdt]

The db-migration that starts requiring the DB_ENCYPTION_KEYS was introduces here:

gitpod/components/gitpod-db/src/typeorm/migration/1643986994402-OAuthRevision.ts

Line 37 in 58dff45

container.get<UserDB>(UserDB); // initializes encryptionProvider as side effect

This change from @geropl is from Feb 11.
However, @jankeromnes said: "This recently broke".
Not sure if there are more contributing factors.