Preview Envs on Harvester: Delete when inactive

[meysholdt]

Is your feature request related to a problem? Please describe

Our Harvester cluster keeps filling up close to its max capacity. We should delete preview environments when they're inactive, just as we do for core-dev.

Describe the behavior you'd like

Implement

gitpod/.werft/platform-delete-preview-environments-cron.ts

Lines 82 to 87 in 539414d

	isInactive(): boolean {
	// We'll port over the logic from CoreDevPreviewEnvironment later, for now we consider
	// Harvester preview environments to never be stale due to inactivity.
	const sliceID = SLICES.CHECKING_FOR_NO_DB_ACTIVITY
	werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=false - Harvester based `)
	return false

By porting over the logic from core-dev:

gitpod/.werft/platform-delete-preview-environments-cron.ts

Lines 139 to 176 in 539414d

	isInactive(): boolean {
	const sliceID = SLICES.CHECKING_FOR_NO_DB_ACTIVITY
	try {
	const statusNS = exec(`KUBECONFIG=${CORE_DEV_KUBECONFIG_PATH} kubectl get ns ${this.namespace} -o jsonpath='{.status.phase}'`, { slice: sliceID })
	if (statusNS != "Active") {
	werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=false - The namespace is ${statusNS}`)
	return false
	}
	werft.log(sliceID, `${this.name} (${this.namespace}) - Checking status of the MySQL pod`)
	const statusDB = exec(`KUBECONFIG=${CORE_DEV_KUBECONFIG_PATH} kubectl get pods mysql-0 -n ${this.namespace} -o jsonpath='{.status.phase}'`, { slice: sliceID})
	const statusDbContainer = exec(`KUBECONFIG=${CORE_DEV_KUBECONFIG_PATH} kubectl get pods mysql-0 -n ${this.namespace} -o jsonpath='{.status.containerStatuses.*.ready}'`, { slice: sliceID})
	if (statusDB.code != 0 || statusDB != "Running" || statusDbContainer == "false") {
	werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=false - The database is not reachable`)
	return false
	}
	const dbPassword = exec(`KUBECONFIG=${CORE_DEV_KUBECONFIG_PATH} kubectl get secret db-password -n ${this.namespace} -o jsonpath='{.data.mysql-root-password}' | base64 -d`, {silent: true}).stdout.trim()
	const connectionToDb = `mysql --host=db.${this.namespace}.svc.cluster.local --port=3306 --user=root --database=gitpod -s -N --password=${dbPassword}`
	const latestInstanceTimeout = 48
	const latestInstance = exec(`${connectionToDb} --execute="SELECT creationTime FROM d_b_workspace_instance WHERE creationTime > DATE_SUB(NOW(), INTERVAL '${latestInstanceTimeout}' HOUR) LIMIT 1"`, { slice: sliceID})
	const latestUserTimeout = 48
	const latestUser= exec(`${connectionToDb} --execute="SELECT creationDate FROM d_b_user WHERE creationDate > DATE_SUB(NOW(), INTERVAL '${latestUserTimeout}' HOUR) LIMIT 1"`, { slice: sliceID})
	const lastModifiedTimeout = 48
	const lastModified= exec(`${connectionToDb} --execute="SELECT _lastModified FROM d_b_user WHERE _lastModified > DATE_SUB(NOW(), INTERVAL '${lastModifiedTimeout}' HOUR) LIMIT 1"`, { slice: sliceID})
	const heartbeatTimeout = 48
	const heartbeat= exec(`${connectionToDb} --execute="SELECT lastSeen FROM d_b_workspace_instance_user WHERE lastSeen > DATE_SUB(NOW(), INTERVAL '${heartbeatTimeout}' HOUR) LIMIT 1"`, { slice: sliceID})
	const isInactive = (heartbeat.length < 1) && (latestInstance.length < 1) && (latestUser.length < 1) && (lastModified.length < 1)
	werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=${isInactive}`)
	return isInactive
	} catch (err) {

[meysholdt]

It would be lice if the job would also log which db rows are the cause for considering an env active. This can help to understand if we're looking at the right signals.

Example: Change

gitpod/.werft/platform-delete-preview-environments-cron.ts

Line 174 in 539414d

werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=${isInactive}`)

to

werft.log(sliceID, `${this.name} (${this.namespace}) - is-inactive=${isInactive}. Active DB-Rows: d_b_workspace_instance:${latestInstance.length}, d_b_user-created:${latestUser.length}, d_b_user-modified:${lastModified.length}, d_b_workspace_instance_user:${heartbeat.length}`);

[meysholdt]

An important use-case for this feature: When a preview env gets created from a branch that only has old commits, it's being GCed immediately today. Implementing this feature would have prevented that.

Context: https://gitpod.slack.com/archives/C032A46PWR0/p1652431918110569?thread_ts=1652427026.649699&cid=C032A46PWR0

[mads-hartmann]

When a preview env gets created from a branch that only has old commits, it's being GCed immediately today. Implementing this feature would have prevented that.

Currently the heuristics in the job means that if either of the conditions is true, it will delete the preview environment. So in this case, the preview environment would still be deleted as the branch is stale.

I think it's a good idea to change that though, e.g. so that if there's any DB activity we don't delete the preview environment.

[mads-hartmann]

@vulkoingim I'm reopening this as there's still a tiny bit missing. Please see my comment here #10379 (review) ☺️

[vulkoingim]

@vulkoingim I'm reopening this as there's still a tiny bit missing. Please see my comment here #10379 (review) ☺️

👀

This part addresses it: https://github.com/gitpod-io/gitpod/blob/main/.werft/platform-delete-preview-environments-cron.ts#L383-L386

And looking at the logs it has caught quite a few already.