Make TakeSnapshot more reliable #6144

geropl · 2021-10-11T08:38:31Z

Description

This fixes #5862 by following this idea: #5862 (comment)

In detail

we add a state: 'pending' | 'error' | 'available'' field to Snapshot (+ message and availableTime)
we add a method content-service.WorkspaceSnapshotExists
we make sure ws-daemon.TakeSnapshot returns early
we add an API method server.waitForSnapshot
we add a SnapshotService to server which is responsible for making sure all
we change the semantics fo server.takeSnapshot to:
- return as early as we have the upload file URL from ws-daemon, stored that in the DB (with state: 'pending') and return the snapshotId
- we make sure SnapshotService monitors this DB entry and regularly checks if it's done or timed out, resulting in an update to the DB
- clients have to take that snapshotId and have to call server.waitForSnapshot to wait for the snapshot to become available (or error in case sth went south). This internally waits for the same Promise SnapshotService is already driving

Regarding deployment

This change has breaking changes to IDE and workspace. To untangle and make the deployment as easy as possible:

once reviewed (and tested) split this PR into two: IDE and workspace + meta
server and workspace are fully backwards-compatible to each other; server API is backwards compatible as well. Thus we have to make sure meta is deployed before IDE, the rest does not matter.

Related Issue(s)

Fixes #5862

IDE companion PR: gitpod-io/openvscode-server#169

How to test

open a dev workspace on this branch: https://gitpod.io/#https://github.com/gitpod-io/gitpod/pull/6144
prepare to execute kubectl rollout restart deployment proxy
start a test workspace with this link: https://gpl-5862-snapshot.staging.gitpod-dev.com/#https://github.com/shaal/gitpod-test-failing-snapshot (thx @shaal ! 🙏 )
F1 -> snapshot -> enter
execute kubectl rollout restart deployment proxy from the dev workspace (this simulates ws connection loss)
note how the call fails (shows that it's backwards compatible!)
~~note how the snapshot process is not interrupted in the IDE of the test workspace~~
~note that after ~~2mins the snapshot is successful~~
~~test the snapshot link and note that it works~~

Release Notes

make "snapshots" more reliable

Documentation

/werft with-clean-slate-deployment

geropl · 2021-10-11T08:38:45Z

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.5

codecov · 2021-10-20T12:03:58Z

Codecov Report

Merging #6144 (3534f69) into main (0514edc) will increase coverage by 14.43%.
The diff coverage is 0.00%.

❗ Current head 3534f69 differs from pull request most recent head 9ed6299. Consider uploading reports for the commit 9ed6299 to get more accurate results

@@             Coverage Diff             @@
##             main    #6144       +/-   ##
===========================================
+ Coverage   19.04%   33.48%   +14.43%     
===========================================
  Files           2       23       +21     
  Lines         168     4680     +4512     
===========================================
+ Hits           32     1567     +1535     
- Misses        134     2988     +2854     
- Partials        2      125      +123

Flag	Coverage Δ
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
components-ws-manager-app	`39.55% <0.00%> (?)`
installer-app	`7.45% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/ws-manager/pkg/manager/manager_ee.go	`0.00% <0.00%> (ø)`
components/local-app/pkg/auth/auth.go
components/local-app/pkg/auth/pkce.go
installer/pkg/components/ws-manager/tlssecret.go	`0.00% <0.00%> (ø)`
components/ws-manager/pkg/manager/metrics.go	`11.11% <0.00%> (ø)`
installer/pkg/components/ws-manager/rolebinding.go	`0.00% <0.00%> (ø)`
components/ws-manager/pkg/manager/probe.go	`0.00% <0.00%> (ø)`
components/ws-manager/pkg/manager/imagespec.go	`0.00% <0.00%> (ø)`
components/ws-manager/pkg/manager/status.go	`73.77% <0.00%> (ø)`
...staller/pkg/components/ws-manager/networkpolicy.go	`0.00% <0.00%> (ø)`
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0514edc...9ed6299. Read the comment docs.

geropl · 2021-11-03T14:04:10Z

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.45

another try---

geropl · 2021-11-03T15:05:09Z

/werft run

👍 started the job as gitpod-build-gpl-5862-snapshot.47

...and again

geropl · 2021-11-03T15:17:42Z

/werft run with-clean-slate-deployment

👍 started the job as gitpod-build-gpl-5862-snapshot.48

The best things come in threes. 🤞

geropl · 2021-11-03T15:55:03Z

/unhold

Ready to merge now

JanKoehnlein · 2021-11-04T10:46:58Z

/lgtm

roboquat · 2021-11-04T10:47:01Z

LGTM label has been added.

Git tree hash: e74e73cd5ea3964fe6b81a871af7253bdf9429f6

gtsiolis

Tried this for a workspace larger than 3.5GB which used to fail and worked as expected! 🔮

Thanks @geropl! 🏁

csweichel · 2021-11-04T11:17:09Z

did not test, but API and service changes lgtm

/lgtm

roboquat · 2021-11-04T11:17:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csweichel, JanKoehnlein

Associated issue: #5862

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~components/content-service/OWNERS~~ [csweichel]
~~components/content-service-api/OWNERS~~ [csweichel]
~~components/gitpod-db/OWNERS~~ [JanKoehnlein]
~~components/gitpod-protocol/OWNERS~~ [JanKoehnlein]
~~components/server/OWNERS~~ [JanKoehnlein]
~~components/ws-daemon/OWNERS~~ [csweichel]
~~components/ws-daemon-api/OWNERS~~ [csweichel]
~~components/ws-manager/OWNERS~~ [csweichel]
~~components/ws-manager-api/OWNERS~~ [csweichel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

shaal · 2021-11-04T14:09:46Z

I can't wait for this feature to be working again 🎉
When do you expect this to be available on gitpod.io ?

gtsiolis · 2021-11-04T14:23:31Z

Thanks, @shaal! The fix should probably land in production within a week.

FYI, we're trying to get better at deploying to production sooner and more often.
We're aiming at deploying at least twice a week. 💡

geropl · 2021-11-08T09:09:25Z

@gtsiolis Just to clarify: This PR is just introducing the server/workspace changes. IDE change will land later, I expect next week.

gtsiolis · 2021-11-08T10:43:57Z

Thanks for clarifying, @geropl!

Is there an open issue for the IDE changes?

Cc @shaal

geropl · 2021-11-08T10:55:12Z

It's the same issue: #5862

Draft PR is here: gitpod-io/openvscode-server#169

roboquat added do-not-merge/work-in-progress do-not-merge/release-note-label-needed labels Oct 11, 2021

roboquat requested review from JanKoehnlein and loujaybee October 11, 2021 08:39

roboquat added the size/S label Oct 11, 2021

geropl removed request for JanKoehnlein and loujaybee October 11, 2021 08:41

geropl force-pushed the gpl/5862-snapshot branch 2 times, most recently from 0f0db41 to ccff46b Compare October 11, 2021 14:03

roboquat added size/XS and removed size/S labels Oct 11, 2021

geropl force-pushed the gpl/5862-snapshot branch from ccff46b to 61c9cef Compare October 12, 2021 07:12

roboquat added size/S and removed size/XS labels Oct 12, 2021

geropl force-pushed the gpl/5862-snapshot branch 2 times, most recently from a6fa036 to d04812d Compare October 12, 2021 09:36

roboquat added team: IDE team: webapp Issue belongs to the WebApp team team: workspace Issue belongs to the Workspace team size/XXL and removed size/S labels Oct 18, 2021

geropl force-pushed the gpl/5862-snapshot branch 3 times, most recently from 61ca4b1 to c19e486 Compare October 20, 2021 11:50

geropl force-pushed the gpl/5862-snapshot branch 4 times, most recently from 4f87159 to 0e47cfc Compare October 21, 2021 07:30

[server, protocl] Introduce 'waitForSnapshot'

8fa4002

geropl force-pushed the gpl/5862-snapshot branch from 13514a5 to 0491ef5 Compare November 3, 2021 13:35

roboquat removed the lgtm label Nov 3, 2021

[server] Build out SnapshotService

ef7cdd5

geropl force-pushed the gpl/5862-snapshot branch from 0491ef5 to ef7cdd5 Compare November 3, 2021 14:20

[db] Bump migration timestamp

9ed6299

geropl force-pushed the gpl/5862-snapshot branch from 8f4fe1c to 9ed6299 Compare November 3, 2021 15:37

geropl requested review from csweichel and removed request for akosyakov November 3, 2021 15:54

roboquat removed the do-not-merge/hold label Nov 3, 2021

roboquat added the lgtm label Nov 4, 2021

gtsiolis reviewed Nov 4, 2021

View reviewed changes

roboquat assigned csweichel Nov 4, 2021

roboquat added the approved label Nov 4, 2021

roboquat merged commit 61ea972 into main Nov 4, 2021

roboquat deleted the gpl/5862-snapshot branch November 4, 2021 11:18

gtsiolis mentioned this pull request Nov 25, 2021

Cannot generate a workspace snapshot #5862

Closed

Make TakeSnapshot more reliable #6144

Make TakeSnapshot more reliable #6144

Uh oh!

Conversation

geropl commented Oct 11, 2021 • edited by gtsiolis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

In detail

Regarding deployment

Related Issue(s)

How to test

Release Notes

Documentation

Uh oh!

geropl commented Oct 11, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

geropl commented Nov 3, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geropl commented Nov 3, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geropl commented Nov 3, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geropl commented Nov 3, 2021

Uh oh!

JanKoehnlein commented Nov 4, 2021

Uh oh!

roboquat commented Nov 4, 2021

Uh oh!

gtsiolis left a comment

Choose a reason for hiding this comment

Uh oh!

csweichel commented Nov 4, 2021

Uh oh!

roboquat commented Nov 4, 2021

Uh oh!

shaal commented Nov 4, 2021 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gtsiolis commented Nov 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geropl commented Nov 8, 2021

Uh oh!

gtsiolis commented Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geropl commented Nov 8, 2021

Uh oh!

Uh oh!

geropl commented Oct 11, 2021 •

edited by gtsiolis

Loading

geropl commented Oct 11, 2021 •

edited by werft-gitpod-dev-com bot

Loading

codecov bot commented Oct 20, 2021 •

edited

Loading

geropl commented Nov 3, 2021 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Nov 3, 2021 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Nov 3, 2021 •

edited by werft-gitpod-dev-com bot

Loading

shaal commented Nov 4, 2021 •

edited by werft-gitpod-dev-com bot

Loading

gtsiolis commented Nov 4, 2021 •

edited

Loading

gtsiolis commented Nov 8, 2021 •

edited

Loading