investigate how to speed up prebuilds on gitpod repo

[akosyakov]

Bug description

Right now there is a suspicion that gradle projects and Intellij indexing of them takes a lot of time. We should confirm whether it is true by profiling prebuilds somehow and understand why incremental prebuilds simply does not reuse dependencies and caches from last prebuild. We change these projects very rare.

Steps to reproduce

Run a prebuild on this repo.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[felladrin]

I've noticed components/ide/jetbrains/gateway-plugin:publish is being triggered every time due to this publishToJBMarketplace: true in WORKSPACE.yaml. (Take this Werft build log as example)

Using GitHub Code Search you can see that publishToJBMarketplace is aways true which will cause Gateway Plugin to always rebuild:

• https://cs.github.com/gitpod-io/gitpod?q=jbMarketplacePublishTrigger
• https://cs.github.com/gitpod-io/gitpod?q=publishToJBMarketplace

[Explosion-Scratch]

Yeah prebuilds are taking a really long time for me (+10min). It would be nice to simply not wait for the prebuild but instead see the progress of it in the editor via terminal

[akosyakov]

@felladrin Is it the same effect in prebuilds? We cab split build and publish components, that only second depends on this flag.

[mustard-mh]

As you can see in this prebuild job(incremental) https://hw-gradle-research.preview.gitpod-dev.com/projects/gitpod/c4d70b3b-fc76-4df4-8312-b2eed367b452

Time cost in incremental prebuild tasks (download/upload from/to GCP etc not included)

Task	StartTime	EndTime	Cost
gradle	Tue 10 May 2022 03:43:35 PM UTC	Tue 10 May 2022 03:45:04 PM UTC	1:30
yarn & build	Tue 10 May 2022 03:43:16 PM UTC	Tue 10 May 2022 03:48:07 PM UTC	5:00
go	Tue 10 May 2022 03:44:44 PM UTC	Tue 10 May 2022 03:44:57 PM UTC	0:15

incremental prebuilds works well in gradle, but not yarn & build

see also incremental-prebuilds-result-log.zip

you need to search with words below to see the real job after pre-prebuild (base) loaded

♻️ Re-running task as an incremental workspace prebuild

[mustard-mh]

incremental prebuilds will download pre-prebuild (base) result first, and this step will cost a lot of time too => prebuild in US cluster but base storage in EU

[mustard-mh]

result of prebuild locate region and count of Gitpod project since May 01 in prod DB

region	count
us43xl	327
us44	2
us44xl	303

They all in us cluster

---

The road will be like this

1. Trigger incremental prebuild in US cluster
2. Workspace download base (maybe larger than 7G) which storage in GCP EU
3. After prebuild, upload result (maybe larger than 7G) to EU ( in a US cluster workspace 😅
4. Start a workspace, to make user access in IDE, which will download prebuild result (maybe larger than 7G) in EU to US cluster workspace

So in this back and forth above we cost a lot of time in uploading and downloading

[mustard-mh]

How to improve it?

• we need node_modules for code intelligent, yarn build can only run in component like gitpod-protocol. component like dashboard don't need to build first
• gradle should not be inside prebuild, it will gen folder about 7G, and every start will download it, and every stop will store it, it cost a lot.
• gradle can be gen after ws start, it'll only cost around 5 mins for all.

improve improvements above can reduce size of prebuild, so it can reduce the time for back and forth in download and upload base prebuild to reduce time of incremental prebuilds.

[mustard-mh]

incremental prebuilds got some conditions, you can check them out here

gitpod/components/server/ee/src/workspace/workspace-factory.ts

Line 194 in 81a79af

private async isGoodBaseforIncrementalPrebuild(